Immune checkpoint inhibitors have remade cancer treatment, and they bring a tax. Up to 40% of patients develop immune-related adverse events, or irAEs, which must be spotted in free-text notes few people have time to read carefully.
A team led by Jack Gallifant and Danielle Bitterman at Brigham and Women’s Hospital and Dana-Farber Cancer Institute built an agentic system that splits the job. Separate pipelines handle pneumonitis, hepatitis, colitis, myocarditis, thyroiditis and dermatitis, running each note through detection, CTCAE v5.0 severity grading, attribution and certainty.
The design choice that carried the result was redundancy. Detection and grading each ran three times, with a judge agent reconciling disagreements and pulling supporting text spans. Removing that step dropped detection F1 from 0.92 to 0.78 while only halving cost.
On 263 expert-annotated notes from 2015 to 2024, the best configuration hit macro F1 of 0.92 for active events and 0.91 for resolved ones. Hepatitis and pneumonitis scored 0.96, myocarditis trailed at 0.86. Severity grading was harder at 0.66. Each note cost about $0.02.
Then came the real test. Deployed silently in a live electronic health record from May to July 2025, the frozen system processed 884 notes in near real time across 31,824 label adjudications. Binary F1 fell to 0.72 to 0.79.
In a randomized crossover study, clinical trial staff using the agent finished annotation 2.71 minutes faster with more complete matches. Pairing an agent with a human reviewer, rather than replacing one, is what the authors argue makes this usable.
