Measuring Agreement Between AI and the Radiologist: A Methodological Review of Evaluation Metrics for Study-Level Concordance in Real-World, Whole-Study CT and MRI

As multimodal large language models (LLMs) are applied to computed tomography (CT) and magnetic resonance imaging (MRI), how agreement between an artificial intelligence (AI) system and the radiologist is measured determines whether reported progress is believable. This structured narrative review maps the metric families for AI–radiologist concordance on CT/MRI reports and assesses each against a demanding setting: whole-study analysis of real-world data, where an individual submits a single study without priors, laboratory data or indication, and the radiologist report is an imperfect, non-exhaustive reference, a premise anchored in the radiologist discrepancy literature. We reviewed lexical, clinical/factuality, composite and LLM-as-judge metrics; inter-rater agreement statistics (the kappa family, Gwet’s AC1, PABAK, Krippendorff’s alpha, intraclass correlation) and the unit of agreement each presupposes; LLM-as-judge biases; and reporting standards. Entity-based and LLM-as-judge metrics with explicit error taxonomies align best with radiologist judgement but require validation. A worked example shows Cohen’s and Fleiss’ kappa collapsing under the class imbalance typical of findings data; prevalence-robust coefficients are more trustworthy, and Fleiss’ kappa belongs to multi-reader panel reliability, not AI-versus-report concordance. We recommend a layered metric stack anchored on significance-tiered finding recall, computed by an LLM judge validated as non-inferior to human–human agreement, and reported under CLAIM 2024, STARD-AI and TRIPOD-LLM.