Confidence Is Not Probability: Verification-Based Confidence Prediction for LLM-Generated Clinical Summaries

Objective. Token-level confidence estimates from large language models (LLMs) do not reflect whether a generated clinical summary is factually correct. We investigate whether verification-based confidence prediction improves upon token-level estimates and whether LLM-as-judge labels can be assumed reliable for clinical summarization evaluation. Methods. Using MIMIC-IV, we construct 500 clinical summarization tasks across four open-source LLMs (Qwen2.5-14B/32B, Phi-4, Llama-3.1-8B), producing 2,000 primary summaries and 1,500 self-consistency samples. Each summary is scored on 26 verification features spanning coverage, factual consistency, natural language inference, numerical accuracy, semantic similarity, and self-consistency. Ground-truth labels are generated by Codex-GPT-5 and validated by two board-certified physicians on 200 summaries. Results. Three judges produce markedly different label distributions (Codex-GPT-5: 93.4% problematic, Phi-4: 10.5%, physicians: 59.5%) with near-zero pairwise agreement (\(\kappa = 0.009\)–$0.151$). Each physician independently identified only 1 of 200 summaries as fully trustworthy. ClinTrust achieves AUROC = 0.846 [95% CI: 0.788, 0.896] against Codex-GPT-5 labels and 0.695 [0.619, 0.768] against physician labels, significantly outperforming single-feature baselines including token log-probability (0.583) and BERTScore-F1 (0.607). Conclusions. Multi-source verification improves upon token-level confidence for predicting summary reliability, but LLM-as-judge labels are highly variable and cannot be assumed reliable without human validation.