Confidence-Aware Explainable AI for Reliable Large Language Model Decision Support

Large language models (LLMs) are increasingly used in decision-support systems because of their ability to interpret complex queries, synthesize information, and generate contextually relevant responses. However, their use in reliability sensitive settings remains challenging because LLMs can produce incorrect or unsupported outputs while expressing a high degree of linguistic confidence. This paper presents a confidence aware explainable AI framework for improving the reliability of LLM-based decision support. The proposed approach combines model confidence, semantic consistency across multiple stochastic generations, and explanation support to construct a composite reliability score. A selective decision module then uses this score to determine whether a generated response should be accepted, flagged for review, or withheld through abstention. The framework is evaluated across factual, commonsense, and multi-step reasoning tasks using multiple open-weight LLMs. Performance is assessed in terms of answer accuracy, expected calibration error, failure-detection AUROC, selective accuracy, and response coverage. The evaluation investigates whether combining complementary reliability signals provides a more informative assessment of response correctness than self-reported confidence, model confidence, or semantic consistency alone. The proposed framework provides a practical reliability layer for LLM-based decision-support systems by jointly communicating confidence, explanation support, and an actionable response decision, thereby helping distinguish sufficiently reliable outputs from those requiring further verification or abstention.