Explainable AI for Criterion-Referenced Oral English Assessment: A Measured Pilot Study

Oral examination is a high-stakes application of artificial intelligence (AI) in higher education, requiring transparency and human oversight. This study presents a pilot AI examiner for university oral English examinations following the IELTS Speaking format. The system combines automatic speech recognition (ASR) using Gemini 2.5 Flash, deterministic extraction of fluency, lexical, syntactic, and semantic indicators, and rubric-constrained assessment using Gemini 2.5 Flash-Lite. Low-confidence cases are automatically escalated, and final weighted scores are calculated programmatically, while instructors retain authority over academic decisions. The pilot analyzed 45 spoken responses, of which 26.7% lacked usable word-level timestamps, preventing reliable calculation of temporal indicators and highlighting ASR-related measurement limitations. A post-examination survey collected responses from 67 students, achieving a 62% response rate, to evaluate the internal consistency of composite indices. Comparisons between AI-generated preliminary scores and instructor-released grades across 23 paired sessions revealed near-zero correlations and a pronounced ceiling effect in instructor scores. Consequently, these grades could not serve as a valid criterion for evaluating AI scoring accuracy. Instructor feedback also identified operational challenges requiring refinement. Based on these findings, the study establishes a three-level trust framework grounded in observable pipeline conditions and measured failure frequencies. It also proposes a validation protocol involving two independent human raters and quadratic-weighted kappa to assess inter-rater agreement and support future validation of AI-generated scores. Overall, the findings demonstrate the feasibility of transparent, rubric-based AI-assisted oral assessment while emphasizing measurement limitations, standardized expert ratings, and human oversight.