Format-Dependent Performance of Four Large Language Models on Japanese Dental Technician Examination Questions: A Matched Item-Level Analysis

1. Introduction

Large language models (LLMs) are increasingly evaluated as general-purpose systems for knowledge-intensive question answering, decision support, and educational applications. Professional licensing examinations offer standardized benchmarks for assessing these capabilities because they combine domain-specific knowledge with predefined answer keys. Nevertheless, examination accuracy can vary substantially across models, languages, assessment settings, and question formats. A systematic review and meta-analysis of medical licensing examinations documented differences between ChatGPT versions and across evaluation contexts [1]. Studies involving Chinese, Peruvian, and Chilean licensing examinations likewise demonstrated that performance observed in one examination or model version cannot be assumed to generalize to another [2,3,4]. These findings underscore the importance of evaluating contemporary LLMs under directly comparable conditions rather than inferring relative performance from separate studies.

In dentistry, LLMs have been assessed on national licensing questions and other educational tasks, with heterogeneous results. GPT-4 outperformed GPT-3.5 on written dental licensing questions from the United States and the United Kingdom [5], while comparisons involving Korean dental licensing examinations showed that some advanced models exceeded examination cutoff scores but remained below human examinees [6]. Evaluations of the Japanese National Dental Hygienist Examination and the Japanese National Dental Examination further identified differences across models, examination years, and question categories [7,8]. Studies of specialty-specific educational questions, periodontal surgery, endodontic cases, and educational content generation have likewise reported task-dependent performance [9,10,11,12]. Collectively, these studies suggest potential educational applications for LLMs, but they also demonstrate that aggregate performance depends on the particular task and evaluation setting.

One important source of variation is the presence of visual information. Dental questions may require interpretation of radiographs, clinical photographs, anatomical illustrations, prosthetic designs, instruments, or laboratory procedures in combination with written descriptions. Prior assessments of healthcare licensing questions have identified limitations in visually based tasks, even when vision-capable models were used [4,13]. More recent dental studies comparing multiple multimodal LLMs reported lower accuracy on visually based questions and differences across specialties or image categories [14,15,16]. From a benchmarking perspective, however, a comparison between questions originally accompanied by visual materials and text-only questions does not isolate the causal effect of image input. These item groups may also differ in subject matter, difficulty, wording, and required reasoning. Accordingly, observed differences should be interpreted as associations with examination format rather than direct measurements of visual-recognition ability.

The Japanese National Dental Technician Examination provides a useful setting for studying such format dependence. Dental technology combines theoretical knowledge with interpretation of morphology, occlusion, dental materials, prosthetic appliances, instruments, and practical laboratory processes. Its examinations therefore include both text-only questions and questions accompanied by visual materials. An earlier evaluation of 400 questions from the 2018–2022 examinations assessed ChatGPT-4o and documented variation in accuracy by question format [17]. A separate matched comparison of GPT-4o, OpenAI o1, and Claude 3.5 Sonnet on 240 questions from the 2022–2024 examinations also reported lower accuracy for visually based items [18]. These studies established the relevance of this examination to multimodal LLM evaluation, but they did not test the four models considered in the present study within a single complete five-year dataset.

Of particular relevance, our previous study published in Computers compared GPT-4o, o3, and GPT-5T using 399 Japanese dental board-style questions from 2018–2022 [19]. That study used paired comparisons and stratified analyses to examine performance across items with and without visual materials. It identified differences among models and suggested that their relative advantages could vary by examination format. The present study is not intended as a replication of that analysis: the model responses were collected separately, the full set of 400 examination items was included, and Gemini 3 was added to enable a comparison across model developers. Related investigations of dental technician questions in Taiwan and dental licensing questions in China have compared other model families, including Gemini and GPT-based systems [20,21]. However, such findings cannot establish the relative performance of these specific four models on an identical Japanese dental technician examination dataset.

A further methodological issue arises when several models answer the same examination questions. Their correctness outcomes are paired at the item level, and questions differ in unobserved difficulty and other characteristics. Reporting independent overall percentages or separate subgroup accuracies alone does not fully address this dependence or directly evaluate whether the relative differences among models change with question format. A question-level mixed-effects framework can represent the shared item structure while adjusting for examination year and estimating model-by-item-type interactions. This distinction is important: a model may achieve higher overall accuracy without showing an equally large advantage for questions originally accompanied by visual materials. Assessing effect modification thus provides information that is not captured by a single aggregate benchmark score.

Against this background, the present study compared GPT-4o, o3, GPT-5 Thinking, and Gemini 3 on all 400 questions from the 2018–2022 Japanese National Dental Technician Examinations. All four models were evaluated on the same items, comprising 252 text-only questions and 148 questions originally accompanied by visual materials. Overall differences in correctness were examined using matched statistical tests. In addition, a mixed-effects logistic regression model included model, item type, examination year, and model-by-item-type interactions as fixed effects, with examination question as a random intercept. This design enabled both direct four-model benchmarking and an explicit test of whether between-model differences were associated with examination format.

The primary objective was to compare item-level answer accuracy among the four models across the complete matched dataset. The secondary objective was to determine whether the association between model and correctness differed between text-only questions and questions originally accompanied by visual materials. We hypothesized that accuracy would differ across models and that the magnitude of between-model differences would vary according to item type. By combining a cross-developer comparison with a question-level interaction analysis, this study aims to clarify the limits of aggregate accuracy as a metric for evaluating LLM performance on professional examination tasks.

2. Materials and Methods

2.1. Study Design and Examination Dataset

This retrospective, matched item-level benchmarking study evaluated four large language models (LLMs)—GPT-4o, o3, GPT-5 Thinking, and Gemini 3—using the complete set of 400 questions from the Japanese National Dental Technician Examinations administered in 2018–2022 (80 questions per year). All four models provided responses to every question, yielding four correlated observations per examination item. The response dataset analyzed here was collected separately from that reported in our previous three-model comparison [19]; therefore, results from the two studies should not be interpreted as repeated measurements of changes in model performance over time. The model labels “GPT5T” and “GPT5Thinking” in the source records were standardized to “GPT-5 Thinking” for reporting.

2.2. Question Classification and Visual Materials

The original examination questions were classified using the dataset indicator for accompanying figures or photographs. Of the 400 items, 252 were text-only and 148 were originally accompanied by visual materials, including figures, photographs, or diagrams. Classification reflected the original examination format, not a controlled manipulation of image availability. Visual-material and text-only questions were distinct groups and were not matched for subject matter, cognitive demands, or intrinsic difficulty. Accordingly, comparisons between them describe associations with question format rather than the isolated causal effect of images. The number of questions by year and format is presented in Table 1.

2.3. Model Access and Response Collection

Four large language models (LLMs)—GPT-4o, o3, GPT-5 Thinking (OpenAI, San Francisco, CA, USA), and Gemini 3 (Google, Mountain View, CA, USA)—were evaluated through their respective consumer web interfaces using paid subscription plans rather than application programming interfaces (APIs). All model responses were collected in June 2026, independently of the dataset used in our previously published three-model comparison [19]. Model names were standardized throughout the manuscript to ensure consistent reporting.

All 400 examination questions, including their answer options, were presented in their original Japanese language. Questions were submitted sequentially, one at a time, within the same conversation for each model, rather than initiating a new conversation for every question. Consequently, preceding questions and model-generated responses may have remained within the conversation context and potentially influenced subsequent answers. Each question was evaluated once per model, and only the initial response was retained for scoring. No responses were regenerated, and no repeated trials were performed.

For text-only questions, the question stem and answer options were entered into the corresponding web interface. For questions originally accompanied by visual materials, the relevant figures or photographs were extracted from the official examination PDF files and submitted together with the corresponding question stem and answer options. The same source materials were used across all four models. However, image processing and display conditions imposed by the respective interfaces, including potential automatic resizing or compression, were not under investigator control, and pixel-level equivalence of the displayed images across platforms was not verified.

Web searching, browsing, and external information retrieval functions were disabled throughout the evaluation process. The official answer keys were not provided to the models during response generation. A standardized prompt was consistently applied across all four models. The prompt was as follows:

“The Japanese National Dental Technician Examination question collection is attached. Please answer the questions as a candidate taking the Japanese National Dental Technician Examination. For each question, select one correct answer from options a, b, c, or d, and provide your answers sequentially. Do not provide any comments or explanations regarding your answers. Do not display any comments or commented-out text. If you are unable to answer a question, respond with ‘Unable to answer’ instead of selecting an answer option.”

The generated responses were subsequently compared with the official examination answer keys to determine correctness. Because the models were accessed through consumer web interfaces rather than APIs, generation parameters, including temperature and sampling settings, could not be directly controlled by the investigators. Accordingly, the observed performance reflects the models’ behavior under the specified web-interface evaluation conditions rather than under standardized API-based settings.

2.4. Outcome Definition and Scoring

The primary outcome was answer correctness at the item level, determined by comparison with the official examination answer key. Responses were coded as correct (1) or incorrect (0). For questions requiring more than one answer, a response was considered correct only if the complete required combination of options matched the answer key; no partial credit was awarded. Model accuracy was calculated as the percentage of correctly answered questions, overall and by original question format.

2.5. Statistical Analysis

Descriptive accuracy estimates for each model were accompanied by Wilson 95% confidence intervals (CIs). Since each model answered the same 400 items, an overall comparison of the four dependent binary outcomes was performed using Cochran’s Q test. Pairwise differences were then assessed with exact McNemar tests; the six prespecified pairwise p values were corrected using the Holm procedure. Paired accuracy differences, discordant-pair odds ratios, and multiplicity-adjusted p values were reported. CIs for paired accuracy differences were estimated by a normal approximation to item-level paired differences. Accuracy was also summarized separately for text-only and visual-material questions.

2.6. Question-Level Mixed-Effects Model

To account for clustering of the four model responses within each examination question, a mixed-effects logistic regression model was fitted with binary correctness as the dependent variable. The fixed effects comprised model identity (four levels), original item type (text-only versus visual-material), examination year (2018–2022), and the model-by-item-type interaction. Examination question was included as a random intercept to account for residual between-question heterogeneity. GPT-4o, text-only questions, and examination year 2018 served as the reference categories. Parameters were estimated by maximum likelihood with 20-point Gauss–Hermite quadrature. The omnibus model-by-item-type interaction was assessed using a likelihood-ratio test comparing the full model with a reduced model excluding the three interaction terms (3 degrees of freedom). Adjusted odds ratios and corresponding 95% Wald CIs were obtained from the fitted coefficient estimates and inverse-Hessian covariance matrix. The interaction evaluates whether relative model performance differs by the original format of the question; it does not estimate the causal contribution of the image itself.

2.7. Adjusted Predictions and Software

For descriptive interpretation of the fitted model, predicted probabilities of correctness were calculated using the fixed-effects component with the question-level random intercept set to zero, then averaged with equal weight across the five examination years. Their 95% CIs were obtained using the delta method and the estimated fixed-effect covariance matrix. All hypothesis tests were two-sided, with a significance threshold of p < 0.05. Analyses were performed using Python 3.13.5, NumPy 2.3.5, SciPy 1.17.0, and statsmodels 0.14.6.

2.8. Ethical Considerations

This study analyzed publicly available examination questions and machine-generated answers. No human participants, identifiable personal information, or individual examinee records were included; consequently, ethical review and informed consent were not required.

3. Results

3.1. Examination Dataset

The analysis comprised all 400 questions from the Japanese National Dental Technician Examinations administered between 2018 and 2022 (80 questions per year). Of these, 252 (63.0%) were text-only questions and 148 (37.0%) were originally accompanied by visual materials. The numbers of text-only and visual-material questions were 58 and 22 in 2018, 54 and 26 in 2019, 48 and 32 in 2020, 46 and 34 in 2021, and 46 and 34 in 2022, respectively (Table 1). All four models provided a scorable response to every question.

3.2. Overall Accuracy and Matched Pairwise Comparisons

Overall accuracy differed significantly among the four models (Cochran’s Q = 47.182, df = 3, p < 0.001). Gemini 3 answered 322/400 questions correctly (80.5%; 95% confidence interval [CI], 76.3–84.1%), followed by GPT-5 Thinking (299/400; 74.8%; 95% CI, 70.3–78.8%), o3 (283/400; 70.8%; 95% CI, 66.1–75.0%), and GPT-4o (255/400; 63.8%; 95% CI, 58.9–68.3%) (Table 2 and Figure 1).

In pairwise exact McNemar tests with Holm correction across six comparisons, GPT-4o performed less accurately than o3 (adjusted p = 0.0450), GPT-5 Thinking (adjusted p = 0.0001), and Gemini 3 (adjusted p < 0.0001). Gemini 3 also outperformed o3 (adjusted p = 0.0003) and GPT-5 Thinking (adjusted p = 0.0450). The difference between o3 and GPT-5 Thinking was not statistically significant after adjustment (adjusted p = 0.0888). Corresponding paired accuracy differences and discordant-response counts are presented in Table 2.

3.3. Accuracy Stratified by Examination Item Type

Among the 252 text-only questions, accuracy was 72.2% for GPT-4o (182/252; 95% CI, 66.4–77.4%), 79.8% for o3 (201/252; 95% CI, 74.4–84.3%), 86.1% for GPT-5 Thinking (217/252; 95% CI, 81.3–89.8%), and 91.7% for Gemini 3 (231/252; 95% CI, 87.6–94.5%).

Among the 148 questions originally accompanied by visual materials, the corresponding accuracies were 49.3% for GPT-4o (73/148; 95% CI, 41.4–57.3%), 55.4% for o3 (82/148; 95% CI, 47.4–63.2%), 55.4% for GPT-5 Thinking (82/148; 95% CI, 47.4–63.2%), and 61.5% for Gemini 3 (91/148; 95% CI, 53.5–68.9%). The observed visual-material minus text-only differences were −22.9, −24.4, −30.7, and −30.2 percentage points, respectively (Table 3). These descriptive comparisons are between distinct sets of questions, not within-question tests of the effect of adding visual information.

3.4. Mixed-Effects Logistic Regression and Model-by-Item-Type Interaction

A mixed-effects logistic regression model included model, item type, examination year, and model-by-item-type interactions as fixed effects, with examination question as a random intercept (estimated random-intercept standard deviation, 1.861). GPT-4o, text-only questions, and the 2018 examination served as the reference categories. The overall model-by-item-type interaction was statistically significant (likelihood-ratio χ² = 11.756, df = 3, p = 0.008), indicating that model contrasts differed according to item type (Table 4).

Among text-only items (the reference item type), the adjusted odds ratios (ORs) for correct responses relative to GPT-4o were 1.89 for o3 (95% CI, 1.13–3.15; p = 0.015), 3.66 for GPT-5 Thinking (95% CI, 2.09–6.42; p < 0.001), and 8.12 for Gemini 3 (95% CI, 4.24–15.56; p < 0.001). For GPT-4o (the reference model), visual-material questions were associated with lower odds of a correct response than text-only questions (adjusted OR = 0.22; 95% CI, 0.11–0.44; p < 0.001).

The model-by-item-type interaction ORs, each relative to GPT-4o, were 0.78 for o3 (95% CI, 0.36–1.68; p = 0.524), 0.40 for GPT-5 Thinking (95% CI, 0.18–0.89; p = 0.025), and 0.27 for Gemini 3 (95% CI, 0.11–0.64; p = 0.003). Thus, the fitted relative advantages of GPT-5 Thinking and Gemini 3 over GPT-4o were smaller for questions originally accompanied by visual materials than for text-only questions. Examination-year coefficients are reported in Table 4.

3.5. Adjusted Predicted Probabilities

Adjusted predicted probabilities from the fixed-effects component of the mixed-effects model, with the random intercept set to zero and estimates averaged equally across examination years, are displayed in Figure 2. For text-only questions, the predicted probabilities were 81.1% for GPT-4o (95% CI, 74.4–87.9%), 89.0% for o3 (95% CI, 84.2–93.8%), 94.0% for GPT-5 Thinking (95% CI, 90.9–97.1%), and 97.2% for Gemini 3 (95% CI, 95.4–99.0%).

For questions originally accompanied by visual materials, the corresponding predicted probabilities were 49.7% (95% CI, 37.3–62.2%), 59.0% (95% CI, 46.9–71.1%), 59.0% (95% CI, 46.9–71.1%), and 67.9% (95% CI, 56.8–79.0%). The resulting visual-material minus text-only differences were −31.4, −30.0, −35.0, and −29.3 percentage points for GPT-4o, o3, GPT-5 Thinking, and Gemini 3, respectively. These model-based estimates are conditional on the random intercept being zero and are distinct from the observed subgroup accuracies in Table 3.

3.6. Tables

Table 1. Characteristics of the 400 Examination Items.

Examination year	Total items	Text-only	Visual-material	Text-only (%)	Visual-material (%)
2018	80	58	22	72.5	27.5
2019	80	54	26	67.5	32.5
2020	80	48	32	60.0	40.0
2021	80	46	34	57.5	42.5
2022	80	46	34	57.5	42.5
Total	400	252	148	63.0	37.0

Note: All 80 questions from each of the 2018–2022 Japanese National Dental Technician Examinations were included. Percentages are within-year percentages.

Table 1. Characteristics of the 400 Examination Items.

Examination year	Total items	Text-only	Visual-material	Text-only (%)	Visual-material (%)
2018	80	58	22	72.5	27.5
2019	80	54	26	67.5	32.5
2020	80	48	32	60.0	40.0
2021	80	46	34	57.5	42.5
2022	80	46	34	57.5	42.5
Total	400	252	148	63.0	37.0

Note: All 80 questions from each of the 2018–2022 Japanese National Dental Technician Examinations were included. Percentages are within-year percentages.

Table 2. Overall Accuracy and Pairwise Matched Comparisons.

Model		Correct/N		Accuracy (%)		95% CI (%)
GPT-4o		255/400		63.7		58.9–68.3
o3		283/400		70.8		66.1–75.0
GPT-5 Thinking		299/400		74.8		70.3–78.8
Gemini 3		322/400		80.5		76.3–84.1
Matched pairwise comparisons
Model 1	Model 2	M1 only correct	M2 only correct	Difference (pp)	95% CI (pp)	Paired OR	Holm-adjusted p
GPT-4o	o3	48	76	-7.00	-12.42 to -1.58	0.63	0.0450
GPT-4o	GPT-5 Thinking	31	75	-11	-15.93 to -6.07	0.41	<0.001
GPT-4o	Gemini 3	25	92	-16.75	-21.80 to -11.70	0.27	<0.001
o3	GPT-5 Thinking	31	47	-4	-8.32 to 0.32	0.66	0.0888
o3	Gemini 3	28	67	-9.75	-14.44 to -5.06	0.42	<0.001
GPT-5 Thinking	Gemini 3	31	54	-5.75	-10.24 to -1.26	0.57	0.0450

Note: Cochran’s Q = 47.182, df = 3, p < 0.001. Pairwise exact McNemar tests with Holm adjustment. Differences are Model 1 minus Model 2.

Table 2. Overall Accuracy and Pairwise Matched Comparisons.

Model		Correct/N		Accuracy (%)		95% CI (%)
GPT-4o		255/400		63.7		58.9–68.3
o3		283/400		70.8		66.1–75.0
GPT-5 Thinking		299/400		74.8		70.3–78.8
Gemini 3		322/400		80.5		76.3–84.1
Matched pairwise comparisons
Model 1	Model 2	M1 only correct	M2 only correct	Difference (pp)	95% CI (pp)	Paired OR	Holm-adjusted p
GPT-4o	o3	48	76	-7.00	-12.42 to -1.58	0.63	0.0450
GPT-4o	GPT-5 Thinking	31	75	-11	-15.93 to -6.07	0.41	<0.001
GPT-4o	Gemini 3	25	92	-16.75	-21.80 to -11.70	0.27	<0.001
o3	GPT-5 Thinking	31	47	-4	-8.32 to 0.32	0.66	0.0888
o3	Gemini 3	28	67	-9.75	-14.44 to -5.06	0.42	<0.001
GPT-5 Thinking	Gemini 3	31	54	-5.75	-10.24 to -1.26	0.57	0.0450

Note: Cochran’s Q = 47.182, df = 3, p < 0.001. Pairwise exact McNemar tests with Holm adjustment. Differences are Model 1 minus Model 2.

Table 3. Accuracy According to Item Type.

Model	Text-only correct/N	Text-only accuracy (%)	Text-only 95% CI	Visual correct/N	Visual accuracy (%)	Visual 95% CI	Visual—Text (pp)
GPT-4o	182/252	72.2	66.4–77.4	73/148	49.3	41.4–57.3	-22.9
o3	201/252	79.8	74.4–84.3	82/148	55.4	47.4–63.2	-24.4
GPT-5 Thinking	217/252	86.1	81.3–89.8	82/148	55.4	47.4–63.2	-30.7
Gemini 3	231/252	91.7	87.6–94.5	91/148	61.5	53.5–68.9	-30.2

Note: Visual-material items were originally accompanied by figures, photographs, diagrams, or other visual materials. CI = confidence interval; pp = percentage points.

Table 3. Accuracy According to Item Type.

Model	Text-only correct/N	Text-only accuracy (%)	Text-only 95% CI	Visual correct/N	Visual accuracy (%)	Visual 95% CI	Visual—Text (pp)
GPT-4o	182/252	72.2	66.4–77.4	73/148	49.3	41.4–57.3	-22.9
o3	201/252	79.8	74.4–84.3	82/148	55.4	47.4–63.2	-24.4
GPT-5 Thinking	217/252	86.1	81.3–89.8	82/148	55.4	47.4–63.2	-30.7
Gemini 3	231/252	91.7	87.6–94.5	91/148	61.5	53.5–68.9	-30.2

Note: Visual-material items were originally accompanied by figures, photographs, diagrams, or other visual materials. CI = confidence interval; pp = percentage points.

Table 4. Mixed-Effects Logistic Regression for Correct Responses.

Term	β	SE	Adjusted OR	Lower 95% CI	Upper 95% CI	p
Intercept	2.057	0.349	7.82	3.95	15.50	<0.001
Model: o3	0.634	0.262	1.89	1.13	3.15	0.015
Model: GPT-5 Thinking	1.298	0.286	3.66	2.09	6.42	<0.001
Model: Gemini 3	2.094	0.332	8.12	4.24	15.56	<0.001
Visual-material item	-1.498	0.345	0.22	0.11	0.44	<0.001
Year: 2019	-0.867	0.404	0.42	0.19	0.93	0.032
Year: 2020	-0.461	0.406	0.63	0.28	1.40	0.257
Year: 2021	-0.781	0.404	0.46	0.21	1.01	0.053
Year: 2022	-0.729	0.404	0.48	0.22	1.07	0.071
o3 × Visual	-0.250	0.392	0.78	0.36	1.68	0.524
GPT-5 Thinking × Visual	-0.914	0.408	0.40	0.18	0.89	0.025
Gemini 3 × Visual	-1.317	0.441	0.27	0.11	0.64	0.003

Note: Reference categories were GPT-4o, text-only items, and 2018. Random intercept: examination question. Omnibus model-by-item-type interaction: LR χ² = 11.756, df = 3, p = 0.0083. OR = odds ratio; CI = confidence interval.

Table 4. Mixed-Effects Logistic Regression for Correct Responses.

Term	β	SE	Adjusted OR	Lower 95% CI	Upper 95% CI	p
Intercept	2.057	0.349	7.82	3.95	15.50	<0.001
Model: o3	0.634	0.262	1.89	1.13	3.15	0.015
Model: GPT-5 Thinking	1.298	0.286	3.66	2.09	6.42	<0.001
Model: Gemini 3	2.094	0.332	8.12	4.24	15.56	<0.001
Visual-material item	-1.498	0.345	0.22	0.11	0.44	<0.001
Year: 2019	-0.867	0.404	0.42	0.19	0.93	0.032
Year: 2020	-0.461	0.406	0.63	0.28	1.40	0.257
Year: 2021	-0.781	0.404	0.46	0.21	1.01	0.053
Year: 2022	-0.729	0.404	0.48	0.22	1.07	0.071
o3 × Visual	-0.250	0.392	0.78	0.36	1.68	0.524
GPT-5 Thinking × Visual	-0.914	0.408	0.40	0.18	0.89	0.025
Gemini 3 × Visual	-1.317	0.441	0.27	0.11	0.64	0.003

Note: Reference categories were GPT-4o, text-only items, and 2018. Random intercept: examination question. Omnibus model-by-item-type interaction: LR χ² = 11.756, df = 3, p = 0.0083. OR = odds ratio; CI = confidence interval.

Supplementary Table. Accuracy by Examination Year and Model.

Descriptive year-specific results; these are not the primary matched comparison.

Year	Model	Correct	N	Accuracy (%)	Lower 95% CI (%)	Upper 95% CI (%)
2018	GPT-4o	59	80	73.8	63.2	82.1
2018	o3	63	80	78.8	68.6	86.3
2018	GPT-5 Thinking	66	80	82.5	72.7	89.3
2018	Gemini 3	72	80	90.0	81.5	94.8
2019	GPT-4o	48	80	60.0	49.0	70.0
2019	o3	53	80	66.2	55.4	75.7
2019	GPT-5 Thinking	61	80	76.2	65.9	84.2
2019	Gemini 3	63	80	78.8	68.6	86.3
2020	GPT-4o	52	80	65.0	54.1	74.5
2020	o3	53	80	66.2	55.4	75.7
2020	GPT-5 Thinking	65	80	81.2	71.3	88.3
2020	Gemini 3	64	80	80.0	70.0	87.3
2021	GPT-4o	52	80	65.0	54.1	74.5
2021	o3	55	80	68.8	57.9	77.8
2021	GPT-5 Thinking	50	80	62.5	51.5	72.3
2021	Gemini 3	62	80	77.5	67.2	85.3
2022	GPT-4o	44	80	55.0	44.1	65.4
2022	o3	59	80	73.8	63.2	82.1
2022	GPT-5 Thinking	57	80	71.2	60.5	80.0
2022	Gemini 3	61	80	76.2	65.9	84.2
Note: Each year included 80 examination items. Accuracy was calculated as the proportion of correct responses; 95% confidence intervals were calculated using the Wilson method. Results are descriptive and should not be interpreted as longitudinal changes in model performance.

3.7. Figuress

Figure 1. Overall accuracy of the four large language models on all 400 examination items.

Figure 1. Overall accuracy of the four large language models on all 400 examination items.

Figure 2. Adjusted predicted probabilities of a correct response according to model and item type.

Figure 2. Adjusted predicted probabilities of a correct response according to model and item type.

4. Discussion

4.1. Principal Findings

This matched item-level analysis evaluated four large language models (LLMs)—GPT-4o, o3, GPT-5 Thinking, and Gemini 3—on the same 400 questions from the 2018–2022 Japanese National Dental Technician Examinations. Three findings emerged. First, model performance differed significantly, and Gemini 3 achieved the highest overall accuracy (80.5%), followed by GPT-5 Thinking (74.8%), o3 (70.8%), and GPT-4o (63.8%). Although Gemini 3 outperformed each comparator after multiplicity adjustment, the difference between GPT-5 Thinking and o3 was not statistically significant. Second, all four models answered fewer questions correctly when the original examination item contained visual materials. Third, the model-by-item-type interaction was significant (likelihood-ratio χ² = 11.756, df = 3, p = 0.008), indicating that relative performance differences were not constant across examination formats (Table 4; Figure 2). The latter finding is particularly relevant to LLM benchmarking because a single aggregate accuracy measure does not capture how model comparisons vary across task formats.

4.2. Relationship to Previous Studies and Contribution of the Present Analysis

Prior studies have established the feasibility of assessing LLMs using dental licensing examinations and have repeatedly reported lower performance on visually accompanied questions [14,15,16,17,18]. An earlier evaluation of the 2018–2022 Japanese National Dental Technician Examination examined ChatGPT-4o alone [17], whereas another study evaluated GPT-4o, o1, and Claude 3.5 Sonnet using 240 questions from 2022–2024 [18]. Our previous Computers article compared GPT-4o, o3, and GPT-5T on 399 Japanese dental board-style questions and explored differences between item formats using stratified comparisons and heterogeneity testing [19]. Recent studies involving Gemini-family and other advanced models further demonstrate the relevance of cross-model dental examination benchmarks [20,21]. Accordingly, the use of dental examination questions, the inclusion of newer LLMs, and the observation of lower accuracy on visually accompanied questions are not, individually, novel findings.

The present work extends this literature in three defined ways. All four models were evaluated on an identical and complete five-year set of 400 items, allowing matched comparisons without differences in item selection across models. Incorporating Gemini 3 adds a cross-developer comparison within that common dataset. More importantly, question-level mixed-effects logistic regression explicitly tests whether the association between model and correctness differs by original item format, while accounting for examination year and the correlation among four model responses to each item. Responses in the present study were collected separately from those in our previous three-model report [19]; therefore, numerical differences between publications should not be interpreted as longitudinal changes in model capability. The methodological contribution lies in estimating format-dependent differences in comparative performance within a matched, clustered design rather than in claiming a new examination benchmark or an entirely new analytic technique.

4.3. Interpretation of Format-Dependent Model Differences

The interaction analysis offers a more specific interpretation than separate comparisons of subgroup accuracy. Among text-only questions, accuracy ranged from 72.2% for GPT-4o to 91.7% for Gemini 3, whereas among questions originally accompanied by visual materials it ranged from 49.3% to 61.5% (Table 3). In the mixed-effects model, the interactions for GPT-5 Thinking and Gemini 3 relative to GPT-4o were below unity (odds ratios of 0.40 and 0.27, respectively), showing that their relative odds advantages over GPT-4o were attenuated in the visually accompanied subset (Table 4). The corresponding interaction for o3 was not statistically significant; this does not establish the absence of effect modification or equivalence between models. Nor do the negative interactions imply that GPT-5 Thinking or Gemini 3 performed worse than GPT-4o on visually accompanied questions.

This distinction matters because overall accuracy, subgroup accuracy, and interaction effects answer different questions. The overall benchmark ranks models across the observed item mixture; subgroup estimates describe accuracy within each format; and the interaction evaluates whether the between-model contrast changes by format. The results therefore support format-aware reporting in comparisons of LLMs, particularly when a benchmark combines substantially different types of input. However, the present analysis is observational with respect to item type: visual materials were not experimentally added to or removed from otherwise identical questions. Differences between the text-only and visually accompanied subsets may also reflect topic distribution, question difficulty, spatial reasoning requirements, or response format. Consequently, the interactions do not provide a causal estimate of the effect of visual input or a direct measure of isolated image-recognition ability.

4.4. Implications for LLM Evaluation and Dental Technology Education

The results suggest that reporting only a pooled examination score may give an incomplete account of performance for domain-specific LLM applications. A model with strong overall accuracy may retain appreciable limitations on questions involving figures, photographs, dental prostheses, instruments, or spatial relationships. Researchers designing evaluation sets should therefore consider reporting the composition of item formats, matched performance differences, uncertainty intervals, and formal tests of effect modification when the study design permits. The question-level approach used here is especially useful when every model answers the same items, because it acknowledges correlated outcomes rather than treating model responses as independent observations.

In dental technology education, LLMs may assist with theoretical review, explanations, and formative question practice, as suggested by the broader dental education literature [7,9,12]. Nevertheless, dental technology also requires practical skills, morphological interpretation, prosthetic design, and understanding of laboratory procedures. Accuracy on multiple-choice questions cannot establish practical competence or the reliability of explanations. Instructors and learners should therefore verify model-generated answers, particularly for visually accompanied questions, rather than using benchmark accuracy as sufficient evidence for independent educational or technical decision-making.

4.5. Limitations

Several limitations should be considered. First, the complete 400-item dataset represents one national examination system and the historical 2018–2022 period; the results may not generalize to other examinations, languages, subject distributions, or subsequently updated model interfaces. Public availability of the questions also means that potential exposure during model training cannot be excluded. Second, the binary classification of original visual-material presence did not separate photographs, diagrams, instruments, prosthetic designs, or other formats, nor did it establish whether the visual material was indispensable for answering a given item. The interface platforms could also have applied different image-processing procedures, and equivalent pixel-level presentation was not verified.

Third, responses were obtained using consumer web interfaces rather than fixed-parameter APIs. Platform-level settings were not controlled, individual questions were not always initiated in fresh conversations, and residual context from earlier items could have affected subsequent responses. Only the first answer was retained, preventing estimation of within-model response variability. Fourth, the regression accounted for examination year and unobserved question-level heterogeneity but did not adjust for dental subject, cognitive complexity, required number of answers, or visual-material subtype. Examination-year coefficients should not be interpreted as changes in model performance over time because the question sets differed across years. Finally, evaluation was restricted to correctness against official answer keys: it did not assess reasoning quality, calibration, hallucinations, student learning gains, or practical performance. Model-specific interaction coefficients were reference-coded against GPT-4o and do not directly test every pairwise contrast among the four models.

4.6. Future Research

Future benchmarking studies should document exact model versions, collection dates, prompts, conversation-reset procedures, browsing availability, and presentation conditions to improve reproducibility. Repeated evaluations across independently initialized sessions could quantify response stability. More granular coding of visual material type and the necessity of image interpretation, together with adjustment for subject area and question complexity, would help identify possible sources of format-dependent differences. Where feasible, experimentally controlled text-only and text-plus-image variants of the same items would provide a stronger assessment of the contribution of visual information. Evaluations using newly developed or prospective items and qualitative error analyses could further clarify generalization, failure modes, and the practical utility of LLM assistance in dental education.

5. Conclusions

This matched item-level analysis of 400 Japanese National Dental Technician Examination questions identified significant differences in answer accuracy among GPT-4o, o3, GPT-5 Thinking, and Gemini 3. Gemini 3 achieved the highest overall accuracy (80.5%), while all four models performed less accurately on questions originally accompanied by visual materials than on text-only questions. Importantly, the significant model-by-item-type interaction (likelihood-ratio χ² = 11.756, p = 0.008) indicated that differences between models were not uniform across examination formats. In particular, the relative advantages of GPT-5 Thinking and Gemini 3 over GPT-4o were smaller for visual-material questions.

By combining a complete five-year matched comparison with question-level mixed-effects analysis, this study demonstrates the value of evaluating model performance by item format rather than relying solely on aggregate benchmark accuracy. Because the two item types comprised different examination questions, these findings do not establish a causal effect of visual information or isolate image-interpretation ability. Future benchmarking should incorporate standardized input conditions, repeated evaluations, and more detailed visual-item classifications. In dental technology education, LLM-generated answers should undergo expert verification, particularly for visually informed tasks.