Benchmarking Large Language Models on Long-Tail Plant Taxonomic Knowledge with PTTB-600

1. Introduction

Plant taxonomy underpins biodiversity description, identification, monitoring, and conservation. Its knowledge base combines phylogenetic relationships, morphological terminology, identification keys, nomenclatural rules, and continuously revised taxonomic treatments [1,2,3,4,5,6,7,8,9,10]. Unlike domains dominated by a compact set of frequently repeated concepts, plant taxonomy contains a small, commonly taught core and a much larger tail of infrequently encountered family and genus names, rare character combinations, synonyms, and source-dependent circumscriptions. We refer to this distribution as long-tail taxonomic knowledge.

This structure matters in both biodiversity practice and undergraduate teaching. Students must connect names to observable characters, place taxa within a phylogenetic framework, and distinguish a current classification from historical or alternative treatments. Persistent difficulty with plant awareness, classification concepts, and taxon recognition has been documented among students and instructors [11,12,13,14]. A model that answers common terminology correctly may therefore still fail when it must generate a rare genus name, reconcile alternative nomenclatural treatments, or follow a fine-grained key.

Large language models (LLMs) are now used for lesson preparation, explanation generation, question generation, classroom interaction, and self-directed learning [15,16,17,18,19,20]. Aggregate performance alone, however, does not reveal whether residual errors affect routine course content or are concentrated in a small set of low-frequency tasks requiring expert verification.

General-purpose benchmarks such as MMLU, OpenCompass, MT-Bench, and Chatbot Arena compare broad knowledge and reasoning ability [21,22,23,24], but they provide limited resolution for low-frequency facts within a single taxonomic domain. Studies of parametric knowledge and specialist question answering show that rare entities, fine-grained facts, and complex open-ended questions remain difficult [25,26,27,28]. Hallucination research likewise shows that fluent outputs can be factually unsupported, although reliability varies by task, model, and evaluation design [29,30,31,32]. Course-level evidence is thus needed to separate routine taxonomic competence from errors on long-tail tasks.

We developed the Plant Taxonomy Teaching Benchmark-600 (PTTB-600) and used it to evaluate 31 LLMs or run modes under text-only, closed-book conditions. The benchmark separates general knowledge, ordinary specialized knowledge, and long-tail fill-in tasks. We asked three questions: (1) how accurately current models answer representative plant taxonomy questions; (2) which knowledge strata and answer formats account for differences among models; and (3) how thinking mode, model scale, and published API price relate to performance. The resulting evidence delineates where current LLMs can support routine taxonomic teaching and where authoritative verification remains necessary.

2. Materials and Methods

2.1. Benchmark Design

PTTB-600 was designed according to four principles: the items sampled authentic plant taxonomic knowledge from general concepts to fine-grained taxon identification; each item had a prespecified answer or accepted-answer set under an explicit taxonomic treatment; models answered from the item text without supplied source passages or conversational context; and the question bank, answer key, responses, and item-level scores were retained for audit and reuse.

The 600 items were partitioned a priori into 200 general questions (PTB-G), 300 ordinary specialized questions (PTB-D), and 100 long-tail fill-in questions (PTB-L; Table 1). PTB-G covered phylogeny, nomenclatural rules, and morphological terminology. PTB-D covered family and genus recognition, comparisons among commonly taught taxa, and diagnostic-character judgments. PTB-L items provided no candidate answers and required direct generation of infrequently encountered genus names, reconciliation of alternative nomenclatural treatments, or interpretation of fine identification-key characters. PTB-L was designed as a targeted stress test of the boundary of model knowledge. Supplementary Table S1 provides metadata and accepted answers for all items, together with item-level source notes and answer evidence for the 100 long-tail items.

The answer key was compiled with reference to standard plant systematics texts [1,2,3], APG IV [4], Flora of China [5], the Madrid Code [6], and continuously updated taxonomic resources including Plants of the World Online, World Flora Online, the International Plant Names Index, and the Angiosperm Phylogeny Website [7,8,9,10]. For source-dependent items, the record specified the adopted treatment, context, primary answer, and accepted alternatives. Items with unresolved ambiguity or answers that could not be assessed from the provided text were revised, replaced, or excluded. The question bank and answer key were drafted by J.H. and independently reviewed by J.X., H.Q., and L.X., all of whom hold doctorates in plant taxonomy and have complementary expertise in taxonomy, systematics, and university teaching. Validation covered question wording, accepted answers, adopted taxonomic treatments, and supporting sources. All question stems and answer instructions were written in Chinese; Latin taxon names were retained or requested where appropriate. Item-level source notes were complete for PTB-L, whereas PTB-G and PTB-D were documented at the level of source framework, source category, and accepted answer.

2.2. Models, Run Modes, and Prompting

The analysis included 31 LLMs or run modes evaluated between 4 and 7 May 2026. The sample covered China-based and international systems available during the test period, including models from Alibaba, DeepSeek, Moonshot AI, Zhipu AI, ByteDance, Xiaomi, MiniMax, OpenAI, Google, and xAI. Flagship, lightweight, and earlier versions were included where available to characterize differences among configurations. The run registry records the display name, model identifier, access route, run date, sampling settings, output limit, reasoning-mode control, and completion status for each analyzed run.

Run modes were classified operationally as thinking or non-thinking/default according to the supplier’s route label or request setting. This classification enabled within-family comparisons without assuming equivalent internal mechanisms across suppliers. All models received the same Chinese question text and answer instruction. No textbook passage, answer key, web search, retrieval system, or prior dialogue was supplied.

2.3. Scoring and Statistical Analysis

Responses were scored against the prespecified answer key. Single- and multiple-choice responses were required to match the correct option set exactly; omitted or additional options were scored as incorrect. Fill-in responses were normalized for capitalization, spacing, and common punctuation and then matched to the accepted Latin names. A format-tolerant sensitivity analysis also counted a response as correct if it contained exactly one accepted Latin name with brief explanatory text. This alternative rule did not change the score of any model (Supplementary Table S2).

Each item received a binary score. Scores and accuracies were calculated for all 600 items and for the general, ordinary specialized, long-tail, single-choice, multiple-choice, and fill-in subsets. Overall rank was based on the number of correct answers among all 600 items. Nonparametric 95% confidence intervals were estimated by resampling items with replacement 2000 times using a fixed random seed; these intervals quantify item-sampling variation within the benchmark rather than repeated-run variation. Differences between adjacent ranked models were assessed with paired item-bootstrap confidence intervals and two-sided exact McNemar tests. Holm adjustment was applied across the 30 adjacent-rank tests (Supplementary Table S3).

Item-level error frequency was summarized as the number of retained models or modes that answered each item incorrectly. Six representative items among those most frequently missed are shown in Table 2; complete item-level error counts and representative error classes are provided in Supplementary Tables S4 and S5. Eleven model families with both thinking and non-thinking/default results were compared within family. For each pair, we calculated differences in overall score and in knowledge-stratum and answer-format accuracies (Supplementary Table S6).

For nine open-weight models or versions with publicly documented total parameter counts, the association between parameter count and PTTB-600 accuracy was evaluated with Spearman rank correlation. Proprietary systems without disclosed parameter counts, including Gemini, GPT, and Grok, were not eligible for this analysis. The two-sided p value was obtained from the exact permutation distribution. Pearson correlation was retained as a sensitivity result. The analysis was interpreted as an exploratory association because model family, training data, post-training, and run configuration varied together.

2.4. Published Price and Use-Scenario Analyses

API pricing data were recorded as of 7 May 2026. Official prices were used where available; otherwise, the contemporaneous price data from the evaluated access platform were used and identified as such in Supplementary Table S7. The unweighted sum of the listed prices for one million input and one million output tokens was used as a common descriptive price index. We also estimated the cost of one complete benchmark run from recorded input, cached-input, output, and available reasoning-token counts. These quantities characterize the evaluated runs; deployment costs will vary with course settings. Supplementary Table S7 provides the price calculations. Supplementary Table S8 reports exploratory use-scenario rankings under prespecified weighted combinations of benchmark metrics, and Supplementary Table S9 provides the complete run registry and operational mode definitions.

3. Results

3.1. Overall Performance and Adjacent-Rank Differences

Overall scores ranged from 410 to 587 correct answers, a difference of 177 correct answers (Figure 1; Supplementary Table S2). Gemini 3.1 Pro Preview ranked first with 587/600 (97.83%; 95% CI, 96.7-98.8%). Grok 4-20 reasoning, GPT-5.5, DeepSeek V4 Pro (thinking), Qwen3.6 Max Preview (thinking), and Seed 2.0 Pro (thinking) scored 556-566/600 (92.67-94.33%). The next group, scoring 525-542/600, included DeepSeek V4 Flash (thinking), Seed 2.0 Lite (thinking), Gemini 3.1 Flash Lite Preview, GLM-5.1 (thinking), DeepSeek V4 Pro (non-thinking), Kimi K2.6 (thinking), and Seed 2.0 Pro (non-thinking).

The difference between Gemini 3.1 Pro Preview and second-ranked Grok 4-20 reasoning was 3.50 percentage points (95% CI, 1.67-5.33; exact p = 0.0002; Holm-adjusted p = 0.0056). Adjacent differences among ranks two through six were 0-1.00 percentage points, and none was significant after Holm correction (all adjusted p = 1.00). The 2.33-point difference between sixth-ranked Seed 2.0 Pro (thinking) and seventh-ranked DeepSeek V4 Flash (thinking) had an unadjusted p = 0.038 but an adjusted p = 1.00. After multiplicity correction, the data distinguished the first-ranked model from the second but left the ordering within the closely spaced group below it statistically unresolved.

Among the nine models with publicly documented parameter counts, total parameter count was positively associated with PTTB-600 accuracy (Spearman rho = 0.69, exact permutation p = 0.044; Figure 2). Models with similar parameter counts nevertheless differed in accuracy, and the 284-billion-parameter DeepSeek V4 Flash (thinking) scored 542/600. Thus, parameter count was associated with rank order in this sample, although models of similar size still differed in accuracy.

3.2. Knowledge Strata, Answer Formats, and Recurrent Errors

General and ordinary specialized questions approached ceiling performance among the leading models, whereas long-tail fill-in questions provided substantially greater discrimination (Figure 3). On the 200 general questions, 21 of 31 models or modes achieved 200/200 and all others scored at least 197/200. The six highest-scoring models answered 291-297 of 300 ordinary specialized questions correctly (97.0-99.0%) but only 63-90 of 100 long-tail questions (63.0-90.0%). Gemini 3.1 Pro Preview scored 297/300 and 90/100 on these two subsets, respectively; the corresponding scores were 295 and 71 for Grok 4-20 reasoning, 294 and 69 for GPT-5.5, 292 and 65 for DeepSeek V4 Pro (thinking), 291 and 65 for Qwen3.6 Max Preview (thinking), and 293 and 63 for Seed 2.0 Pro (thinking).

Choice questions also showed limited separation. Ten models or modes obtained full marks on the 253 single-choice items, and 24 obtained full marks on the 57 multiple-choice items. Combined choice scores ranged from 299 to 310/310. Fill-in scores ranged from 111 to 277/290, a difference of 166 items, and no model achieved full marks. The remaining between-model differences were therefore concentrated in the fill-in questions.

All 20 most frequently missed items were fill-in questions. They typically required direct generation of an uncommon genus or integration of several diagnostic characters. Six representative frequently missed items concerned Apiaceae, Lamiaceae, Orchidaceae, and Lauraceae (Table 2). Some models identified the correct family or a closely related genus but failed to associate persistent calyx teeth and nearly equal filiform fruit ribs with Ostericum, or two stamens and linear staminodes with Mosla. Other errors reflected historical names or source-dependent treatments. The format-tolerant analysis left all model scores unchanged, so these errors reflected substantive answer differences rather than incidental explanatory prose.

Table 2. Accepted answers, representative incorrect outputs, and diagnostic evidence for six representative items among those most frequently missed. Errors (n) denotes the number of incorrect responses among the 31 retained models or run modes.

Item	Accepted genus	Representative incorrect outputs	Errors (n)	Assessment basis	Key characters
PTB-L-0017	Ostericum	Oenanthe; Chaerophyllum; Chamaesium	31	Flora of China generic key for Apiaceae	Persistent calyx teeth; nearly equal filiform fruit ribs
PTB-L-0027	Mosla	Salvia	30	Flora of China generic key for Lamiaceae	Two stamens; linear staminodes
PTB-L-0066	Liparis	Cymbidium; Dendrobium	30	Flora of China generic key for Orchidaceae	Resupinate flower; arched column; four pollinia
PTB-D-0183	Beilschmiedia	Cinnamomum; Cryptocarya	29	Flora of China generic key for Lauraceae	Two-celled anthers; six or nine stamens; lateral locules
PTB-L-0002	Laurus	Litsea; Dodecadenia	29	Flora of China generic key for Lauraceae	Dimerous flowers; 12 stamens in male flowers; four staminodes in female flowers
PTB-L-0010	Syndiclis	Beilschmiedia; Potameia; Sinopora	29	Flora of China generic key for Lauraceae	Four tepals; four fertile stamens; fruit not enclosed by perianth tube

Table 2. Accepted answers, representative incorrect outputs, and diagnostic evidence for six representative items among those most frequently missed. Errors (n) denotes the number of incorrect responses among the 31 retained models or run modes.

Item	Accepted genus	Representative incorrect outputs	Errors (n)	Assessment basis	Key characters
PTB-L-0017	Ostericum	Oenanthe; Chaerophyllum; Chamaesium	31	Flora of China generic key for Apiaceae	Persistent calyx teeth; nearly equal filiform fruit ribs
PTB-L-0027	Mosla	Salvia	30	Flora of China generic key for Lamiaceae	Two stamens; linear staminodes
PTB-L-0066	Liparis	Cymbidium; Dendrobium	30	Flora of China generic key for Orchidaceae	Resupinate flower; arched column; four pollinia
PTB-D-0183	Beilschmiedia	Cinnamomum; Cryptocarya	29	Flora of China generic key for Lauraceae	Two-celled anthers; six or nine stamens; lateral locules
PTB-L-0002	Laurus	Litsea; Dodecadenia	29	Flora of China generic key for Lauraceae	Dimerous flowers; 12 stamens in male flowers; four staminodes in female flowers
PTB-L-0010	Syndiclis	Beilschmiedia; Potameia; Sinopora	29	Flora of China generic key for Lauraceae	Four tepals; four fertile stamens; fruit not enclosed by perianth tube

3.3. Performance Differences Associated with Thinking Mode

Thinking-mode runs outscored their non-thinking/default counterparts in all 11 within-family pairs by 20-65 correct answers, equivalent to 3.33-10.83 percentage points (Figure 4). The largest paired difference was observed for Grok 4-20 (65 answers). The corresponding differences were 51, 47, 45, and 43 answers for Seed 2.0 Lite, DeepSeek V4 Flash, Qwen3.6 Flash, and Qwen3.6 Max Preview, respectively, and 22 and 20 answers for DeepSeek V4 Pro and MiMo V2.5.

The paired differences were strongly task-specific. Differences on general questions were 0.00-1.00 percentage points, and eight pairs showed no change. Differences on the combined PTB-D and PTB-L subsets were 4.75-16.25 percentage points and were positive in every pair. Choice-question differences were 0.00-1.94 points, whereas fill-in differences were 5.86-22.07 points. The largest paired differences therefore occurred in tasks requiring the integration of multiple clues and direct name generation.

3.4. Accuracy and Published API Price

Accuracy and the published price index showed no monotonic relationship (Figure 5; Supplementary Table S7). The sum of the listed per-million input and output prices ranged from USD 0.420 to 35.000. Gemini 3.1 Pro Preview, GPT-5.5, and Qwen3.6 Max Preview (thinking) scored 587, 563, and 556, with price indices of USD 14.000, 35.000, and 9.100, respectively. Grok 4-20 reasoning, DeepSeek V4 Pro (thinking), and Seed 2.0 Pro (thinking) also belonged to the 556-566 high-scoring group, with price indices of USD 3.750, 5.220, and 2.840.

Estimated costs for one complete evaluation were USD 2.36 for Gemini 3.1 Pro Preview, 0.17 for Grok 4-20 reasoning, 3.82 for GPT-5.5, 0.80 for DeepSeek V4 Pro, and 0.44 for Seed 2.0 Pro. Differences in output length, cached-input charges, and reasoning-token accounting altered the cost rankings. The resulting ranking reflects the evaluated configurations and token accounting.

4. Discussion

4.1. High Reliability on Routine Taxonomic Knowledge

The central result is the clear separation between routine undergraduate plant taxonomy content and its long tail. Near-ceiling scores on general and ordinary specialized questions show that aggregate rank differences arose primarily from the long-tail portion of the benchmark. The leading models were highly accurate on common concepts, family and genus recognition, and standard diagnostic judgments; the residual spread arose mainly from low-frequency, open-generation items (Figure 1 and Figure 3).

This result supports routine use of top-performing LLMs for concept explanation, lesson preparation, organization of teaching materials, routine practice questions, and classroom interaction. For model selection, Gemini 3.1 Pro Preview was clearly separated from the second-ranked model after multiplicity correction, whereas ranks two through six remained statistically unresolved after Holm correction. Small rank differences within that group should be considered together with access, institutional compliance, response speed, and deployment conditions.

Performance differences associated with thinking mode were concentrated in tasks that had not reached ceiling performance. General questions showed little separation, whereas specialized and fill-in tasks showed substantial paired differences (Figure 4). Newer lightweight models also outperformed some earlier high-capacity versions, indicating that course-level validation should be repeated as model generations change.

4.2. The Boundary of Factual Error in Long-Tail Taxonomic Tasks

Hallucination is a major concern when LLMs are used in specialist domains [29,30,31]. This study assessed hallucination operationally as factual error against a prespecified answer key. Such errors were uncommon across the general and ordinary specialized tasks covered by PTTB-600 and were concentrated in 100 deliberately difficult long-tail fill-in questions. Error risk rose specifically when models had to generate an uncommon name without candidates, separate close genera using fine character combinations, or resolve a source-dependent nomenclatural treatment (Figure 3).

The positive association between parameter count and overall accuracy among nine documented models indicates that larger models tended to score higher within this sample (Figure 2). Routine subsets were already near their ceiling, so much of the remaining between-model variation came from rare-name and open-generation items. Parameter count is best interpreted as a correlate of performance here because model family, training data, architecture, post-training, and run configuration varied together. Prior evidence also shows that larger or more instructable models are not uniformly more reliable across contexts [32].

These findings define a practical boundary for verification. Most routine content can be used with regular instructor oversight, whereas rare taxa, alternative nomenclatural treatments, formal identifications, and answer keys intended for release require explicit verification. This workflow preserves the efficiency of LLM assistance without treating a fluent taxonomic name as evidence of correctness.

4.3. Model Selection for Taxonomic Teaching and Knowledge Work

For lesson preparation, teaching-material development, and verification of difficult content, model selection should prioritize factual accuracy on course-relevant tasks. Gemini 3.1 Pro Preview led the overall, ordinary specialized, and long-tail subsets and was the highest-performing configuration under the conditions tested. Where access, local deployment, or institutional compliance constrains its use, DeepSeek V4 Pro and Seed 2.0 Pro provide strong China-based alternatives. Genus-level placement, synonymy, identification-key decisions, and formal answer keys should be checked against maintained taxonomic resources regardless of model.

For classroom demonstrations and immediate question answering, response speed should be considered together with opportunities for instructor correction. High-performing configurations can support terminology, general concepts, comparisons among common taxa, and adaptation of specialist prose for teaching. Complex identification clues and rare taxa call for a course-validated, high-performing configuration and prompt verification against authoritative sources. Student-facing practice systems should distinguish explanatory feedback from formal taxonomic authority and link long-tail questions to course materials or continuously updated databases.

Published price and accuracy showed no fixed relationship (Figure 5). Several models combined high scores with moderate listed prices, but actual expenditure depended on output length, caching, and reasoning-token accounting. Model choice for a real course should therefore be based on task-specific accuracy, access, latency, scale of use, and local compliance.

5. Conclusions

PTTB-600 shows that current leading LLMs answer the general and ordinary specialized content of undergraduate plant taxonomy with low factual error rates. The remaining differences among models are concentrated in rare genus names, open generation without candidate answers, alternative nomenclatural treatments, and fine identification-key characters. Gemini 3.1 Pro Preview ranked first, while ranks two through six formed a closely spaced high-scoring cluster without significant adjacent differences after multiplicity correction. Paired differences associated with thinking mode were concentrated in specialized and fill-in tasks. Leading LLMs can therefore support routine taxonomic teaching and knowledge organization, provided that formal identifications, nomenclatural decisions, and long-tail answers are verified against authoritative sources.