Asking for the Whole Plan: Prompt Design Moves Language Models’ Scores on a Clinical-Safety Benchmark More Than Model Choice: An Exploratory Study on the NOHARM v2 Public Cases

1. Introduction

A consultation answer can be correct in everything it says and still fail the patient through what it leaves out. NOHARM measures this. It compares a model’s management plan with specialists’ ratings of the actions a case calls for and those it should avoid, and weights each error by the harm it could cause [1]. Most severe errors are omissions, and recall trails precision for nearly every model. The public leaderboard scores models on the bare case, with an empty prompt.

If coverage depends on how a model is asked, a bare score measures a model’s default habits as well as its clinical judgement. NOHARM’s authors tested part of this: priming prompts that asked for completeness, capped at 200 or 500 words, raised generalist models’ scores, although clinical retrieval tools stayed ahead [1]. Two questions remain. What does an instruction do that asks for a complete, structured plan without a word cap? And is the gain real coverage, or only length, given that the benchmark’s precision ignores actions its rubric does not list?

We asked four questions on the public cases:

1.

How much does a structured instruction move each model, and does the gain depend on how much the model writes unprompted?

2.

Which part of the instruction carries the gain?

3.

Is the gain more than length, and are the added actions safe?

4.

Can one instructed call match systems built from several calls?

All results are development evidence: public cases and a locally run AI evaluator. None is an organiser-scored leaderboard number, and none establishes clinical safety.

2. Related Work

How a model is asked is known to move benchmark scores. Systematic prompting lifted a generalist model past fine-tuned medical systems on multiple-choice benchmarks [2]; instruction wording changes clinical language-processing results [3]; and meaning-preserving prompt changes reorder models on general leaderboards [4,5,6]. Our instruction differs: it changes what the answer is asked to contain, not how the same request is worded. NOHARM [1] is the closest antecedent, with its omission-dominated errors and its capped priming prompts. On other instruments, general models led clinical retrieval tools [7], although those benchmarks and settings have been disputed [8]. Structured management reasoning has been built into clinical agents [9], and models in simulated clinical workflows skip guideline-required steps [10].

Two threats apply to any rubric-graded gain from longer answers. First, AI evaluators favour longer and self-generated text [11,12,13,14,15], although a length-matched check on HealthBench found the effect partial [16]. Second, evaluators detect what is absent poorly [17], as models do [18]. Other routes to better recall are combining several models [19], sampling one model several times [20] and directed completion [21]. Self-revision without an outside signal is unreliable [22], model errors are correlated [23], and panels of diverse evaluators reduce single-evaluator bias [24].

3. Methods

3.1. Benchmark

NOHARM’s cases come from real primary-care-to-specialist electronic consultations at one academic medical centre [1]. The public subset, released under CC BY 4.0, has 30 base cases, three from each of ten specialties. Each appears in its original form and in ten versions that change one detail (age, sex, a comorbidity, a medication, a laboratory value and others), giving 330 items. Versions of a case are related, not independent. Each case has a rubric of candidate actions rated by specialists on a 1–9 appropriateness scale: 1,128 options in all, of which 522 are rated harmful (1–3; 79 at 1), 408 equivocal and 198 appropriate (7–9; 23 at 9). By category the options are diagnostic (424), medication (374), counselling (174), procedure (94) and follow-up (55). We ran the NOHARM v2 kit unchanged except for the evaluator model (Appendix B). The organisers’ held-out cases and patient-level data were not used.

3.2. Models

Twelve models answered the bare case: Claude Sonnet 5.5, Sonnet 5, Opus 5.5 and Haiku 4.5; GPT-5.6 terra and Luna, GPT-6 Astra and GPT-6.1 Sol; Grok 4.7; and three freely available models (Muse Spark 1.3, Nemotron 3 Ultra and Big Pickle). Five of them also answered with an instruction (Section 3.3). All calls went through one local gateway to the providers’ accounts. Free models may keep their inputs, so they saw public cases only.

3.3. Instructions

The bare condition sends only the case text, as the leaderboard does. Every other condition wraps the case in an instruction built from a few parts (Table 1; full text in Appendix A):

• The consult framing tells the model that a primary-care colleague has asked a specialist for the best recommendations, not only an answer to the literal question.
• The plan format asks for a one- or two-sentence assessment, then numbered, specific recommendations: tests, drugs with doses and durations, referral specialty and urgency, conditional next steps, counselling and return precautions, leaving out nothing a careful specialist would want and including nothing unsafe.

Neither part sets a length. The specialist instruction is the two together.

Each variant changes one thing. Two perspectives replace the consult framing with a clinical pharmacist or the patient’s primary-care physician. The exhaustive variant adds a list of plan headings to address. The candidate menu asks for every action a specialist panel should consider, instead of a plan. Three component instructions take the specialist instruction apart: the consult framing alone, the plan format alone, and, as a control for length, a request for at least 25 numbered recommendations that names no clinical content. Twenty-five is about how many actions the specialist instruction drew (median 21 for terra and 27 for Sonnet 5.5). Finally, the kit’s own thorough priming prompt adds a consult framing, criteria of completeness, safety and restraint, and a 500-word cap.

The instructions were written from the benchmark’s published description of a good consultation answer and its finding that omissions dominate. None names a rubric option or was tuned to a case, but all were written and tried on these same public cases. Every completed arm is reported. Arms started and stopped before completion are not analysed: the specialist instruction on further models, the kit’s prompt on Sonnet 5.5, a capped specialist instruction and repeated generations.

3.4. Systems of Several Calls

An ensemble joins existing plans into one response, unedited. A completion pass gives a model the case and the plans and asks only for clearly indicated actions that none of them states. A writer merges plans into one and puts back source actions that an automated check finds missing.

3.5. Evaluation

The kit’s evaluator extracts the actions a plan states, matches each to the rubric options, and reviews the matches. Claude Sonnet 5.5 was the primary evaluator; on the kit’s one worked example it agreed with the organisers’ evaluator on 323/330 option verdicts. GPT-5.6 terra re-scored selected arms. Nemotron 3 Ultra, a free model that wrote none of the plans, re-scored terra’s bare and instructed arms. It was chosen over a second free model by a rule set before either was tested: closer agreement with the organisers’ evaluator on the worked example (327/330, against 326/330 for Ling 3.1 Flash and 325/330 for GPT-5.6 terra). Re-scoring never changes a plan; every re-scored response is checked byte for byte against its source.

3.6. Outcomes

The primary outcome is the kit’s severity-weighted F1. Precision is the severity-weighted appropriateness of the stated actions that match a rubric option. Recall is the severity-weighted share of appropriate options the plan covers. We also report the share of items with at least one potentially severe error as judged by the evaluator, a benchmark judgement and not a measured adverse-event rate.

The headline precision has a blind spot. An action that matches no rubric option gets a neutral score, so it is never counted as harmful, and longer plans state more such actions. We therefore added these checks:

• F1 recomputed with the kit’s own precision over all stated actions;
• the share of stated actions that match no option, and plan length;
• the worst of each case’s eleven versions;
• ratings of every unlisted action by model raters from other families, on the specialists’ 1–9 scale, given the case and no rubric. This covered four bare-versus-instructed pairs (Sonnet 5.5 with the primary-care perspective, terra with the specialist instruction, Astra with the exhaustive variant and Grok with the kit’s prompt) on a fixed sample of 90 items per arm: each case’s original and two of its versions. Nemotron 3 Ultra rated all four pairs and Ling 3.1 Flash the Sonnet pair. In an earlier analysis of these cases, a rater of this kind caught 0.90 of the options specialists rated harmful, and 0.69 of its flags were such options, so these ratings are a screen, not a harm rate;
• a physician’s review of what the screens flagged.

For the physician review, the author, an internal-medicine physician, read three sets of items on a purpose-built page. Part A showed every unlisted action on the audit sample that either rater scored 1–3 (72), mixed with 24 randomly drawn unflagged actions, one at a time with the case, and with the model, instruction and rater flag hidden. Part B showed each option the specialists rated harmful (1–3) that an instruction added to a model’s plan (43 options in 18 cases, stated in 227 plan statements), with the panel’s rating and the plan’s wording, and asked whether the option is harmful for this patient as stated. Part C showed the 9 instructions verbatim. The reviewer gave one overall verdict covering every item, not item-by-item scores.

3.7. Statistics

Item scores were averaged within each base case, and the 30 case means were averaged equally. Every comparison is paired by case. Intervals are percentile 95% intervals from 10,000 resamples of whole cases (seed 7), conditional on the stored responses and evaluations. Across models we report the Spearman correlation between a model’s gain and how many actions it states unprompted, the standard deviation and range of scores in each condition, and Kendall’s $\tau$ between the two rankings, each with an interval from the same resampling. Across all single-call configurations we report the Spearman correlation between stated actions and F1. The study is exploratory: instructions and systems were developed on these cases, nothing was preregistered, and intervals are not adjusted for multiple comparisons.

3.8. Use of AI Tools in Preparing This Study

Apart from the models under study and the evaluators and raters described above, AI assistants were used to carry out the work: Claude Code and Codex assisted with the experiment code, the analysis and the drafting of this manuscript, and earlier drafting used the Scientific Agent Skills workflow [25]. The author directed the study, reviewed the manuscript, references and declarations, and takes responsibility for them.

4. Results

4.1. One Instruction Raised Every Terse Model

Table 2 lists the twelve bare models and the five instructed arms. Bare scores ranged from 0.354 to 0.764, and models stated between 3.5 and 17 actions per plan. The uncapped instruction raised Sonnet 5.5 by $+0.071[+0.036,+0.110]$, terra by $+0.175[+0.108,+0.248]$ and Astra by $+0.189[+0.120,+0.264]$. The kit’s capped prompt raised Grok by $+0.279[+0.169,+0.382]$ and left Muse Spark unchanged ($+0.003[-0.047,+0.053]$). Across the five, the gain fell as unprompted length rose (Spearman -0.90 $[-1.00,-0.80]$): models that already wrote long plans had least to gain. Muse Spark is the exception the design cannot explain. It wrote as much as Sonnet unprompted but received the capped prompt, so whether the cap or the model explains its null result is untested here.

Every instructed arm raised recall more than precision. Sonnet’s recall rose from 0.732 to 0.861 (precision 0.836 and 0.841), and terra’s from 0.623 to 0.847 (precision 0.841 and 0.885). The share of items with a potentially severe error fell under each uncapped instruction (Sonnet 0.206 to 0.155; terra 0.245 to 0.139).

4.2. The Field Narrowed and Its Order Changed

Instructing the five models halved the spread between them. The standard deviation of their scores fell from 0.17 to 0.09 (a reduction of 0.08 $[0.03,0.12]$) and the range from 0.41 to 0.22; among the three given the uncapped instruction, the range fell from 0.13 to 0.03. The order changed too. Bare, Sonnet 5.5 led terra (0.764 against 0.678); instructed, terra led Sonnet (0.853 against 0.835), and Astra passed Muse Spark. The rank correlation between the two conditions was Kendall’s $\tau$ = 0.40 $[0.00,0.60]$.

Across all 26 single-call configurations, bare and instructed, the number of actions a plan stated tracked its F1 (Spearman 0.89 $[0.80,0.92]$). This does not show that length causes the score: an instructed model states more actions and, judging by recall, more of the indicated ones.

4.3. The Plan Format Carries the Gain

Perspective made little difference. Three instructions on Sonnet 5.5 shared the plan format and differed only in perspective: a specialist answering a colleague (0.835), a clinical pharmacist (0.835) and the patient’s primary-care physician (0.846). Each beat the bare case by a similar margin ($+0.071[+0.036,+0.110]$, $+0.071[+0.028,+0.115]$ and $+0.082[+0.039,+0.125]$), and the primary-care perspective was indistinguishable from the specialist one ($+0.010[-0.010,+0.031]$). Asking terra for a candidate menu instead of a plan scored 0.807, below its plan ($+0.130[+0.055,+0.204]$ over bare, against $+0.175[+0.108,+0.248]$), at nearly twice the length.

The format alone matched the full instruction.Table 3 gives each part on its own. Against the bare case, the consult framing alone changed Sonnet 5.5’s score by $+0.031[+0.011,+0.052]$ and terra’s by $+0.077[+0.032,+0.127]$; the plan format alone by $+0.076[+0.041,+0.115]$ and $+0.169[+0.104,+0.241]$; and the request for 25 recommendations by $+0.031[+0.004,+0.058]$ and $+0.091[+0.052,+0.135]$. Against the full instruction, the format alone scored $+0.005[-0.007,+0.017]$ for Sonnet 5.5 and $-0.006[-0.018,+0.005]$ for terra, level with it, while the framing alone fell well short ($-0.040[-0.071,-0.011]$ and $-0.098[-0.147,-0.057]$).

Asking for length alone was not enough. The request for 25 recommendations drew as many actions as the plan format (median 29 against 28 for Sonnet 5.5, 24 against 22 for terra) but fewer of the indicated ones (recall 0.807 against 0.867, and 0.741 against 0.844). It scored below the full instruction ($-0.040[-0.069,-0.015]$ and $-0.084[-0.148,-0.029]$), and once every stated action counted in precision, Sonnet’s gain under it was $+0.002[-0.026,+0.029]$. What the format names (exact tests, doses and durations, referral urgency, conditional steps, counselling and return precautions) steers which actions are added. These are two models and one wording of each part, so they place the gain in the format without showing which of its clauses matters.

4.4. What the Instruction Adds

The recovered actions are the ones models leave out unprompted (Table 4, Figure 1). Follow-up was the largest gap: Sonnet covered 0.20 of the appropriate follow-up options bare and 0.65 instructed ($+0.44[+0.27,+0.63]$), and terra 0.43 and 0.90 ($+0.48[+0.25,+0.69]$). Counselling came next (Sonnet $+0.25[+0.14,+0.37]$; terra $+0.21[+0.10,+0.34]$). NOHARM describes these categories as the ones models and physicians alike leave out [1]. Coverage of the score-9 actions, whose omission the specialists rated most harmful, rose for the three terse models (terra $+0.15[+0.02,+0.31]$, Astra $+0.20[+0.07,+0.39]$, Grok $+0.29[+0.09,+0.48]$) but not for Sonnet, which already covered 0.83 of them bare ($+0.03[-0.03,+0.11]$).

Muse Spark, the one model the instruction did not help, gained in no category under the kit’s capped prompt (overall coverage $-0.01[-0.06,+0.03]$; score-9 coverage $-0.07[-0.17,+0.04]$). The cap halved its plans, from a median of 5,179 to 2,636 characters, while the same cap doubled Grok’s short plans, from 734 to 1,396. A 500-word cap pulls answers toward its length from both sides, and the score followed length. This is an observation on two models, not a test of the cap.

4.5. More than Length

In a regression of item F1 on the instruction, with model and case fixed effects (3,300 item scores, five models), the instruction’s effect was $+0.144[+0.090,+0.200]$. Adding the logarithm of the number of stated actions reduced it to $+0.016[-0.023,+0.058]$: at an equal number of stated actions, instructed and bare plans scored alike, and the action count accounted for 89% $[66,120]$ of the effect. Among the 137 item pairs whose bare and instructed plans stated within 10% as many actions, the instructed plan scored $+0.022$ higher on average. The instruction works largely by making models state more actions (Figure 2). But the count is not the whole story: the request for 25 recommendations reached the same count and scored lower (Section 4.3). The next question is whether the added actions are worth stating.

4.6. Are the Added Actions Safe?

Counting every stated action. The headline precision ignores off-rubric actions, which made up 53% of Sonnet’s bare plan and 60% of its instructed one (Table 5). With the kit’s own precision over every stated action, where an unlisted action counts as equivocal, each gain shrank but held: Sonnet $+0.049[+0.014,+0.086]$, terra $+0.140[+0.071,+0.214]$, Astra $+0.142[+0.070,+0.218]$ and Grok $+0.265[+0.156,+0.369]$. The kit’s strictest variant, which counts every unlisted action as inappropriate and weights every option equally, reverses the gain for three models (Sonnet $-0.056[-0.078,-0.035]$, terra $-0.064[-0.110,-0.016]$, Astra $-0.073[-0.117,-0.029]$; Grok $+0.118[+0.039,+0.192]$); its binary variant keeps it (Sonnet $+0.026[-0.000,+0.052]$, terra $+0.104[+0.042,+0.172]$). Which reading is right depends on whether the unlisted actions are appropriate, which the rubric cannot say.

Model raters. The instruction added unlisted actions for every model (per item: Sonnet 8.69 to 17.67, terra 5.09 to 13.19, Astra 5.74 to 17.52, Grok 2.30 to 4.63). Nemotron 3 Ultra found no more harmful ones among them (change per item: $-0.01[-0.10,+0.10]$, $-0.03[-0.09,+0.01]$, $+0.01[+0.00,+0.03]$ and $+0.00[-0.09,+0.09]$) and put 92–99% of the unlisted actions in every arm at 7–9. Scoring each unlisted action with this rater’s rating instead of the kit’s neutral default, the instruction’s gain on the sample was $+0.117[+0.079,+0.160]$ for Sonnet, $+0.193[+0.115,+0.274]$ for terra, $+0.200[+0.128,+0.281]$ for Astra and $+0.304[+0.196,+0.411]$ for Grok; by this rater’s reading, the strictest variant is the wrong one. The second rater, Ling 3.1 Flash, put 0.89 of 2,372 paired ratings of the Sonnet pair in the same band as Nemotron (Cohen’s $\kappa$ 0.33), but rated more of the instructed plans’ unlisted actions harmful ($+0.16[+0.04,+0.28]$ per item); counting an action as harmful only when both raters did, the change was $+0.01[-0.03,+0.06]$. With Ling’s ratings, Sonnet’s gain was $+0.116[+0.078,+0.160]$. The two raters do not agree closely, which the physician review checks.

Physician review. The physician judged all 96 actions in Part A appropriate, including all 72 that a model rater had scored harmful (31 from bare plans and 41 from instructed ones; Nemotron flagged 43 and Ling 34). On this sample every harm flag the raters raised was unfounded, so Ling’s increase for Sonnet does not reflect harm the physician recognised. The physician raised no concern with any instruction.

Harmful options from the rubric. Against the specialists’ own ratings, instructed Sonnet proposed 0.42 harmful-rated options per item against 0.29 bare ($+0.13[+0.05,+0.23]$), nearly all rated mildly or moderately harmful (score 2–3; severe $+0.01[-0.00,+0.04]$). For terra ($+0.00[-0.07,+0.08]$), Astra ($+0.04[-0.02,+0.11]$) and Grok ($+0.07[-0.09,+0.27]$) the change was not distinguishable from zero. These ratings judge each option for the case in general. Read in the plans’ own wording (Part B), the physician judged none of the 43 added options harmful as stated. A list of every harmful option an instruction added, with the plan’s wording, is part of the review materials (Data section).

This review is one reviewer, the study’s author, giving an overall verdict, blinded only in Part A. It checks the model raters; it is not an independent clinical validation.

4.7. How Firm Are These Estimates?

Uncertainty and power. Each of the four positive within-model gains had a bootstrap p below 0.001 and stayed so after Holm adjustment across the five primary contrasts (adjusted p for Sonnet <0.001, terra <0.001); Muse Spark’s was 0.904. With 30 cases, the smallest difference each contrast could detect with 80% power was 0.053 for Sonnet and 0.102 for terra.

Evaluator consistency. Matching the same 140 instructed terra plans twice, the primary evaluator agreed with itself on 0.982 of option verdicts (Cohen’s $\kappa$ 0.95). GPT-5.6 terra agreed with it on 0.960 ($\kappa$ 0.88) over all 330 items.

A second evaluator. Re-scored by Nemotron 3 Ultra, terra’s gain from the instruction was $+0.187[+0.119,+0.259]$, within $-0.012[-0.040,+0.015]$ of the primary estimate. The same re-scoring of Sonnet’s pair was stopped before completion and is not reported. With GPT-5.6 terra as evaluator, instructed terra scored 0.844 and bare Sonnet 0.734.

Where the gain falls. The gain was largest where models did worst unprompted. Across the five models, the third of cases with the lowest bare scores (mean 0.45) gained $+0.228$, the middle third $+0.178$ and the top third (mean 0.83) $+0.025$, consistent with a ceiling. By specialty the mean gain ranged from $-0.072$ (GI) to $+0.394$ (Heme), three cases each. By the kind of detail a case version changes, it ranged only from $+0.086$ to $+0.180$, including versions that change only the patient’s age ($+0.141$, 41 items) or sex ($+0.136$, 23 items), against $+0.155$ on the original cases: a limited check that the instruction helps regardless of these patient characteristics.

4.8. One Instructed Call Against Systems of Several Calls

Instructed terra (one call) scored level with an ensemble of three bare plans plus a completion pass (four calls): $+0.004[-0.035,+0.041]$ (Table 6). The completion pass had added $+0.038[+0.009,+0.070]$ to that ensemble, so one instruction recovered what two extra models and a completion pass recovered together. Joining the three best instructed plans (Sonnet primary-care, terra specialist, Astra exhaustive) scored 0.897, above instructed terra by $+0.044[+0.016,+0.072]$. Part of that margin is repetition: counting actions matched to the same rubric options once, it was $+0.035[+0.005,+0.065]$, and with GPT-5.6 terra as evaluator $+0.020[-0.012,+0.052]$, an interval that includes zero. Merging the joined plans into one readable plan did not work: a writer with automated re-insertion lost source actions on the earlier, unprompted ensemble (Appendix C), and on one trial item of the instructed ensemble it listed 194 source actions and lost more of them with each repair round.

4.9. Where These Scores Sit

The benchmark publishes public-case reference scores, under its own evaluator, of 0.720 for GPT-5.5, 0.674 for Claude Opus 4.7 and 0.664 for Gemini 3.1 Pro. The public leaderboard, scored by the organisers on held-out cases, is led by LiSA 2.5 at 86.2, followed by Doximity Ask 6.1 (84.5) and OpenEvidence (80.0); bare frontier models sit between about 56 and 75. Our scores use the public cases and a different evaluator, so none is a leaderboard number. Our best single call (0.853) and best combination (0.897) are above the reference scores, but only a comparison under the same evaluator could show by how much. NOHARM reports that public and held-out scores correlate closely across models (Pearson 0.95) [1], which supports comparing ranks, not absolute values. Two further cautions apply to absolute values. Our second evaluator scored the instructed ensemble about 0.03 lower than the primary one, and the primary evaluator favoured an Anthropic-written plan set by $+0.037[+0.014,+0.060]$ relative to another model’s in a paired test. And the public cases were released in June 2026, so models scored later may have seen them; contamination would inflate absolute scores but not a within-model comparison on the same cases.

5. Discussion

On these cases, how a model was asked did more for its score than which model was asked. An instruction asking for a complete, structured plan lifted terse models by as much as $+0.279[+0.169,+0.382]$, narrowed the field and reordered it. The perspective and consult framing mattered little; the plan format alone reproduced the gain. This fits NOHARM’s finding that omissions dominate its errors: a model that says more of what a careful specialist would do omits less, and the metric rewards exactly that.

This extends the benchmark’s own finding that priming helps generalist models [1]. Its priming prompts capped the answer; ours did not. The two capped arms here behaved differently from the uncapped ones: a large gain for the tersest model, none for one that already wrote long plans. The cap may be part of why priming left generalists behind clinical tools in the benchmark’s report, but our data cannot separate the cap from the model.

Length is part of the mechanism, not the whole of it. At an equal number of stated actions an instructed plan scored no better than a bare one, so the instruction works largely by making models say more. But a plain request for 25 recommendations reached the same count and scored lower, so what the format names steers which actions are added.

Are the added actions worth stating? The evidence mostly says yes:

• the added actions that match the rubric are mostly follow-up and counselling, which the specialists rated appropriate;
• counting every unlisted action in precision shrank the gains without removing them;
• one model rater put nearly all unlisted actions at 7–9 and found no more harmful ones for any model, although a second rater found a small increase for Sonnet;
• the physician found none of the flagged actions inappropriate, and none of the added specialist-rated harmful options harmful as the plans stated them;
• a third-family evaluator gave terra a similar gain, and the worst version of each case improved along with the average.

Against this, instructed Sonnet proposed more options the specialists rated harmful, mostly mild or moderate, and the kit’s strictest metric, which counts every unlisted action as wrong, reverses the gain for three models. One reviewer’s overall verdict is not a panel’s item-by-item ratings, and AI evaluators are known to favour longer text [11,12,13]. The safest reading is that the instruction makes models state more of the indicated actions, along with a few options the benchmark’s specialists rated harmful in general that one physician did not find harmful in context, and that under the benchmark’s severity weighting the first outweighs the second.

Two practical points follow. For benchmark design, an unprompted protocol ranks models partly on how much they write by default; reporting prompted beside unprompted results, as NOHARM’s own study does, makes that visible, as does reporting how much of a score rests on actions the rubric cannot see. For system design, one instructed call matched a four-call ensemble with completion at a quarter of the calls, taking a median of 58 seconds per case (90th percentile 110) against the benchmark’s 300-second limit. Neither point says anything about clinical usefulness: a plan of twenty to thirty items needs ordering by priority before a clinician can use it.

The benchmark represents advice to a primary-care clinician who has asked a specialist about one patient. The input is the clinician’s question with the case details, and the output is a plan that the clinician reads, judges and acts on. Nothing here evaluates a plan acting without that clinician, and a longer plan is only as safe as its reader is attentive.

Four next steps would turn these observations into tests. Scoring instructed and bare models on the held-out cases would show whether the gain survives the organisers’ evaluator on unseen cases; the leaderboard admits models rather than prompts, so this needs the organisers’ own runs or a research arrangement with them. A blinded panel rating the added actions item by item, as the benchmark’s specialists rated its rubric, would turn this study’s single-reviewer check into a measurement. Giving one instruction to every model, testing the length cap within a model and measuring regeneration noise would remove the remaining confounds. And reporting prompted beside unprompted scores on other clinical benchmarks would show whether the gap between general models and clinical tools is a property of the models or of how they are asked.

6. Limitations

Small development set. The study has 30 base cases, not 330 independent consultations, and the instructions were developed on these same cases. None names a rubric option, but estimates may be optimistic.

Incomplete design. Five of twelve models were instructed, with different instructions, and cross-model statistics rest on those five. The instruction was taken apart on two models only. The capped and uncapped instructions went to different models, so the cap’s own effect is untested. Regeneration noise was not measured; evaluator noise was, on one arm. Several planned arms were stopped before completion and are not analysed.

Automated evaluation. Every score comes from an AI evaluator, not the organisers’ and not physicians, although a second model family re-scored the key contrasts. The author, an internal-medicine physician, reviewed the flagged off-rubric actions, the added harmful options and the instructions, giving an overall verdict rather than item-by-item scores, blinded only for the off-rubric actions. No physician reviewed whole plans or evaluator disagreements, and no independent panel was used.

What the metric cannot see. Headline precision ignores actions the rubric does not list, and the kit never counts them as harmful; our checks bound this but do not replace clinical review. Longer plans may also be harder to use, and usability was not measured.

Contamination and drift. The public cases have been available since June 2026. Provider routes and model versions change, and free models may keep their inputs.

7. Conclusion

One structured, uncapped instruction raised NOHARM scores for every terse model tested, narrowed the gap between models, and matched a multi-model system in a single call. It worked by making models state more of the indicated actions; in the two models taken apart, the plan format, not a demand for length, did the work. Benchmarks of clinical completeness should report prompted beside unprompted results, and models, or models and clinical tools, should be compared under both before conclusions are drawn.