Measuring Prompt-Side Uncertainty with Four Diagnostic Heads: Internal Validation and Prospective Falsification

1. Introduction

Large language models can produce fluent responses to requests that do not determine a single defensible answer. A user may leave a required constraint unstated, use a term with multiple discrete interpretations, rely on a category with a fuzzy boundary, or express an intention that supports several plausible tasks. In each case, the model can answer confidently while silently selecting an interpretation that the user did not intend. The resulting failure begins in the request, before factuality, reasoning quality, or response calibration can be evaluated.

Most established uncertainty methods address a different stage of the interaction. Work on semantic uncertainty groups model generations by meaning and estimates dispersion across those groups, allowing uncertainty to be assessed despite linguistic variation (Kuhn et al., 2023). Semantic entropy has also been used to identify confabulations in generated answers (Farquhar et al., 2024). Related research asks whether language models know when they know an answer, express calibrated confidence, or abstain when answering is risky (Kadavath et al., 2022; Lin et al., 2022; Tomani et al., 2024; Wen et al., 2024; Xiong et al., 2024). These approaches are relevant to response reliability, but they do not by themselves identify why the input fails to license one response strategy.

Prompt-side uncertainty creates an action problem as well as a measurement problem. A system can respond directly, rewrite the request into an explicit interpretation, or ask a clarifying question. None of these actions is universally preferable. Direct response avoids another interaction turn but may commit to a harmful assumption. Rewriting can expose an assumption without imposing the latency of a question, yet the rewrite itself may distort the user’s goal. Clarification can resolve missing information, but unnecessary questions increase effort and may interrupt otherwise straightforward work. Research on explicit ambiguity handling and selective clarification demonstrates that models can be trained or prompted to recognize when interaction should continue rather than proceeding under a dominant interpretation (Kim et al., 2024; Zhang & Choi, 2025; Zhang et al., 2025). The unresolved methodological issue is how to measure different sources of prompt-side uncertainty before comparing the costs and rewards of possible actions.

This study addresses that issue through a staged validation design. The present scope does not test a complete action router or a universal definition of a good prompt. It evaluates whether four predefined prompt-side constructs can be operationalized by independent estimator heads with reproducible outputs and construct-aligned internal signals. The four measures target intent ambiguity (U_intent), semantic or interpretive ambiguity (U_sem), fuzzy-boundary uncertainty (U_fuzzy), and missing-context uncertainty (U_context). They were selected as action-relevant input conditions that require different elicited objects and different mathematical summaries. The decomposition is operational, non-exhaustive, and testable; it excludes output factuality, reasoning quality, safety, style, domain adequacy, and user preference except when those outcomes are independently measured downstream.

The study asks one bounded measurement question: Across a controlled development corpus and a separately constructed prospective English corpus, to what extent can four independent estimator heads produce reproducible outputs and support construct discrimination and exploratory incremental-value comparisons? The evidence comprises a frozen estimator protocol; Pilot-300 internal validation; immutable first-pass and targeted-recovery outputs; a provenance-limited 30-sample human adjudication; and the Pilot-120 prospective measurement and construct-ablation result. The wider research program poses a separate, currently untested question: whether using these measurements to select or optimize Direct, Rewrite, or Clarify improves independently scored downstream utility. Separating measurement from action utility makes the prospective failure informative rather than allowing an internal score increase to stand in for prompt quality.

The paper contributes three linked but bounded artifacts. First, it specifies four auditable prompt-side measurements and implements them through a prototype Prompt Uncertainty Auditor skill. Second, Pilot-300 provides internal engineering and construct-signal validation under controlled templates. Third, Pilot-120 supplies a prospective falsification result: the semantic head/corpus pairing fails its intended construct contrast and shows no demonstrated incremental construct-classification benefit. The prototype skill is an implementation candidate rather than the already validated v4 research estimator, and its outputs are diagnostic features rather than a four-part definition of prompt quality. Independent outcome scoring and same-prompt action experiments remain necessary before the framework can support routing or optimization claims.

2. Related Work and Conceptual Foundations

2.1. Prompt Conditions: Ambiguity, Vagueness, Underspecification, and Clear Controls

Natural-language uncertainty has several sources. Ambiguity permits competing discrete interpretations. A request to “summarize the bank report,” for example, may leave unresolved which institution or which report is intended. Vagueness differs because the relevant category or threshold lacks a sharp boundary. Terms such as “brief,” “nearby,” or “senior” can have a recognizable center without a determinate cutoff. Underspecification occurs when information needed for the task is absent, even if the words that are present have a stable interpretation. A request to plan a trip without a budget may be linguistically clear but operationally incomplete.

The distinction between ambiguity and vagueness has a substantial linguistic and psychological basis. Verheyen and Storms (2013) modeled them as separable sources of uncertainty in concept judgments rather than interchangeable labels. More recent work in human-robot interaction likewise treats uncertainty, vagueness, and ambiguity as related but distinguishable interaction problems (Sun et al., 2026). The present taxonomy adopts this distinction for measurement. It does not claim that every prompt belongs to exactly one category. A prompt can contain both missing context and multiple interpretations, and the four heads are therefore independent measures rather than mutually exclusive classifiers.

A clear control is defined operationally as a prompt that contains sufficient information to perform the intended task under the study’s construction rules. “Clear” does not mean easy, factual, safe, or guaranteed to receive a correct answer. It concerns the input specification. A clear prompt can still exceed the model’s knowledge, trigger a reasoning error, or produce hallucinated content. Conversely, an ambiguous prompt can sometimes receive a factually correct answer by chance. Separating prompt condition from output quality prevents the estimator from using answer correctness as a shortcut.

This distinction also explains why hallucination taxonomies are relevant but not identical to the present target. Surveys of hallucination in large language models organize failures involving factual inconsistency, fabrication, and divergence from sources or instructions (Huang et al., 2023). Some such failures may be downstream consequences of prompt uncertainty, but prompt uncertainty is not defined by whether a hallucination occurs. The estimator receives only raw prompt text. Gold labels, constructed condition fields, response quality, and downstream outcomes are excluded from the provider-facing input.

AmbigQA pairs plausible answers with disambiguated versions of an open-domain question (Min et al., 2020). CLAMBER evaluates recognition and clarification of ambiguous information needs (Zhang et al., 2024). More recent work links underspecified classification instructions to prompt sensitivity (Pecher et al., 2026). These studies establish direct precedents for investigating input ambiguity and incomplete specification. Our contribution is a four-construct operationalization and its bounded internal and prospective evaluation.

2.2. Output Uncertainty and the Need for Input-Side Measures

Model uncertainty is commonly estimated from probabilities, sampled responses, semantic equivalence, verbalized confidence, or self-evaluation. Kuhn et al. (2023) showed that uncertainty estimation can account for linguistic invariance by operating over semantic equivalence classes rather than surface strings. Farquhar et al. (2024) extended semantic entropy as a method for detecting confabulations in open-ended generation. These methods ask whether candidate outputs converge on a meaning and whether that convergence predicts reliability.

Other approaches elicit a model’s confidence or willingness to answer. Lin et al. (2022) investigated whether models can express uncertainty in words, while Kadavath et al. (2022) examined whether models can evaluate the correctness of their own answers. Xiong et al. (2024) evaluated the extent to which large language models express calibrated uncertainty. Abstention research then uses uncertainty or other signals to decide whether a model should withhold an answer (Tomani et al., 2024; Wen et al., 2024).

Prompt-side measures answer a preceding question: does the request itself license a stable interpretation and an adequately specified task? The answer can influence whether generation should begin, but it is not reducible to answer confidence. A knowledgeable model may be highly confident about one reading of an ambiguous prompt and still answer the wrong question. A model may also be uncertain about a difficult answer to a clear prompt. Treating these situations as one score obscures the intervention. Clarification may help the former but not the latter; retrieval or abstention may help the latter but not the former.

SPUQ perturbs inputs, samples outputs, and aggregates their variation to quantify uncertainty in model behavior (Gao et al., 2024). The present measurements instead describe unresolved properties of the observed request, using model-elicited intermediate representations before downstream task execution. This distinction concerns the measurement target, not an absence of earlier input-variation methods.

The four measures were selected by four design criteria. First, each targets an input-side failure that can change whether a system should answer, rewrite, or request information. Second, each requires a distinct elicited object: probabilities over candidate intents, probabilities and distances over interpretations, graded boundary memberships, or paired distributions before and after material context. Third, each has an auditable mathematical summary with explicit edge cases and a nominally bounded scale. Fourth, the set supports localized inspection and intervention rather than collapsing all uncertainty into one generic caution score. Under these criteria, U_intent uses normalized Shannon entropy over plausible intent categories; U_sem uses Rao quadratic diversity to combine interpretation probability with consequence-sensitive distance; U_fuzzy uses normalized De Luca-Termini fuzzy entropy over graded memberships or boundary judgments; and U_context uses normalized Jensen-Shannon divergence between the observed-prompt and context-completed distributions.

This selection is neither a claim of completeness nor evidence that four is the uniquely correct number of dimensions. The heads may correlate, one head may add no incremental value, and omitted factors may dominate downstream quality. Output factuality, instruction following, safety, reasoning quality, style, domain conventions, user preference, and interaction cost remain separate constructs or outcomes. The mathematical familiarity of the four formulas validates neither their model-elicited inputs nor their sufficiency. Complementarity must be tested through correlation and collinearity diagnostics, incremental prediction, single-measure models, and leave-one-measure-out ablation; if a simpler baseline performs equivalently, the four-head claim must be narrowed.

2.3. Clarification and Localized Attribution

Ambiguity-aware models can be evaluated not only by whether they detect uncertainty but also by how they respond. Kim et al. (2024) studied explicit ambiguity handling rather than rewarding a model for always selecting a dominant interpretation. Zhang and Choi (2025) focused on clarifying when necessary, which frames clarification as a selective behavior rather than a default. Zhang et al. (2025) modeled future conversation turns to support asking clarifying questions, making the likely value of another turn part of the decision.

These studies motivate attribution. A scalar score can indicate that a prompt is uncertain, but it does not tell the system or the user which text span created the problem. A localized span can support a focused question or an explicit rewrite. It can also enable a causal intervention test: changing the highest-ranked span should reduce the relevant score or improve task outcomes more than changing a random span, if the attribution identifies a decision-relevant cause.

The current estimator records auxiliary span and rationale fields alongside the core numeric payload. That design improves auditability, but the auxiliary output has a different validation status from the score. A syntactically valid span is not necessarily human-valid attribution. Manual review must determine whether the selected phrase is present, whether it corresponds to the claimed construct, and whether revising it changes the downstream decision. The 30-sample internal adjudication addresses presence and construct-alignment judgments, but causal span validity, independent replication, and template-independent human validation remain incomplete.

Ramesh et al. (2026) localize prompt ambiguity using a linear probe and attribution to token representations. Their work directly addresses ambiguity localization, whereas our spans are auxiliary outputs of construct-specific heads. Neither a plausible span nor its agreement with a study key establishes that editing it causes better task outcomes.

2.4. From Measurement to Cost-Sensitive Action

Direct response, rewriting, clarification, abstention, retrieval, and verification solve different problems. Abstention is appropriate when the system cannot support a safe answer, but it may be excessive when one low-cost question would supply the missing detail. Clarification can reduce interpretive risk, yet repeated questions impose interaction costs. Rewriting can expose an assumption while preserving a single-turn workflow, although a model-generated rewrite can introduce a new assumption. A useful policy must therefore compare actions on the same prompt under an explicit utility function.

The literature on abstention establishes that selective answering can improve risk control when uncertainty signals are informative (Tomani et al., 2024; Wen et al., 2024). Confidence expression and self-evaluation offer candidate signals for that choice (Kadavath et al., 2022; Lin et al., 2022; Xiong et al., 2024). However, a policy’s value cannot be inferred from detector discrimination alone. Two prompts with the same uncertainty score may have different error costs, response lengths, user burdens, or near-tied action rewards.

The full project architecture links prompt condition to four measurements and an attributed span, then to Direct/Rewrite/Clarify actions, observable outcomes, utility, failure attribution, and validation-gated updating. Scope A implements and evaluates only the measurement layer and preliminary attribution-review infrastructure. The action and update layers are planned confirmatory work. Accurate detection therefore does not prove low-regret routing, and reducing a diagnostic score does not by itself prove that a prompt improved.

For the planned action study, a prompt or intervention is operationally better only when it improves an outcome that is independent of the four diagnostic scores. Candidate outcome components are task success, intent alignment, factuality or faithfulness, constraint satisfaction, unsupported assumptions, usefulness, domain-specific risk, token and latency cost, and required interaction turns. These components must be frozen before evaluation and combined only through declared utility profiles. A head value of .70 or .85, an AUC between .70 and .85, or a 70%-85% increase in an optimization objective is not a 70%-85% prompt-quality score. Any percentage must name its statistic, numerator and denominator, aggregation rule, comparison baseline, evaluation set, uncertainty interval, and whether it was optimized or independently assessed.

Prompt Optimization with Textual Gradients (ProTeGi) uses task feedback to revise prompts and select candidates (Pryzant et al., 2023). Huang et al. (2026) study explicit semantic disambiguation before inference. These approaches motivate downstream comparisons, but their reported effects cannot be transferred to our diagnostic scores or prototype. A reduction in uncertainty requires an independently measured task benefit before it can be described as prompt improvement.

3. Methods

3.1. Study Design and Evidence Stages

The study separates historical development, current internal evaluation, prospective measurement, and future confirmation. Historical development includes the 36-item protocol and a pre-v4 exploratory Pilot-300 action pilot. Current internal evidence comprises Pilot-300, the frozen v4 estimator, first-pass and recovery outputs, quantitative summaries, and a 30-sample key-assisted human adjudication. The prospective line comprises the separately constructed Pilot-120 corpus evaluated with the unchanged v4 heads. Pilot-120 measurement and action generation were executed; its independent downstream outcome evaluation was not. Further confirmation after any construct or estimator redesign remains future work. Results are interpreted within these evidence stages.

Results were not pooled across stages. The historical protocols used different estimators or comparison structures. Pilot-300 contains five task families and controlled development templates; Pilot-120 contains four task groups and separately sourced or authored material. These differences prevent interpreting their combined size as one validation sample or treating Pilot-120 as full execution of the earlier 300-prompt confirmation protocol.

The unit of analysis for the core internal evaluation was the prompt by estimator head. A prompt family was the operational unit for first-pass completion and targeted recovery reporting. Dataset partitions were defined in advance, although the present analysis is an internal development analysis rather than a one-time final held-out test. The paper uses “internal validation” to describe this bounded stage.

Figure 1. Evidence stages and remaining evaluations. Pilot-300 internal analysis and Pilot-120 prospective measurement are completed, with 119 of 120 Pilot-120 families available for complete-case analysis. Pilot-120 shows weaker discrimination and a semantic construct failure. Action generation is completed, but independent outcome scoring and action-policy evaluation remain pending. The parallel prototype is not equivalent to the frozen research estimator. Boxes labelled FUTURE indicate future work, not observed improvement.

Figure 1. Evidence stages and remaining evaluations. Pilot-300 internal analysis and Pilot-120 prospective measurement are completed, with 119 of 120 Pilot-120 families available for complete-case analysis. Pilot-120 shows weaker discrimination and a semantic construct failure. Action generation is completed, but independent outcome scoring and action-policy evaluation remain pending. The parallel prototype is not equivalent to the frozen research estimator. Boxes labelled FUTURE indicate future work, not observed improvement.

3.2. Pilot-300 Dataset

Pilot-300 contains 300 controlled synthetic English prompts. The corpus spans five task families: classification, code generation, information extraction, planning, and summarization. Each family contributes 60 prompts. The dataset is balanced across four prompt-condition groups, with 75 ambiguity, 75 vagueness, 75 underspecification, and 75 clear examples. It is also balanced across four decision-profile labels used for later study design. The predefined split contains 180 training, 60 validation, and 60 test rows.

All 300 rows have the provenance label controlled_synthetic_template. The dataset contains 50 unique generator-template identifiers. This construction provides controlled contrasts and task coverage, but it is also the primary validity threat. The estimator and protocol were developed within the same broader template environment. Even when the provider receives only raw prompt text, lexical or structural cues can correlate with the constructed conditions.

Strong construct-target examples were distinguished from the broader condition labels for internal gate analysis. A strong target was designed to instantiate one head’s construct with lower ambiguity about the intended diagnostic mapping. Clear prompts served as negative controls. This contrast tests whether a head responds under a favorable construct-aligned condition. It is not equivalent to performance across all naturally occurring manifestations of ambiguity, vagueness, or underspecification.

The balanced construction serves experimental control but changes the meaning of descriptive performance. In an operating environment, clear requests may be much more common than any single uncertainty condition, and different domains may have very different base rates. AUC is less sensitive to prevalence than accuracy, yet threshold-specific false alarms and positive predictive value are not. For this reason, the internal analysis does not estimate how often a deployed system would interrupt a user unnecessarily. That estimate requires a naturalistic prevalence sample and an action threshold chosen under explicit costs.

The five task families also test breadth within a constrained design rather than population coverage. Classification and information extraction can make output requirements relatively explicit. Planning and summarization often admit several defensible objectives. Code generation can combine hidden environmental constraints with a clear functional request. Coverage across these families is useful because a construct signal that appears in only one family could reflect task wording. It still cannot represent the open-ended distribution of user requests.

Finally, the constructed condition label was not supplied to the estimator. This protects against direct label leakage, but indirect leakage remains possible through surface regularities. A generator may express underspecification with repeated missing-slot patterns or fuzzy boundaries with recurrent adjectives. Template-independent confirmation must therefore change both content and form. It should also include counterexamples in which common cue words occur in otherwise clear requests and uncertain requests are expressed without the cues observed during development.

3.3. Four-Head Estimator

The estimator used four independent calls or logical heads. Each head received the raw prompt and returned a typed core payload plus auxiliary attribution metadata. Condition labels, gold formula inputs, dataset fields, response outcomes, and downstream utility values were excluded from provider-facing prompts.

For a normalized distribution over K plausible intents, with probabilities p₁,…,pₖ, intent uncertainty was computed as normalized Shannon entropy:

$$U_{intent}={\displaystyle \frac{-{\sum }_{i=1}^K{p}_i\log p_i}{\log K}},K\ge 2.$$

For K=1, the implementation defines Uᵢntent=0, avoiding the undefined ratio 0/0. The score approaches zero when one elicited intent dominates and increases when mass is distributed across plausible intents. The head operationalizes competing task intentions; it does not estimate whether the model knows how to perform the task.

For candidate interpretations with probabilities pᵢ and pairwise distances d(i,j), semantic or interpretive uncertainty used Rao quadratic diversity:

$$U_{sem}=\sum _{i=1}^K\sum _{j=1}^K{p}_i{p}_{j}d\left(i,j\right).$$

The implementation fixed d(i,i)=0 exactly so floating-point self-distance could not create false positive dispersion. This measure differs from normalized entropy because two equally likely paraphrases should produce less semantic diversity than two equally likely interpretations with different consequences.

For graded membership values μᵢ ∈ [0,1], fuzzy-boundary uncertainty used normalized De Luca-Termini fuzzy entropy:

$$U_{fuzzy}={\displaystyle \frac{1}{n\log 2}}\sum _{i=1}^n\left[-{\mu }_i\log {\mu }_i-\left(1-{\mu }_i\right)\log \left(1-{\mu }_i\right)\right].$$

Terms at membership 0 or 1 contribute no fuzzy entropy under the continuous-extension convention 0log 0=0. Membership near .5 contributes more. The head targets uncertain boundaries or cutoffs, not multiple discrete meanings.

Missing-context uncertainty used normalized Jensen-Shannon divergence between an elicited distribution under the observed prompt, P, and a distribution under a context-completed alternative, Q:

$$U_{context}={\displaystyle \frac{1}{\log 2}}\cdot \left[{\displaystyle \frac{1}{2}}{D}_{\left(KL\right)\left(P\parallel M\right)}+{\displaystyle \frac{1}{2}}{D}_{\left(KL\right)\left(Q\parallel M\right)}\right],M={\displaystyle \frac{P+Q}{2}}.$$

Here P and Q are normalized probability distributions on a common finite support, and zero-mass terms follow 0log(0/q)=0. The measure is high when supplying material context changes the relevant distribution. It is low when plausible completion leaves the task interpretation stable. The normalization places all four core scores on a nominal zero-to-one scale, although equal numeric values across heads do not imply equal construct intensity or action cost.

Each response was checked against the expected schema. The analytical pipeline preserved immutable first-pass outputs and wrote targeted recovery responses separately before constructing a recovered analytical file. Recovery was restricted to failed head calls rather than rerunning complete prompt families. This design retains the distinction between service reliability and analytical completeness.

3.4. Internal Gates and Analyses

Engineering gates required parseable outputs, valid schemas, finite values, and scores within the expected range. The protocol also checked that only raw prompt text was sent to the provider. Auxiliary attribution warnings were counted separately from core score validity. A warning rate at or below 10% allowed core-formula validation to proceed, but did not validate the span locator.

Construct-signal analysis compared 25 strong target examples with 75 clear controls for each head. Receiver operating characteristic area under the curve (ROC AUC) was the primary discrimination summary, computed as the probability that a randomly selected strong-target score exceeded a randomly selected clear-control score, with one-half credit for ties. Bootstrap 95% confidence intervals used 4,000 stratified-within-class resamples with replacement, percentile endpoints, and random seed 20260726. These intervals represent sampling variation within the controlled dataset. The analysis also counted the number of task families in which strong examples produced positive signals and evaluated clear-control nonzero behavior. A secondary comparison used all target-labelled cases against clear controls. That comparison was retained because it is less selected than the strong-target gate and exposes heterogeneity in construct intensity.

No threshold from this analysis was treated as a deployment or prompt-quality threshold. AUC measures rank separation within the sampled contrast and is invariant to many monotonic transformations. It does not measure calibration, user burden, correctness, usefulness, or the cost of selecting the wrong action. The four head values are dimension-specific diagnostics on nominal zero-to-one scales, not percentages of overall quality and not directly commensurate across heads. The manuscript therefore reports every value with the named statistic, comparison groups, sample sizes, aggregation rule, and interval where available; it does not report an undefined composite 70%-85% goodness score. Confidence intervals conditional on the controlled corpus likewise do not represent uncertainty across unseen generator processes.

The interpretation of each gate followed a claim ladder. Schema validity was the lowest rung because it showed only that the requested fields could be produced and processed. Range, finite-value, and clear-control checks ruled out several trivial implementation failures. Strong-target discrimination provided the next rung by asking whether the score ordering aligned with the intended construct under designed conditions. Task-family coverage then tested whether the signal appeared outside a single application. None of these rungs included an independent sample, so the combined outcome was classified as internal construct-signal evidence.

Pilot-300 was not used to make a final held-out generalization claim because its generator environment was already part of development. Threshold optimization and cross-head severity rankings were not interpreted as prompt-quality improvements. Pilot-120 subsequently supported exploratory cross-head correlation and construct-classification ablation analyses, described in Section 3.8. Neither study evaluated independent downstream outcomes for the v4 estimator. The earlier pre-v4 action pilot is reported separately in Section 4.4.

The audit further treated recovery as a missingness problem. A failed provider call can be unrelated to prompt content, but it can also correlate with length, complexity, or response structure. The recovered analysis retained every original failure record and documented each additional call. This approach permits analytical completion while preserving information needed to test whether recovered families differ from first-pass families. The present sample of four affected families is too small for a meaningful subgroup estimate, so recovery status is reported descriptively rather than adjusted away.

3.5. Human Review and Key-Assisted Processing

A 30-sample blind-review package assessed formula inputs and auxiliary attribution. Two reviewers independently completed seven judgment fields for H001–H030, yielding 210 field-level comparisons. Reviewer B’s file contains identity and review-time metadata for all rows. Reviewer A’s review-time fields remain blank. Judgment completion and time provenance are therefore reported separately; reviewer identifiers are omitted from the manuscript.

After the independent sheets were returned, key-assisted processing supplied research-key context for disagreements and conservative human-confirmation flags. It preserved both original ratings and did not constitute a third rating. The first author adjudicated all 46 flagged fields, approving final values with an author-supplied timestamp of 2026-08-02 16:56:17; no time zone was recorded. Key-informed first-author adjudication is a potential confirmation-bias channel and is not independent validation.

The present manuscript reports descriptive field-level counts and adjudicated category totals. It does not report a chance-corrected interrater reliability coefficient because class prevalence is uneven and the pilot contains only 30 samples. Exact agreement is used only as a transparent description of whether A and B selected the same category before the key was opened. The final values describe this internal adjudication rather than an independent third rating or an external accuracy criterion.

3.6. Reproducibility and Engineering Checks

The internal reproducibility audit checked configuration and output consistency, row and schema counts, recovery records, and cost accounting. It classified Pilot-300 as an internal pass. Fifty-three fixture tests exercised pipeline behavior, agreement tooling, power-analysis scaffolding, and decision-evaluation infrastructure. The retrospective configuration summary below distinguishes recorded settings from unavailable environment provenance.

Both Pilot-300 and Pilot-120 requested the DeepSeek API model identifier deepseek-v4-flash at https://api.deepseek.com/chat/completions, with temperature 0, max_tokens 700 per head, thinking disabled, non-streaming JSON-object responses, and a 180-second request timeout. top_p was not sent. No provider seed was requested; the documentary seeds are not API sampling seeds. Automatic retries and provider fallback were disabled. Pilot-300 and Pilot-120 r1 used 10 workers; Pilot-120 recovery r2 used four. The client uses Python standard-library urllib.request rather than a third-party provider SDK. Exact historical Python runtime versions were not recorded in the inspected run records; temperature 0 does not guarantee repeatability across provider updates.

The Pilot-300 configuration is dated July 26, 2026, and Pilot-120 r1/r2 measurement records are dated August 6, 2026 (UTC). The prompt version is four-independent-heads-v4. The Pilot-120 configuration documents DeepSeek-V4-Flash-0731 as the served version; this is a recorded configuration label rather than independent verification of provider weights. Configuration hashes and estimator code identity are recorded in the revision evidence manifest. Reproducing the analysis requires the frozen research inputs and outputs, not only the public prototype.

Fixture results are fixture-only evidence. They show that code paths respond as expected to constructed inputs. They do not estimate model performance, statistical power for the eventual design, real reviewer agreement, action regret, or closed-loop improvement. The same boundary applies to the preliminary skill package. It is a prospective interface artifact and differs from the v4 research estimator in context handling, normalization, and fallback behavior. It was not used as a substitute for the frozen estimator in this analysis.

3.7. Prototype Prompt Uncertainty Auditor Skill

The project packages a version of the measurement concept as a prototype prompt-uncertainty-auditor skill. The prototype implementation is available at https://github.com/lovefunCode/Skillsets/tree/main/prompt-uncertainty-auditor. The skill accepts a prompt and optional context, elicits two to five candidate intents and interpretations, identifies fuzzy spans and missing constraints, computes four uncertainty measures, and returns a compact diagnostic report with a “where to optimize” section. Its bundled script normalizes candidate probabilities, computes normalized intent entropy, estimates semantic dispersion from supplied semantic vectors or a deterministic TF-IDF fallback, reports raw De Luca-Termini fuzzy entropy with a separately normalized interpretation, and computes normalized Jensen-Shannon context sensitivity.

This artifact is the prototype diagnostic interface between measurement and action. It identifies the prompt spans, assumptions, or competing intents that a rewrite or clarification should address. A later router can consume the scores and diagnostic fields when choosing Direct, Rewrite, or Clarify. The skill does not yet implement a validated action policy, and it does not reproduce the frozen v4 research estimator exactly. In particular, its semantic-vector fallback, fuzzy-score reporting, and construction of the context distribution can change the meaning of scores. The artifact is therefore versioned as a prototype and must undergo equivalence and regression testing before its outputs can be merged with v4 evidence.

3.8. Prospective Pilot-120 Protocol and Exploratory Analyses

Pilot-120 comprises 120 English prompt families in four task groups: code generation; given-material question answering; information extraction or classification; and instruction, summarization, or structured writing. Each task group contains six families from each of five strata: clear, intent ambiguity, semantic ambiguity, vagueness, and underspecification. Forty families derive from hash-identified public records, 40 are matched derivatives of distinct records, and 40 are researcher-authored. No Pilot-300 or legacy prompt was included. Pre-execution audits covered schema, encoding, source provenance, duplicate and prior-corpus similarity, sensitive patterns, gold-input exposure, and split leakage. Source-derived material remains subject to its source-specific redistribution terms.

Pilot-120 applied the original four-head v4 prompts, schemas, and formulas without tuning to its results. The earlier confirmation protocol specified AUC ≥ .75, a bootstrap lower bound above .50, positive direction in at least four of five tasks, higher target mean and median than clear controls, and clear nonzero rate ≤20%. That protocol also required 300 prompts and 1,200 valid heads. Pilot-120 has four task groups and different cell sizes, so it is a prospective measurement pilot rather than full execution of that protocol. We report its discrimination against the earlier .75 criterion without silently adapting the five-task or sample-size requirements. No public preregistration is claimed. The subsequent construct classifier and the confidence intervals added during manuscript revision are exploratory analyses.

The measurement plan scheduled 480 head calls. A hard stop terminated r1 without automatic retry or fallback. The unresolved set comprised 44 head IDs without locally recorded output and 14 explicitly failed calls. Recovery r2 submitted each of these 58 IDs once under a separate run ID, with four workers. Missing local records could duplicate requests in flight at r1 termination. Canonical selection used r2 for those 58 IDs and r1 for all others, irrespective of apparent score quality. A persistent schema failure was retained without manual repair or a further rerun. Complete-case analyses excluded the one family lacking a valid semantic head from every feature-set comparison, preserving paired samples.

Construct classification used a custom multinomial logistic (softmax) implementation in Python, with five predefined stratum labels. Each of four folds held out one entire task group. Means and population standard deviations were estimated only on the training fold, with a scale of one for a constant feature. Weights and intercepts started at zero. Full-batch gradient descent ran for 2,400 epochs with learning rate 0.05/sqrt(1 + epoch/200), L2 coefficient 0.1 on non-intercept weights, and no class weighting. The procedure is deterministic and requires no random seed. Models used no scores (intercept only), each score alone, all four scores, and each three-score leave-one-head-out set. No hyperparameter search was performed by the analysis script. Out-of-fold predictions were pooled across all 119 complete families; macro-F1 averaged five class-specific F1 values, assigning zero where undefined. This predicts constructed strata, not task quality.

Supplementary confidence intervals were computed during manuscript revision using 4,000 resamples and NumPy default_rng seed 20260726, with percentile endpoints at 2.5% and 97.5%. AUC resampling retained the target and clear sample sizes separately; ties received half credit. Pilot-300 all-target intervals and Pilot-120 intervals quantify variation conditional on the observed corpus. For the semantic ablation, paired out-of-fold prediction records were resampled within task-by-stratum cells. The same sampled families were used for both feature sets; models were not refitted in each resample. This interval is conditional on the fitted folds and does not capture training-set, provider-run, or new-task variation. No significance claim or multiplicity-adjusted hypothesis test is made.

Pilot-120 also generated Direct, Rewrite, and Clarify outputs under frozen action templates. An interrupted first run recorded 186 episodes; targeted recovery generated the remaining 174. The combined package contains 360 episodes, 480 distinct provider response IDs, and 120 complete within-family action matrices. Six calls reached the 700-token output limit and were retained. Generation completeness is an engineering result; the outputs lack independent outcome scores and therefore cannot establish utility, routing performance, or prompt improvement.

3.9. Ethics, Data Governance, and AI Assistance

Pilot-300 used controlled synthetic prompts; Pilot-120 additionally used public-derived and researcher-authored material. The internal review involved colleagues judging study materials. No formal institutional determination for this activity is documented, so neither ethics exemption nor a formal determination that it falls outside human-subjects research is claimed. The manuscript reports aggregate judgments, omitting reviewer identifiers. Identifiable review records, credentials, provider secrets, and blinding keys require separate access controls and are excluded from the proposed public materials.

OpenAI Codex supported artifact organization, drafting and revision, consistency checks, figure preparation, and simulated pre-submission review. AI-assisted text and checks were treated as drafts or analytical aids, not independent empirical evidence. The authors retain responsibility for verifying the analyses, citations, and final submitted content.

3.10. Planned Independent Outcome Evaluation

Future confirmation must freeze any redesigned construct definitions and estimator version before collecting a new corpus. Semantic alignment review should first determine which interpretations remain materially live under explicit context. The existing failure must remain part of the evidence record. Expanded human assessment should use independent reviewers, documented provenance, and any required institutional review.

Independent action evaluation will compare Direct, Rewrite, and Clarify on the same prompts using frozen task-success, intent-alignment, unsupported-assumption, cost, and interaction-burden outcomes. Fixed policies, surface features, no-formula models, single-head and leave-one-head-out variants, and generic prompt optimizers provide matched baselines. Span repair requires a top-span versus matched-control intervention. Reverse optimization will revise versioned components only when independent utility and clear-prompt regression criteria are met; diagnostic score reduction alone is not a success criterion. These evaluations remain unrun.

4. Results

4.1. Execution Integrity and Analytical Completeness

All 300 Pilot-300 prompts reached analytical completion. Across four heads, the recovered analytical file contained 1,200 valid schemas. The immutable first pass completed 296 of 300 prompt families. Targeted recovery made five head calls across the four incomplete families, rather than rerunning all 16 head outputs for those families. This distinction matters: the 300-of-300 figure describes the recovered analytical dataset, not first-pass service reliability.

Four prompts generated auxiliary attribution warnings, a rate of 1.333%. The warning rate was below the protocol’s 10% threshold for allowing core-formula analysis to proceed. It does not show that the warned or unwarned spans were correct. The internal audit also verified that the expected analytical records and schemas were present after recovery.

The recovered Pilot-300 full run had an estimated provider cost of approximately USD 0.117; the micro run and recovered full run together were approximately USD 0.140. These are historical run-ledger estimates, not deployment prices or invoice-certified totals.

4.2. Internal Construct-Signal Results

All four heads separated their strong construct-target examples from clear controls within Pilot-300. U_intent achieved ROC AUC = .88, 95% bootstrap CI [.78, .96], with positive signals in five of five task families. U_sem achieved AUC = .80, 95% CI [.70, .90], with positive signals in four of five task families. U_fuzzy achieved AUC = 1.00, 95% CI [1.00, 1.00], with positive signals in five of five task families. U_context achieved AUC = .96, 95% CI [.90, 1.00], also with positive signals in five of five task families.

Table 1. Internal Strong-Target Versus Clear-Control Separation. 

Estimator head	Operational target	Strong target, n	Clear control, n	ROC AUC	95% bootstrap CI	Positive task families
U_intent	competing intents	25	75	.88	[.78, .96]	5/5
U_sem	semantically distinct interpretations	25	75	.80	[.70, .90]	4/5
U_fuzzy	uncertain category boundaries	25	75	1.00	[1.00, 1.00]	5/5
U_context	sensitivity to missing context	25	75	.96	[.90, 1.00]	5/5

Note. Results are internal development-set estimates on controlled synthetic templates. Confidence intervals are percentile intervals from 4,000 stratified-within-class bootstrap resamples and quantify within-corpus sampling variation only. They are not template-independent confirmation.

Table 1. Internal Strong-Target Versus Clear-Control Separation. 

Estimator head	Operational target	Strong target, n	Clear control, n	ROC AUC	95% bootstrap CI	Positive task families
U_intent	competing intents	25	75	.88	[.78, .96]	5/5
U_sem	semantically distinct interpretations	25	75	.80	[.70, .90]	4/5
U_fuzzy	uncertain category boundaries	25	75	1.00	[1.00, 1.00]	5/5
U_context	sensitivity to missing context	25	75	.96	[.90, 1.00]	5/5

Note. Results are internal development-set estimates on controlled synthetic templates. Confidence intervals are percentile intervals from 4,000 stratified-within-class bootstrap resamples and quantify within-corpus sampling variation only. They are not template-independent confirmation.

The broader all-target versus clear-control comparison was weaker: U_intent = 0.627 [0.580, 0.673]; U_sem = 0.600 [0.553, 0.647]; U_fuzzy = 0.860 [0.807, 0.907]; U_context = 0.740 [0.687, 0.794]. Each contrast contains 75 target-labelled and 75 clear prompts; intent and semantic scores use the broad ambiguity class. These supplementary 95% intervals were added during revision using the procedure in Section 3.8. The strong-target comparison is a favorable designed contrast, while the all-target comparison exposes heterogeneity within the broader labels.

Clear controls had a nonzero rate of zero for each head in the audited summary. Within the controlled construction, this satisfies the negative-control gate. Exact zero behavior may also indicate that clear templates contain cues that are easier to recognize than natural clear prompts. It must therefore be reproduced in a corpus that does not share the development templates.

The U_fuzzy result requires particular caution. AUC = 1.00 describes exact rank separation in this internal contrast. It should not be called a perfect measure. The result can arise because the strong fuzzy items expose prominent boundary language or because the estimator prompt channels a predictable response. Template-independent data, surface-form perturbations, and simple cue baselines are needed before a stronger interpretation is defensible.

4.3. Key-Assisted Human-Review Status

The completed content overlap contains Reviewer A and Reviewer B judgments for H001–H030. Across 210 field-level comparisons, 169 fields agreed and 41 disagreed, for an overall descriptive exact agreement of 80.5%. Agreement varied by field: candidate-intent content and probability judgments each agreed on 24 of 30 samples (80.0%); semantic interpretations agreed on 28 (93.3%); fuzzy-boundary probes on 26 (86.7%); context and missing-slot judgments on 23 (76.7%); uncertain-span attribution on 25 (83.3%); and overall core-measurement acceptance on 19 (63.3%).

The conservative v2 triage rule flagged every A/B disagreement and every agreement on a non-yes value. This produced 46 field rows requiring human confirmation. The difference between 46 triage rows and 41 disagreements consists of five non-yes agreements that still required substantive verification. All 46 rows were subsequently marked approved with a final value: 17 yes, 9 partial, and 20 no. None of the final values introduced a category absent from both original reviewers.

Combining those 46 values with the 164 unflagged A/B agreements on yes produced 181 yes, 9 partial, and 20 no outcomes across all 210 fields. The corresponding yes proportion was 86.2%, and the yes-or-partial proportion was 90.5%. By field, final yes proportions were 83.3% for candidate-intent content, 83.3% for candidate-intent probabilities, 100.0% for semantic interpretations, 93.3% for fuzzy-boundary probes, 86.7% for context and missing slots, 83.3% for uncertain-span attribution, and 73.3% for overall core-measurement acceptance.

Table 2. Field-Level Blind-Review Agreement and Final Internal Adjudication. 

Judgment field	Pre-key A/B agreement, n (%)	Final yes, n (%)	Final partial, n	Final no, n
Candidate-intent content	24 (80.0)	25 (83.3)	2	3
Candidate-intent probabilities	24 (80.0)	25 (83.3)	2	3
Semantic interpretations	28 (93.3)	30 (100.0)	0	0
Fuzzy-boundary probes	26 (86.7)	28 (93.3)	2	0
Context and missing slots	23 (76.7)	26 (86.7)	2	2
Uncertain-span attribution	25 (83.3)	25 (83.3)	1	4
Overall core-measurement acceptance	19 (63.3)	22 (73.3)	0	8
All fields	169/210 (80.5)	181/210 (86.2)	9	20

Note. Each field-specific denominator is 30. Pre-key agreement describes whether Reviewers A and B selected the same category before key-assisted processing. Final values combine unflagged yes agreements with 46 key-assisted, human-approved adjudications. They are internal quality-control judgments, not estimator-accuracy estimates or external validation.

Table 2. Field-Level Blind-Review Agreement and Final Internal Adjudication. 

Judgment field	Pre-key A/B agreement, n (%)	Final yes, n (%)	Final partial, n	Final no, n
Candidate-intent content	24 (80.0)	25 (83.3)	2	3
Candidate-intent probabilities	24 (80.0)	25 (83.3)	2	3
Semantic interpretations	28 (93.3)	30 (100.0)	0	0
Fuzzy-boundary probes	26 (86.7)	28 (93.3)	2	0
Context and missing slots	23 (76.7)	26 (86.7)	2	2
Uncertain-span attribution	25 (83.3)	25 (83.3)	1	4
Overall core-measurement acceptance	19 (63.3)	22 (73.3)	0	8
All fields	169/210 (80.5)	181/210 (86.2)	9	20

Note. Each field-specific denominator is 30. Pre-key agreement describes whether Reviewers A and B selected the same category before key-assisted processing. Final values combine unflagged yes agreements with 46 key-assisted, human-approved adjudications. They are internal quality-control judgments, not estimator-accuracy estimates or external validation.

The adjudicated values describe internal quality control rather than estimator accuracy. Agreement with the research key does not establish that the key is substantively correct. Reviewer A’s completion-time metadata remain missing, and the first-author adjudication timestamp lacks a time zone. Independent, template-independent human replication has not been completed.

The complete 30-sample overlap and adjudication remove the earlier missing-subset and pending-final-value limitations, but they do not close the broader human-validation gate. No chance-corrected reliability coefficient, independently replicated adjudication result, or human-valid causal attribution result is claimed.

4.4. Historical and Fixture Evidence

The legacy 36-item protocol generated 315 requests, of which 268 were scorable and 47 remained persistently truncated. The run is useful for documenting failure modes and motivating stronger recovery controls. It cannot be merged with Pilot-300 because the protocol, units, and execution conditions differ. It also executed uncertain_version only. The legacy outputs therefore cannot support a clarified-versus-uncertain causal comparison.

A separate pre-v4 Pilot-300 action pilot used 300 controlled synthetic prompt families, with Direct, Rewrite, and Clarify generated for every prompt. All 900 episodes produced outputs and machine scores. The action generator used DeepSeek-V4-Pro, while DeepSeek-V4-Flash supplied the earlier formula inputs and machine outcome judgments. The dataset, manifest, raw outputs, scores, deterministic analysis, and audit hashes are retained; the structural audit passed 27 of 27 checks. Because the action pilot preceded the repaired v4 estimator, used template-generated rubrics, and relied on same-family machine judging rather than blinded independent outcome assessment, it is classified as exploratory historical-development evidence.

Across the pre-v4 action pilot, mean balanced utility was .7784 for Direct, .7782 for Rewrite, and .6456 for Clarify. The paired Direct-minus-Rewrite difference was .00025 with a bootstrap 95% interval of [-.0295, .0301], so those two actions were not distinguishable under the pilot’s utility definition. Clarify incurred more turns and tokens, and its lower aggregate utility cannot be generalized beyond the frozen synthetic prompts, machine judge, and balanced weights. The empirically best action nevertheless varied by prompt: Direct for 171, Rewrite for 80, and Clarify for 49 of 300 families.

The stricter router result was negative. The 60-prompt test split contained an unseen task family, information extraction, but task and split were therefore confounded. Traditional-feature, four-formula-only, traditional-plus-four-formula, and all four leave-one-formula-out routers each predicted Direct for all 60 cases. Every model obtained accuracy .617, macro-F1 .254, and mean regret .161 because Direct was the empirical best action for 37 of 60 test prompts. Removal of any one formula caused no measurable test loss, and adding all four formulas did not improve over traditional features. This majority-action collapse is evidence against incremental-value wording for the pre-v4 operationalization; it is not a test of the repaired v4 estimator.

All 53 fixture tests passed in the current engineering package. This result supports the availability of testable code paths for agreement, power, routing, and safety checks. No fixture value is reported as empirical reviewer agreement, final sample size, routing accuracy, regret reduction, or closed-loop gain.

4.5. Prospective Pilot-120 Measurement and Construct Ablation

The canonical Pilot-120 package contains 480 distinct head IDs, 479 schema-valid heads, and 119 complete four-head families. The persistent failure, P120-IE-IAM-01__u_sem, returned multiple readings with the schema-reserved label ambiguity_kind=none. It was retained as invalid under the policy in Section 3.8. The complete-case intent contrast therefore includes 23 target families, while the other target groups and clear controls each include 24. The combined locally recorded cost lower bound, including action generation, was approximately CNY 1.075; possibly unrecorded r1 in-flight responses prevent an invoice-total interpretation.

Table 3 reports prospective discrimination with supplementary 95% confidence intervals. All four AUC point estimates fall below the earlier .75 confirmation criterion; none can be described as confirmed by this pilot. The semantic failure is additionally structural: all 24 semantic-ambiguity targets produced one live reading and U_sem=0, while three clear prompts produced nonzero scores. The head removes readings resolved by explicit context, whereas each semantic gold item lists two plausible readings. Whether those readings remain materially live requires human adjudication. The 24-row review queue remains pending, so the failure cannot yet be assigned exclusively to labels, elicitation, or the diversity formula.

The full-four classifier obtained accuracy .294 and macro-F1 .223, compared with .202 and .108 for the intercept-only model. Single-score macro-F1 values were .189 for intent, .085 for semantic, .194 for fuzzy-boundary, and .141 for context. Removing intent, semantic, fuzzy-boundary, or context changed macro-F1 by −.005, +.018, −.036, and −.027, respectively. For removal of the semantic score, the paired 95% bootstrap interval was [−.028, .059], conditional on fixed out-of-fold predictions. The point estimate favors removal, but the interval does not establish a reliable improvement or equivalence. There is no demonstrated incremental benefit of U_sem in this comparison. Pairwise absolute Spearman correlations were at most .355. Diagonal entries of the inverse Spearman correlation matrix ranged from 1.027 to 1.186; these are rank-based collinearity diagnostics, not conventional raw-feature regression VIFs. Low collinearity alone cannot explain or resolve the construct failure.

5. Discussion

5.1. What the Internal Results Establish

The v4 estimator operationalized four prompt-side uncertainty measures with complete analytical coverage after targeted recovery. Each head produced a construct-aligned signal against clear controls, and the pattern appeared across multiple task families. The strongest supported conclusion is therefore one of internal feasibility: the four-head design can be executed reproducibly and can distinguish designed strong-target examples from controlled clear prompts.

This conclusion has more content than a software smoke test. Valid JSON alone would show only that a provider followed an output schema. The ROC analyses connect elicited inputs and formulas to predefined contrast groups. Positive-task coverage tests whether a signal is confined to one task family. Clear controls test whether scores fire indiscriminately. The combination supports continued validation of the measurement design.

The conclusion remains narrower than construct validation in ordinary use. Pilot-300 was generated under controlled templates, the study protocol was developed within that environment, and the separately constructed Pilot-120 pilot is not a representative naturalistic sample. The development process may have selected formula inputs or head instructions that fit the available conditions. Confidence intervals quantify uncertainty within the observed sample, not uncertainty across future domains or generation processes.

The difference between strong-target and all-target AUCs reinforces this limit. U_sem, for example, separated strong interpretation cases from clear controls more effectively than it separated every target-labelled case from clear controls. This pattern can reflect variation in construct intensity, overlap among constructs, or imprecision in broad condition labels. It also suggests that binary target labels are an incomplete representation of prompt-side uncertainty. Confirmatory work should retain graded severity and co-occurrence rather than forcing prompts into a single category.

5.2. Interpreting the Four Heads

U_intent and U_sem address related but separable questions. Intent entropy concerns the distribution over plausible user goals. Semantic diversity concerns how consequentially different the interpretations are. Two candidate intents can be labeled differently yet produce nearly equivalent actions, or they can have similar probabilities and lead to incompatible outputs. Rao diversity introduces distance so that probability dispersion is weighted by interpretive difference.

U_fuzzy addresses boundary uncertainty rather than candidate multiplicity. Its exact separation is the highest internal result and the one most vulnerable to template explanation. Controlled synthetic prompts may use overt gradable adjectives or missing thresholds that are easy for the fuzzy head to recognize. Natural prompts may express the same problem through indirect context, domain conventions, or numeric values whose boundary depends on a community. The next study should compare U_fuzzy with a lexical baseline and test paraphrases that remove overt boundary words without changing the underlying decision.

U_context estimates how much a context completion changes a relevant distribution. Its high internal AUC suggests that the head detected designed missing-information cases. Yet context sensitivity can be difficult to delimit. Almost any task can improve with additional detail, and a model can invent a highly consequential completion even when the observed prompt is adequate. Human review must therefore assess whether the missing information is material to the requested decision, not merely whether more information could be imagined.

The independent-head design makes failure modes inspectable, but independent execution does not imply independent constructs. Real prompts can activate several heads for defensible reasons. A vague goal such as ’make it professional’ may also hide the intended audience and deliverable. Confirmatory analysis should report the correlation matrix, variance-inflation factors or condition numbers, conditional predictive contribution, single-measure performance, and four leave-one-measure-out ablations. If two heads are nearly interchangeable or removing one does not reduce independent predictive or decision value, the paper should describe redundancy and simplify the framework rather than preserving four dimensions by definition. The target is useful decomposition, not artificial orthogonality or a predetermined head count.

Measurement validity will require evidence from several directions. Convergent evidence would show that a head correlates with independent human judgments of its intended construct and responds predictably when that construct is strengthened or resolved. Discriminant evidence would show that the same manipulation does not mechanically increase every head. Known-groups evidence would compare carefully designed positive and negative cases without reusing development templates. Consequential evidence would test whether using the score improves a later decision without imposing unacceptable costs. The current study supplies a first known-groups result, but it does not complete the other forms of validation.

This distinction matters because the formulas can appear more objective than the elicitation process warrants. Entropy, diversity, fuzzy entropy, and divergence are deterministic once their inputs are fixed. The difficult step is obtaining candidate intents, probabilities, distances, memberships, and context-completed distributions that correspond to human-interpretable constructs. A future study should therefore evaluate both the final scalar and the intermediate payload. Two heads can produce similar scores for different reasons, and those reasons determine whether a rewrite or question would address the problem.

5.3. Human Review as a Separate Validity Gate

The blind-review work illustrates why key-assisted processing cannot replace adjudication. The processing rule identified that 169 of 210 comparable fields matched and that 46 rows merited confirmation. A human then approved final values for all flagged rows. This separation matters because the research key encodes the study designers’ expected interpretation and can itself be wrong, incomplete, or too rigid for a defensible alternative.

The 30-sample overlap and field-level adjudication are complete, but a headline coefficient would still require care. Several fields have strongly imbalanced yes/no/partial prevalence, and overall acceptance shows substantially lower exact agreement than semantic interpretation. A single pooled coefficient could conceal those differences. Adjudicator identity and the supplied completion time are now recorded, but Reviewer A’s missing completion-time metadata and the unspecified adjudication time zone leave the review package short of full traceability.

The appropriate next step is procedural and confirmatory. Reviewer A’s completion time should be filled or documented without changing judgment content, and the source files should remain locked. Agreement should be reported by field and class with uncertainty intervals and prevalence. A larger, template-independent sample is then needed before the study characterizes how consistently the auxiliary attribution aligns with human judgment.

Span validity also requires an intervention test. Agreement that a highlighted phrase looks relevant is correlational evidence. A stronger claim would require showing that resolving the top-ranked span changes the relevant uncertainty measure or improves an outcome more than a matched random edit. Until that experiment is run, the manuscript describes spans as auxiliary attributions, not causal explanations.

5.4. Implications for Direct, Rewrite, and Clarify Decisions

The four measures create candidate inputs for an action policy, but no current v4 result shows that the policy reduces regret. The pre-v4 action pilot strengthens this boundary by supplying a direct negative result: every evaluated router, including the four-formula and leave-one-formula-out variants, collapsed to always Direct on the task-held-out test. That finding rejects an incremental-routing claim for the earlier operationalization. It neither proves that the four theoretical constructs are useless nor licenses transferring v4’s stronger internal construct separation into an action benefit. This distinction follows from the clarification and abstention literature. Recognizing uncertainty can support selective behavior (Kim et al., 2024; Tomani et al., 2024; Zhang & Choi, 2025), but the preferred behavior depends on the stakes, task, utility weights, and interaction context.

A low-cost clarification may dominate direct response when one missing variable determines the entire answer. A direct response with an explicit assumption may dominate clarification when the task is reversible and delay is expensive. Rewriting may help when the user’s wording can be made explicit without requesting new information. The same uncertainty score can therefore correspond to different actions under different utility profiles.

Pilot-120 has already generated all three actions for each prompt. The next step is independent scoring of their task performance, unsupported assumptions, interaction burden, token and latency costs, and domain-specific risks. A complete output matrix is not a reward matrix. Once the scoring rubric and utility profiles are frozen, router evaluation should emphasize mean and upper-tail regret; action-label accuracy alone ignores near-ties and harmful errors.

The planned comparison matrix includes fixed policies (always Direct, always Rewrite, always Clarify), surface-feature routing based on length, readability, and ambiguity lexicons, a generic uncertainty score, a zero-shot language-model judge, intent-entropy-only thresholding, a generic rewrite without uncertainty or span signals, a generic prompt optimizer, and a four-measure policy without feedback. The full method is compared with each baseline under the same prompt instances, model/provider settings, action-generation budget, output limits, reward components, and blinded evaluation. Four single-measure models and four leave-one-measure-out models test necessity and incremental value; top-span versus length-matched random-span repair tests attribution. Section 4.4 reports the narrower pre-v4 router and leave-one-out comparison, whose negative result motivated estimator repair. The full current-v4, template-independent, independently scored comparison has not been completed, so the present manuscript provides its comparison contract rather than a positive performance conclusion.

5.5. Reproducibility, Costs, and System Boundaries

Separating first-pass outputs from recovery improved the interpretability of execution reliability. Reporting 300 complete prompts without noting that four families required recovery would obscure a service-level failure. Reporting only 296 first-pass families would discard valid targeted recovery and understate analytical coverage. Both numbers are needed because they answer different questions.

The cost records show that the internal experiment was inexpensive at its observed scale, but they should not anchor deployment projections. Pricing can change, and natural prompts may require longer outputs, retries, or additional controls. A four-head design also creates more calls than a single-head design. Confirmatory evaluation should report latency and cost distributions, not only totals, and should test whether heads can be combined without degrading construct separation.

The prototype skill is the paper’s prototype diagnostic artifact rather than a purely conceptual appendix. It provides a reusable interface for diagnosing a prompt, reporting four measures, identifying candidate spans for inspection, and proposing a rewrite. That interface makes the measurement theory actionable and gives future Direct/Rewrite/Clarify routing a stable input contract. It is not the validated v4 estimator. Differences in normalization, U_context handling, semantic fallback, and output conventions can change scores and downstream decisions. A deployment artifact should either reproduce the frozen research implementation or undergo a new validation cycle. Version identity must be treated as part of the evidence chain.

The prototype also supplies the unit of change for reverse optimization. Adjudicated errors should update the narrowest responsible component: elicitation instructions, formula input validation, span localization, score normalization, routing thresholds, or utility weights. Each update becomes a candidate skill version with an explicit failure hypothesis and regression tests. Validation-gated acceptance and rollback prevent a correction for one failure family from silently degrading clear prompts or another task family. This loop is specified but has not been executed as an effectiveness experiment.

5.6. Limitations

The main limitation is construct and corpus scope. Pilot-300 contains controlled synthetic English templates across five task families, and the development process touched its generator environment. Lexical or structural cues may explain part of its internal separation, particularly the fuzzy-head result. Pilot-120 uses separately sourced, derived, or authored material across four task groups, but its balanced construction and small cells do not represent natural prompt prevalence.

Both corpora are English-only and omit multilingual interaction, multimodal inputs, long conversational histories, and many professional conventions. Pilot-120 is not a full execution of the earlier five-task, 300-prompt confirmation protocol. Its complete-case analysis also omits one intent-ambiguity family owing to a semantic schema failure. The confidence intervals condition on the observed corpus and do not quantify uncertainty across new tasks, source populations, or repeated provider runs.

Third, formula inputs are model-elicited. A mathematically correct entropy or divergence can still summarize poorly formed candidate sets, probabilities, distances, memberships, or context completions. The study evaluates the combined elicitation-and-formula pipeline, not the formula in isolation.

Fourth, clear-control zeros may reflect construction artifacts. Natural clear prompts can contain words associated with ambiguity or omit optional details without becoming underspecified. Negative controls must be expanded to include adversarially clear and superficially ambiguous examples.

Human evidence is limited to a 30-sample internal review and first-author, key-informed adjudication. Reviewer A lacks review-time metadata; the adjudication timestamp lacks a time zone. Shared assumptions and access to the intended interpretation can favor the study design. Independent replication, per-class uncertainty assessment, and a causal span-intervention study remain outstanding.

Provider nondeterminism and recovery can introduce selection effects. The documented model identifier and configuration are part of the measurement instrument; provider-side updates can alter elicited inputs even when prompts and temperature remain unchanged. The public prototype also differs from the frozen estimator in semantic fallback, fuzzy normalization, and context construction. Reproducing research scores requires the research implementation and canonical outputs. Repeated runs and explicit version provenance are needed to assess stability.

Seventh, the legacy protocol cannot provide a causal comparison because it ran uncertain_version only and retained 47 persistent truncations. Its evidence is historical. Fixture tests likewise cannot replace empirical observations.

The study does not establish four-measure sufficiency, downstream prompt improvement, or lower-regret v4 routing. The pre-v4 exploratory routers collapsed to always Direct on their held-out task. Pilot-120 provides weaker construct discrimination and no demonstrated incremental value for the semantic score; its small positive deletion effect has an interval crossing zero. These observations narrow the current operationalization but do not show that semantic ambiguity is irrelevant in general. Independent outcome evaluation remains necessary.

5.7. Redesign and Independent Retesting

Pilot-120 identifies a specific redesign question: when gold annotations list two readings, do both remain defensible after applying explicit context? Human review should answer that question before revising either the data labels or the head. Candidate corrections must be frozen and tested on new material. Reusing Pilot-120 to tune the head and then reporting it as confirmation would erase the prospective character of the original failure.

The methodological priority is to separate construct repair from empirical confirmation. Future designs should align task coverage, sample size, recovery policy, and decision criteria before execution. The earlier five-task confirmation gate cannot be silently converted into a four-task rule after seeing Pilot-120. Both failed and successful comparisons should remain visible, with independent source and task variation in the next corpus.

Pilot-120 action outputs remain available for future independent evaluation, subject to governance and scoring requirements. Evaluators should be blinded to condition, head set, action, optimizer, model/provider, and pair relation where feasible. Unscored action generation supplies no evidence that rewriting improves prompts or that a router selects useful interventions.

Reviewer A’s remaining time provenance should be completed, and the adjudicated results should be replicated on a template-independent sample before the key is used for broader substantive conclusions. Formal agreement should be accompanied by per-class counts because a single coefficient can conceal imbalanced judgments. The top-span intervention should test whether attribution supports repair.

Only after the measurement and human-validation gates should the project evaluate actions. The complete same-prompt Direct/Rewrite/Clarify reward matrix and independent outcome rubric must be frozen before router fitting. Full-four, single-measure, leave-one-measure-out, no-formula, surface-feature, zero-shot, generic-rewrite, generic-optimizer, and fixed-policy conditions should be evaluated under matched generation and scoring controls. The central success criterion is lower regret or higher independent utility without an unacceptable clear-prompt regression, not a lower uncertainty score by itself. Reverse-optimization proposals should then be generated from adjudicated failure classes, applied to versioned skill components, and accepted through validation data only. Every accepted change remains reversible, and a final held-out test is accessed once after decisions are fixed.

6. Conclusion

Four diagnostic heads produced auditable prompt-side uncertainty scores and strong internal separation under controlled Pilot-300 templates. After targeted recovery, all 300 prompts had four valid head outputs; strong-target AUCs ranged from .80 to 1.00. These results establish internal feasibility under the sampled construction, not general construct validity, four-measure necessity, or overall prompt quality.

Prospective Pilot-120 measurement narrowed that result. All four target-versus-clear AUCs were below the earlier .75 criterion, and the semantic head returned zero diversity on every semantic target. Removing that head changed macro-F1 from .223 to .241, but the paired interval included zero. The corpus/head mismatch requires adjudication and independent retesting. Internal key-assisted review and engineering checks do not substitute for that evidence.

The contribution is an inspectable measurement framework with bounded internal signal, a prototype diagnostic interface, and preserved prospective failures. It provides no validated prompt optimizer or v4 action policy. Independent outcome scoring, prototype-equivalence testing, and new confirmation after redesign remain necessary before claiming practical prompt improvement or broad generalization.