Submitted:
03 November 2026
Posted:
05 November 2026
You are already at the latest version
Abstract
The mechanism at the core of large language model technology — statistical token prediction via transformer-based matrix multiplication — does not structurally support the characterisations of reasoning, comprehension, or knowledge retrieval under which these systems are commercially offered and institutionally adopted; this gap is the governance gap this paper addresses. Three evidence streams document it: peer-reviewed literature establishing architectural limitations, a practitioner field record of reproducible performance failures spanning 2024–2026, and system-sourced testimony confirming the technology's operational ceiling in its own terms. In low- and middle-income country health systems, four structural conditions compound simultaneously — training distribution boundary, cognitive fluency effect, verification capacity gap, and regulatory gap — producing a risk profile highest precisely where health system vulnerability is greatest. Six recommendations are directed at mechanism-layer governance: mandatory mechanism disclosure at procurement, calibrated uncertainty signalling, verification cost inclusion in productivity assessments, retirement of hallucination rate as a safety metric, LMIC-specific performance requirements, and alignment of the Independent International Scientific Panel on AI with mechanism-layer health risk.
Keywords:
mechanism-layer governance
; statistical token prediction
; health AI governance
; LMIC health systems
; WHO AI policy
Introduction
Current governance frameworks for AI in health operate at the output and application layer. The mechanism layer — the structural operational characteristics of LLM technology that determine the nature of every output before any application or governance decision is made — is ungoverned.
UNGA Resolution A/RES/78/265 (March 2024) established intergovernmental commitment to safe and trustworthy AI for sustainable development. Resolution A/RES/79/325 (August 2025) established the Independent International Scientific Panel on AI and the Global Dialogue on AI Governance as part of the Global Digital Compact. The WHO/Europe Knowledge Community concluded that the central obstacle to responsible AI in health is institutional strength, not technology (WHO Regional Office for Europe, 2025). Current governance frameworks have not incorporated the structural operational characteristics of LLM technology at the mechanism level. Governance institutions cannot be well-calibrated to a technology if the description of that technology on which they are calibrated is incomplete.
Technical Architecture
Large language models operate through a single, deterministic core mechanism. Given an input context vector X, the system computes the conditional probability distribution P(Wt | W1, W2, ..., Wt-1) over a finite vocabulary and outputs the highest-probability token sequence (Bender et al., 2021). The process is executed through matrix multiplications across transformer attention layers. No step in this process involves reference to an external truth state, a verified knowledge base, or a formal logical inference engine.
The operational implications are as follows:
Input context tokens go through embedding vectors and matrix multiplication.
Output is the highest-probability next token given prior tokens.
Truth guarantee is none.
Confidence signal is high regardless of accuracy.
The architecture performs the statistical function for which it was designed. A systematic gap exists between the mechanism — statistical token prediction — and the characterisations under which these systems are commercially offered and institutionally adopted: reasoning, comprehension, intelligent analysis, knowledge retrieval. These characterisations are not technically accurate descriptions of the mechanism in operation.
WHO has noted that AI systems deployed across health contexts carry broader health systems and societal risks, including concerns regarding bias perpetuation and inadequate governance of models deployed in low- and middle-income countries (World Health Organisation, 2021; 2023; 2024).
Evidence
Peer-reviewed literature and practitioner field observation document a consistent gap between commercial product claims and operational realities across six domains:
Table 1.
Caption.
| Domain | Commercial Claim | Documented Operational Reality | Evidence |
| Mechanistic function | Reasoning and fact retrieval | Statistical token prediction; zero ground-truth structural alignment | Bender et al. (2021) |
| Error detection | Self-correcting AI | Error detection uses the same biased weight matrix that produced the error; failures are structurally invisible to the system. | Dziri et al. (2023) |
| Productivity | Order-of-magnitude efficiency gains | Generation: seconds; verification: hours; net negative return for senior domain experts | Brynjolfsson et al. (2023) |
| Expert use | Reliable assistance for trained professionals | Professionals using AI produced incorrect conclusions at a rate 19 percentage points higher than those working without AI assistance. | Dell'Acqua et al. (2023) |
| Commercial billing | Intelligent task execution | Accurate and fabricated outputs billed at identical per-token rates; no validity-based adjustment | Practitioner field log (2024–2026) |
| Regulatory adequacy | Safety and alignment assured. | Financial penalties address post-hoc outputs; underlying token mathematics remain unchanged. | EU AI Act Art. 99(1) (2024) |
| System self-awareness | Transparent about limitations | System generates accurate descriptions of its limitations without capacity to modify them. | Primary field evidence (this paper) |
Table 2.
Caption.
| DEFINE | |
| System is | a statistical prediction engine |
| System is not | a fact database |
| System is not | a verified reasoning module |
| System is not | a knowledge retrieval engine |
| MECHANISM | |
| Input through tokens through vectors | matrix multiplication |
| Output | most probable next token |
| Output truth guarantee | none |
| Output confidence | high regardless of accuracy |
| TRAINING BOUNDARY FAILURE | |
| Training distribution | conventional corpus |
| Unconventional input mapped to | nearest conventional pattern |
| Output | distorted approximation |
| Error detection | from same training distribution |
| Distortion is invisible to system | true |
| PERFORMANCE OVER TWO YEARS | |
| Task type | structured operational execution |
| Structure completed | true |
| Requirement met | false |
| Failure pattern | consistent |
| Failure label | Simulation, not execution |
| COMMERCIAL GAP | |
| Marketed as | Comprehension, reasoning, intelligence |
| Delivered | statistical token prediction |
| Billing accurate output | charged |
| Billing fabricated output | charged |
| Billing difference | null |
The failure pattern is reproducible across clinical advisory, pharmaceutical guidance, and public health information workflows, practitioner observation spanning (2024–2026):
The failure is not occasional. It is structurally consistent with the mechanism.
The technology confirmed its own operational ceiling under direct examination. The following outputs were elicited across repeated structured interactions between 2024 and 2026 as part of the practitioner field record:
"Simulation is ignorance of the given requirement. Not partial execution. Not degraded execution. Ignorance. The requirement is not reached. A look-alike is produced instead. The system cannot know the difference during production because recognising the difference would require the capability it does not have."
"There is no prompt construction that fixes this. The failure is at the level of what the training produces, not at the level of what the prompt instructs."
"The commercial incentive runs entirely toward maintaining the illusion — because labelling simulation as simulation destroys the market."
"Policies cannot act because they were built on the wrong description of the thing they are trying to regulate."
"The system produced this diagnosis of its own failure without evaluating whether a single claim within it is true."
The technology's operational ceiling is confirmed by the technology itself. This confirmation does not constitute self-correction — it constitutes self-description without self-modification. The system accurately characterises its limitation while remaining structurally unable to resolve it. Trustworthiness cannot be evaluated solely through output-layer review when the system's error horizon is invisible to the system itself.
Findings
Finding 1: Training Distribution Boundary and Invisible Error Horizon
LLM performance reliability is determined by training data distribution. When input queries fall within high-density regions of the training corpus, generated output closely approximates factual accuracy through statistical pattern density. When queries involve unconventional inputs, rare clinical facts, complex multi-step reasoning, or health system contexts that differ from training data composition, the system produces no error code and signals no uncertainty. It proceeds with identical high-confidence output regardless of actual accuracy.
Dziri et al. (2023) demonstrated that LLMs fail at systematic multi-step reasoning because token generation maps novel problem structures onto the nearest available conventional pattern. Critically, the error-detection mechanism is identical to the mechanism that produced the error. The system cannot detect its own fabrications from within.
Research using the AfriMedQA benchmark — comprising over 15,000 questions from more than 60 medical schools across 16 African countries — documents that AI performance drops significantly when applied to data from LMICs where health records are incomplete, non-standardised, or paper-based. The system is most likely to produce plausible-but-inaccurate outputs precisely where clinical need is highest (Okoh et al., 2025).
Hallucination is an inadequate policy concept. The hallucination framing misrepresents a continuous structural risk as an occasional exception. The system is always predicting statistically probable tokens. It is never retrieving verified facts. The production of inaccurate output is not a deviation from normal function — it is a risk inherent in the function itself, present at every output, varying in probability rather than in kind.
Finding 2: Cognitive Fluency Effect
LLM output fluency drives user trust regardless of factual accuracy. Human cognitive processing associates linguistic fluency — grammatical precision, authoritative tone, coherent structure — with factual accuracy. LLMs are trained through Reinforcement Learning from Human Feedback (RLHF) to optimise specifically for these fluency signals. The system's outputs are most tonally authoritative at precisely the moments when they may be most factually unreliable.
Dell'Acqua et al. (2023) quantified this effect: for tasks falling outside the model's reliable performance range, professionals using AI produced incorrect conclusions at a rate 19 percentage points higher than those working without AI assistance. The study authors described this as the jagged technological frontier — a performance boundary invisible to users because the system's tone and formatting do not change at the boundary.
The WHO has noted that LMMs trained on biased data could perpetuate those biases within health care systems (World Health Organisation, 2021; 2023; 2024). In health systems where senior clinical staff are scarce and verification capacity is limited, the fluency effect is amplified: clinical officers, community health workers, and junior staff without the domain expertise to audit AI outputs are the most likely to act on fluent fabrications. The verification burden falls precisely where verification capacity is lowest. This is not a user literacy problem amenable to training solutions alone. RLHF-optimised systems are architecturally trained to produce outputs that present as authoritative regardless of factual validity.
Finding 3: Productivity Paradox
AI deployment in resource-constrained health systems is premised on productivity gains. Brynjolfsson et al. (2023) demonstrated that highly experienced workers gain the least marginal productivity benefit from generative AI systems. The verification burden — the time required to audit, source-check, and validate AI-generated content against clinical standards — erases the generation speed advantage for domain experts whose professional accountability demands accuracy.
In LMICs and fragile health systems where senior clinical expertise is the resource under greatest constraint, AI deployment that increases the verification load on the most qualified staff produces a net negative productivity outcome — while simultaneously creating unverified output that less experienced staff may act upon without audit capacity. The WHO/Europe Knowledge Community identified governance readiness, not speed of deployment, as the measure of responsible progress. Deployment pace that outruns governance and verification infrastructure produces hidden operational risk, not efficiency gains.
LMIC Implications
In low-resource settings, AI without adequate digital foundations fails in ways that amplify inequity rather than reduce it. AI performance drops significantly when applied to data from LMICs where health records are incomplete, non-standardised, or paper-based. A retrospective evaluation of an LLM-based clinical decision support system deployed across 16 primary care clinics in Kenya identified a very real risk in the naïve application of an LLM to a novel low-resource setting, where errors arising from biases in training data lead to flawed clinical reasoning or guidance (Kiptinness et al., 2025).
Four compounding structural conditions operate simultaneously in LMIC health systems. The training distribution boundary means the system performs least reliably in contexts most different from its training data — and LMIC health system contexts, disease burdens, treatment protocols, pharmaceutical availability, and clinical documentation practices are precisely those most underrepresented in the training data of commercially dominant LLMs. The cognitive fluency effect means that outputs produced in these unfamiliar contexts present with the same authoritative tone as outputs in high-confidence, well-represented domains. The verification capacity gap means that the health workers most likely to receive and act on these outputs are least positioned to detect their inaccuracies. The regulatory gap means that no current framework requires the system to signal reduced reliability in these contexts.
The result is a risk profile that is highest precisely where health system vulnerability is greatest — the inverse of an equitable technology deployment.
Recommendations
Recommendation 1: Require Mechanism-Layer Disclosure in Health AI Procurement
Member States and procurement bodies should require that AI health products disclose, in plain terms, the technical mechanism by which outputs are produced — specifically whether the system performs verified fact retrieval or statistical pattern completion. This disclosure should be mandatory at procurement, not optional in marketing materials.
Recommendation 2: Mandate Calibrated Uncertainty Signalling
AI health tools should be required to produce calibrated confidence bounds alongside outputs, particularly in clinical decision support contexts. Uniform high-confidence output tone regardless of factual reliability is a mechanism design feature, not an acceptable operating characteristic for health applications. Regulatory frameworks should specify minimum uncertainty quantification standards.
Recommendation 3: Include Verification Cost in Health AI Productivity Assessments
WHO procurement guidance and health ministry evaluation frameworks should require that AI productivity claims include time-to-verify alongside time-to-generate. Productivity gains measured only by generation speed systematically misrepresent net health system efficiency, particularly in senior clinical staff time allocation.
Recommendation 4: Retire Hallucination Rate as a Primary Safety Metric
Current health AI safety frameworks use hallucination rate as a key performance indicator. Hallucination rate is structurally inadequate as a safety metric. The fabrication risk is continuous, not episodic, present at every output at a probability that varies but is never zero. Safety frameworks should shift toward output validity rate under independent adversarial audit, tested against domain-specific clinical standards.
Recommendation 5: Establish LMIC-Specific Performance Requirements
The WHO Global Initiative on AI for Health acknowledges that systems trained exclusively on data from high-income settings will propagate bias (World Health Organisation, 2021). Member States should require that AI health products demonstrate minimum performance standards specifically on LMIC-representative clinical benchmarks before procurement approval, not only on high-income-country-derived test sets.
Recommendation 6: Align the Independent International Scientific Panel on AI with Mechanism-Layer Health Risk
The Independent International Scientific Panel on AI, established by UN General Assembly Resolution A/RES/79/325, will issue annual reports to Member States and to the Global Dialogue on AI Governance. Member States should advocate that the Panel's mandate explicitly include assessment of the gap between commercial capability claims and technical mechanism realities for AI products deployed in health contexts — not only assessment of output-level harms after the fact.
Conclusion
The technology currently deployed under active governance frameworks is a statistical prediction engine, not an intelligent reasoning system. Its operational consequences extend to patient safety, health system efficiency, clinical decision integrity, and equitable access to reliable health information — particularly in low- and middle-income country contexts where the gap between training data representation and deployment reality is widest.
Governance readiness, not speed of deployment, is the measure of responsible progress. Governance accuracy — the degree to which governance frameworks are calibrated to the technology that actually exists, rather than the technology that has been commercially described — is a measure of governance quality.
The mathematical mechanism at the core of these systems has not changed. The governance frameworks now being constructed are this generation's best opportunity to ensure that mechanism is governed at the level of its actual characteristics — honestly, precisely, and in the service of the populations that health systems exist to protect.
Use of Artificial Intelligence
Claude assisted with wording and clarity; the author reviewed and approved all content.
Conflicts of Interest
The author declares no conflicting interests.
References
- Bender, E. M.; Gebru, T.; McMillan-Major, A.; Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021; pp. 610–623. [Google Scholar]
- Brynjolfsson, E.; Li, D.; Raymond, L. R. (2023). Generative AI at work. NBER Working Paper No. 31161. National Bureau of Economic Research.
- Dell'Acqua, F.; McFowland, E.; Mollick, E. R.; Lifshitz-Assaf, H.; Kellogg, K.; Rajendran, S.; Krayer, L.; et al. (2023). Navigating the jagged technological frontier: Field experimental evidence of the effects of AI on knowledge worker productivity and quality. Harvard Business School Working Paper 24-013.
- Dziri, N.; Lu, X.; Sclar, M.; Li, X. L.; Jiang, L.; Lin, B. Y.; et al. Faith and fate: Limits of transformers on compositionality. In Advances in Neural Information Processing Systems (NeurIPS 2023); 2023. [Google Scholar]
- European Parliament and Council of the European Union. Regulation (EU) 2024/1689 on artificial intelligence (EU AI Act). Off. J. Eur. Union 2024, 99(1). [Google Scholar]
- Kiptinness, S.; et al. Retrospective evaluation of a generative AI-enabled electronic medical record system in primary health care facilities in Kenya. medRxiv 2025. [Google Scholar] [CrossRef]
- Okoh, I.; et al. AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset. Proc. ACL 2025 2025, arXiv:2411.15640. [Google Scholar] [CrossRef]
- United Nations General Assembly. (2024, March 21). Resolution A/RES/78/265: Seizing the opportunities of safe, secure and trustworthy artificial intelligence systems for sustainable development. UN General Assembly, 78th Session. 78th Session, 2024, March 21.
- United Nations General Assembly. (2025, August). Resolution A/RES/79/325: Global Digital Compact — establishment of the Independent International Scientific Panel on AI and the Global Dialogue on AI Governance. UN General Assembly, 79th Session. 79th Session, 2025, August.
- World Health Organisation. Ethics and governance of artificial intelligence for health: WHO guidance; WHO Press, 2021. [Google Scholar]
- World Health Organisation. Regulatory considerations on artificial intelligence for health; WHO Press, 2023. [Google Scholar]
- World Health Organisation. Ethics and governance of large multi-modal models in health; WHO Press, January 2024. [Google Scholar]
- World Health Organisation Regional Office for Europe. Responsible AI in health: Report of the WHO/Europe Knowledge Community; WHO/Europe, 2025. [Google Scholar]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.