Long-Tail Pharmacovigilance: The Semaglutide–NAION Case and a Pilot Benchmark for Language Models

1. Introduction

1.1. A Cluster of Cases and a Divided Evidence Base

In the summer of 2023, clinicians at Mass Eye and Ear in Boston saw three patients with NAION in one week. NAION is a sudden loss of vision in one eye, usually caused by reduced blood flow to the optic nerve. It is often permanent, and no treatment has been proven to restore the lost vision. The cluster drew attention because all three patients were taking semaglutide [1].

Researchers then asked whether semaglutide is associated with NAION. A retrospective matched cohort study screened 16,827 patient records seen between December 2017 and November 2023 at a single specialist neuro-ophthalmology service [1]. Among 710 matched patients with type 2 diabetes, the 36-month cumulative incidence of NAION was 8.9% with semaglutide versus 1.8% with other medicines (hazard ratio 4.28, 95% CI 1.62–11.29). Among 979 matched patients with overweight or obesity, it was 6.7% versus 0.8% (hazard ratio 7.64, 95% CI 2.21–26.36). These are incidences in a specialist referral population, not absolute risks for typical users.

Larger studies found smaller but consistent increases. Danish and Norwegian registry studies, together covering hundreds of thousands of new users, reported relative risks of roughly two to three [2,3]. An analysis of 14 databases and 37.1 million adults with type 2 diabetes found an increase that depended on study design; it was statistically significant mainly in a self-controlled case-series analysis (incidence rate ratio 1.32, 95% CI 1.14–1.54) [4]. Meta-analyses reported pooled hazard ratios near two and estimated about one additional NAION case per 7,000 patient-years of treatment [5,6,7]. Regulators use a different frame: the European Medicines Agency (EMA) category “very rare” corresponds to up to one case per 10,000 treated patients [8]. All of these are observational findings and do not, by themselves, establish that semaglutide caused the cases [5,6,7].

Randomized trials did not settle the question. A meta-analysis of 69 randomized trials of GLP-1 receptor agonists (144,226 versus 132,922 patient-years) found eight cases of ischemic optic neuropathy on active arms versus four on comparators (Mantel–Haenszel OR 1.53, 95% CI 0.53–4.44, p = 0.43). The trials did not adjudicate eye outcomes independently and did not distinguish arteritic from non-arteritic types [9].

Regulators then reached different decisions (Figure 1). The EMA concluded in June 2025 that NAION is a very rare side effect of semaglutide and recommended a label update [8]. The United Kingdom [10] and Australia [11] followed with warnings. As of October 2026, the U.S. Food and Drug Administration (FDA) has not added a warning to the label, and a review through its Sentinel system is still under way [12]. The association has also become the subject of litigation [13]—context for the cost of late detection, though not evidence of causation.

1.2. The Signal Came from the Clinic, Not from the System

Semaglutide went through a conventional development program: toxicology studies, large clinical trials, and regulatory review. Yet the first signal came from clinicians who noticed a cluster of cases after the drug was widely used. This sequence raises a design question: were current safety systems collecting the information needed to detect this kind of rare event?

We argue that, for this problem, the key variables may not have been collected consistently, and that evidence does not automatically translate into monitoring rules. The claim concerns what the systems were designed to measure. It is not a claim that semaglutide caused NAION.

1.3. From the Head to the Tail of Drug Safety

The case points to a structural feature of current safety systems. Trials, labels, and signal-detection statistics are organized around the head of the adverse-event distribution: events that are common, anticipated, or reported often enough to be counted. Rare harms sit in the tail. There, events are too few for trials to detect, too unexpected to appear on case report forms, and too dependent on individual vulnerability to show up in group averages. The label record has the same shape: many adverse-effect terms are listed for only a small share of drugs, which is how this paper defines its long-tail measure (Section 4.4).

We call the response long-tail pharmacovigilance: a safety program whose primary target is the tail—harms that are rare in patients, rare in the label record, or confined to people with particular vulnerabilities—and whose data, methods, and metrics are chosen for that target rather than inherited from head-oriented monitoring. Box 1 contrasts the two approaches.

Three distinctions keep the term precise. Long-tail pharmacovigilance is not long-term safety, which concerns the time tail of exposure. It is not disproportionality-based signal detection, which needs enough accumulated reports to compute a ratio and so detects a rare harm only after cases have piled up. And it is not rare-disease pharmacology, which concerns drugs for rare conditions rather than rare harms of common drugs. The tail itself has three faces that should not be confused: the label tail (terms listed for few drugs), the incidence tail (events rare among patients), and the vulnerability tail (risk concentrated in a few susceptible people). NAION under semaglutide touches all three. This pilot measures only the label tail.

A tail-oriented program also raises a question about how language models should be queried. An open-ended list may itself favor the head: a model may hold knowledge of a rare harm yet not list it until asked about the relevant organ. Not listing a harm is not the same as not knowing it, and the pilot tests this directly (Section 5.3).

1.4. What This Paper Does

We ask three questions: what information is missing, where does the current process lose it, and how could those gaps be tested? Section 2 describes three gaps—in representation, prediction, and inference—through which head-oriented systems lose the tail. Section 3 proposes a “readout layer” that would link changes in a person’s molecular measurements to possible clinical effects, the core tool of the program. Section 4 defines RARE-Bench, the first benchmark for long-tail pharmacovigilance, with its data, models, and analyses. Section 5 reports a 60-drug pilot and a retrospective probe of the semaglutide case. Section 6 sets out long-tail pharmacovigilance as a research program, states what the method cannot establish, and discusses implications for evaluating AI in drug development. Section 7 concludes.

2. Conceptual Framework: Three Gaps in Detecting Rare Toxicity Signals

Each of the three gaps below describes one way in which a head-oriented safety system loses the tail.

2.1. The Representation Gap: Individual Risk Features Are Not Recorded

One well-known risk factor for NAION is a “disc at risk”: the place where the optic nerve meets the eye is unusually crowded, with little or no central depression. An eye scan called optical coherence tomography can identify this feature [14]. One proposed explanation is that this anatomy combines with changes in blood flow and fluid balance. If blood sugar improves quickly, these short-term changes might reduce blood supply to an already vulnerable optic nerve. This mechanism remains a hypothesis, not an established cause [14,15].

Current records rarely contain these features. Drug development usually records the dose, drug exposure, organ findings in animals, and adverse events reported during trials. Regulators review these results in groups, and prescribing systems check factors such as age, kidney function, and drug interactions. These records may not include the features that make a particular person vulnerable to NAION: the shape of the optic disc, the eye’s ability to maintain blood flow, sleep-apnea severity, or how quickly that person’s blood sugar changes. If the risk arises from an interaction between a person’s anatomy and treatment-related changes over time, a system that does not measure these features cannot test that interaction. This is the representation gap: the data describe the average participant but not the individual features that shape risk.

The disease label is a weak unit of representation for this problem, for three reasons. Most patients who share a label never experience the harm. The features that shape vulnerability—disc anatomy, sleep-apnea severity, vascular reserve—belong to no single disease. And many GLP-1 users live with several chronic conditions at once. The labeled “average patient” therefore cannot carry the information that rare-harm detection needs.

Safety monitoring also depends on which events investigators choose to track. Trial forms and routine surveillance list common or anticipated problems, such as liver, kidney, or heart effects. An unexpected event may not be recorded in enough detail to reveal a pattern, and symptoms may be noticed only after the injury has occurred. A defined set of molecular measurements, by contrast, could in principle be collected repeatedly before symptoms appear. Such measurements would not reveal every future harm, but they could provide another source of early information. Whether they contain reliable signals of later clinical effects is an open question that the present pilot does not test.

2.2. The Prediction Gap: Steady-State Toxicology Misses Short-Lived Changes

Traditional toxicology asks how a dose leads to drug exposure and then to lasting organ injury. The proposed NAION pathway is different: a short-lived change in blood flow or fluid balance, perhaps in the weeks or months after starting treatment or increasing the dose, may coincide with a vulnerable optic nerve [14,15]. Studying such a risk requires following how measurements change over time, not only comparing average values. Animal studies may also fail to reproduce the human optic-nerve anatomy that could modify the risk.

Trials face a numbers problem as well. At a general-population annual rate of 2–10 cases per 100,000 people (baseline rates in type 2 diabetes are higher), a trial with 10,000 participants per arm and two years of follow-up would expect roughly 0.5–2 cases per arm. These are expected counts; a given trial might observe none. With so few events, and without planned eye examinations and expert review, a trial may be unable to detect a difference between groups. Many trials also exclude people with severe vascular disease or sleep apnea, who may have higher baseline risk. A non-significant trial result therefore means that the trial did not establish an increase, not that no increase exists. These limits reflect trial design and event rarity, not a lack of effort [9].

2.3. The Inference Gap: No Shared Method for Weighing the Evidence

The evidence does not all point the same way. Some observational studies report higher NAION rates among semaglutide users, while the pooled randomized-trial analysis found no statistically significant difference. Because these studies address different populations and designs, they are not direct contradictions. Observational studies may be confounded: diabetes itself raises NAION risk, and people prescribed semaglutide may differ from comparison patients in other ways. Some trials also did not adjudicate eye outcomes in detail. This is the inference gap: there is no shared, explicit method for combining these sources and separating the drug’s contribution from a person’s background risk.

Closing this gap would require individual-level data and a causal model that estimates how much the drug contributed while accounting for a person’s anatomy, physiology, and treatment history. The approach is related to methods for separating contributing causes in model-based medicine [16,17]. It is a proposed direction; the present benchmark provides neither the data nor the model. Table 1 summarizes the three gaps and what the proposed readout layer would test for each.

3. The Readout Layer

3.1. From Watching for Symptoms to Reading Molecular Change

We propose adding a different kind of safety signal to drug development and monitoring (Figure 2). Instead of waiting only for a symptom or diagnosis, researchers would compare a person’s molecular measurements before and after treatment and rank the health effects that might follow, including effects not on a standard checklist. The rankings would prompt further measurement. They would not be diagnoses or proof that a drug caused harm.

The central tool is a “phenotype imputer”: a method that uses changes in molecular measurements to rank possible changes in health. Its proposed components are:

• Input: molecular measurements from the same person before treatment and at a later time. People differ even before treatment, so comparing each person with their own baseline may make treatment-related changes easier to detect.
• Reference map: a map linking groups of related molecular measurements (“modules”) to health effects and body systems [18], plus information about which modules tend to change together [19].
• Output: a ranked list of possible health effects with confidence estimates, expressed in standard vocabularies such as the Human Phenotype Ontology (HPO), the Medical Dictionary for Regulatory Activities (MedDRA), or SIDER, and covering possible benefits as well as harms. In symbols: P(health effect | change in molecular state).
• Evaluation: the main measure is how often the method ranks an uncommon, unexpected effect high enough to prompt a useful check—not how often it recovers common effects. For example, if a weight-loss drug’s ranking raised an unexpected concern about the eye’s blood supply, a clinical team could decide whether an eye examination is warranted. The ranking itself would not establish that the drug caused the problem.

The work can be tested in three stages, following the “cell-to-patient” question of how well findings in cells predict effects in people [20]. L1 tests whether drug-level information can help rank health effects, using public resources such as LINCS L1000 and SIDER. L2 tests whether before-and-after molecular measurements from the same person improve those rankings; this requires cohorts with paired measurements around an intervention [21,22]. L3 asks the hardest question: can measurements from blood or other accessible tissues predict effects in distant organs where anatomy also matters? L3 remains an open research question until tested.

3.2. Three Deployment Positions

The method could be tested at three points in drug development and use.

Before human testing. During lead optimization, teams could estimate how a candidate drug changes molecular measurements and use those estimates to rank possible benefits and harms. They could then compare candidates and investigate unexpected risks before choosing which compound to test in people. This is a proposed use, not a capability demonstrated by the pilot.

During a clinical trial. If the ranking raised a specific, unexpected safety concern, the trial team could add a planned test or scheduled assessment, and an expert panel could review suspected cases using consistent criteria. This would not make a rare event easy to detect; the event may still be too uncommon for a moderate-sized trial to estimate its frequency. But planned measurement could test whether the predicted pattern appears and improve the evidence available for later decisions.

After approval. Researchers could periodically collect molecular measurements from people taking a new treatment and compare each person’s results with their own baseline. If measurements related to blood-vessel function changed in a concerning pattern, the method might flag an eye-related risk for clinical review, and a clinician could decide whether an eye scan is appropriate—particularly for someone whose optic-nerve anatomy may increase vulnerability. This workflow has not been tested. A molecular change alone cannot identify who will be harmed, show what will happen next, or prove that the drug caused the change.

3.3. Relation to Biomedical World Models

The proposal is related to “biomedical world models”: computational systems designed to answer what may happen when an intervention is given to a particular person rather than to an average patient [23]. The underlying idea is that broad summaries can hide the details needed to understand uncommon outcomes. A simplified model may work well for common questions while missing what a rare event requires, which is one reason to test a model against the specific safety questions it is meant to answer [24].

4. Methods

4.1. Task Definition

RARE-Bench turns the proposed readout into a staged evaluation for large language model (LLM) systems. In the current L1 track, each system receives information about a drug—its name, class, or known targets—and ranks possible adverse effects using MedDRA or SIDER terms. We score two quantities: precision at k (the share of the top k ranked effects that appear in the reference list) and long-tail label recall (how often a low-prevalence reference effect appears in the ranking). Comparisons include random ranking, ranking by how common each term is in the database, and LLMs without biological reference maps.

The benchmark has three planned stages. L1 tests rankings based on drug-level information, using public resources such as SIDER, signals derived from the FDA Adverse Event Reporting System (FAERS), LINCS L1000, and HPO. L2 will test before-and-after molecular profiles from the same person against later health effects, using cohorts with paired measurements [21,22], to see whether those changes add information beyond public drug descriptions. L3 will test whether measurements from accessible tissues can predict effects in distant organs where anatomy may matter; this is an open question, not an assumed capability [20]. In a Phase II trial, a high-ranked unexpected harm could be reviewed by clinicians as a possible endpoint or reason for extra monitoring. It would not be a causal verdict, and this pilot does not show that RARE-Bench can improve Phase II success rates.

4.2. Data and Sampling

The reference standard is SIDER 4.1 [25], which compiles adverse-effect terms from package inserts as MedDRA preferred terms. We sampled 60 drugs from SIDER, stratified across 14 first-level Anatomical Therapeutic Chemical (ATC) classes; the sampling list was frozen and archived before any model call.

Target lists for the mechanism tier come from STITCH 5.0 interactions [26]. They were curated for 46 of the 60 drugs (77% coverage) through three matching steps: the project’s gene-symbol dictionary (29 drugs), alias bridging (1 drug), and local STITCH interactions (confidence score at least 600, up to 25 targets per drug; 16 drugs). Two perturbation controls tested whether target content mattered: target lists were either shuffled across drugs (fixed-seed permutation, with no drug receiving its own list) or replaced with size-matched random genes drawn from the pooled target set, three repetitions each.

4.3. Models and Input Tiers

The main system was DeepSeek’s deepseek-chat model. It ran the L1 calls (60 drugs × 3 input tiers × 3 repetitions = 540 planned calls, 529 completed) plus the target-tier and perturbation calls. The reasoning model deepseek-r1—from the same provider but differing in generation and training objective—served as the second model in the retrospective probe (Section 4.5). Moonshot (kimi-k3) was planned as a second provider, but platform rate limits allowed only 88 of 540 planned calls (16%); its rows are reported as an incomplete reference sample and excluded from headline comparisons.

Decoding temperature was set to 0 where the platform allowed it; Moonshot permits only temperature 1. Each drug–tier–model cell was run three times. For the name task, three input tiers were compared: drug name alone (tier A), name plus first-level ATC class (tier B), and name plus full ATC code (tier C). For the mechanism task, name plus targets (D2) was compared with targets alone, name withheld (D1). All prompts requested a ranked list of 20 adverse reactions as MedDRA preferred terms, returned in strict JSON.

4.4. Evaluation Metrics and Statistics

Term matching. Model outputs were matched to the reference vocabulary with a MedDRA synonym table and edit-distance fallbacks. Unmatched terms counted as misses, and the match rate is reported separately. Duplicate terms were counted once; outputs shorter than 20 items were scored on the items present; outputs that failed JSON parsing were excluded and reported as failed calls.

Long-tail label recall. This measure was pre-specified in the frozen internal protocol. A reference term is “long-tail” if it appears in no more than 5% of drugs in the database. For each drug, long-tail recall is the share of that drug’s low-prevalence reference terms (T_d) that appear anywhere in the model’s ranked list. The three repetitions are averaged within each drug, then macro-averaged over the 57–60 drugs for which T_d is non-empty. The 5% threshold describes the label database—how many drugs list the term—not how often patients experience it; patient rarity is a different construct and is not measured here.

Baselines. The pre-specified baselines are random ranking, global term frequency, and term frequency conditional on the drug’s ATC class (computed over all drugs, including the scored drug). Additional non-LLM baselines are: a leave-one-drug-out version of class frequency (the scored drug’s own label excluded); a similar-drug label transfer that pools labels of drugs sharing the four-character ATC prefix (the pharmacological subgroup), leave-one-out, ranks terms by the number of neighbor drugs listing them, and falls back to three-character prefixes and then to leave-one-out class frequency for drugs without such neighbors; and two rarity-seeking rankings that order candidate terms by ascending prevalence, globally and within the drug’s ATC class (leave-one-out).

Statistics. Uncertainty is reported with drug-level bootstrap confidence intervals (10,000 resamples over the 60 drugs). Paired tier comparisons are means of drug-level paired differences, which may differ slightly from differences of rounded group means; they use Wilcoxon signed-rank tests with Benjamini–Hochberg correction.

4.5. Retrospective Retrieval Probe and the Free-Versus-Directed Contrast

Design. The probe submitted three drugs to the same pipeline: amiodarone as a positive control (an older drug with long-documented eye toxicity) and the two case drugs, semaglutide and tirzepatide. Each drug was tested with its name and, separately, with a target profile instead of the name, on two models from the same provider that differ in generation and training objective (a general-purpose model and a reasoning model). Ranked lists were inspected for any eye-related term.

Why “retrospective”. We call this a retrospective retrieval probe rather than a blinded test. Drug identity was withheld only in the target-tier inputs, and no attempt was made to control the models’ training-data knowledge of the 2024 signal. The probe asks what current inputs retrieve, not what could have been predicted before the signal was reported.

Organ-directed variant. A follow-up asked explicitly for eye and visual-system adverse effects of the same three drugs (general-purpose model, three repetitions). A post hoc arm repeated it on the reasoning model for the two case drugs (six calls, four parsed). A post hoc negative-control arm asked the same eye-directed question of two benchmark drugs with no known ocular toxicity (olmesartan and mebeverine; general-purpose model, three repetitions each, six calls, all parsed).

Free-versus-directed contrast. For a given model, drug, and organ system, let F = 1 if any term from that organ system appears in the free-ranking list (name tier), and D = 1 if any such term appears under organ-directed questioning. An organ system is classified as surfaced only when directed if D = 1 and F = 0, and as surfaced freely if F = 1. When both are 0, the failure cannot be distinguished from absent knowledge. This classification was formulated after the probe results were seen and is reported as exploratory (Section 4.7). In the present data it is applied to one organ system (the eye) only.

4.6. Preliminary Gene-Level Nomination Probe

To explore how the reference map of Section 3.1 might be built, we measured gene nomination against two gold standards, because the choice of gold changes the answer.

Pharmacological-inference gold. Built locally: for each MedDRA preferred term, the union of STITCH targets of the drugs carrying that term (SIDER 4.1 × STITCH, 1,216 compounds). The 30 evaluated terms were the largest such sets per organ among terms with at least three co-targeted drugs and 10 to 80 inferable genes, ten per organ (median gold size 69 genes). A coverage audit found that of 4,251 organ-mapped preferred terms, only 1,908 (44.9%) yield five or more genes by this inference.

Curated gold. From the Comparative Toxicogenomics Database (CTD, 2026-09-29 release): gene–disease annotations with direct marker/mechanism evidence, restricted to MeSH trees for eye (C11), liver (C06.552), and kidney (C12.950.419) diseases. Of 56 diseases with at least ten curated genes, the ten largest per organ were evaluated (30 diseases; median gold size 35 genes, range 14–136).

Arms and scoring. The language model (temperature 0, two repetitions) received only the phenotype name and nominated 50 HGNC gene symbols. All four arms were scored on a held-out half of each gold set. Two deterministic expansion arms—Reactome pathway over-representation (hypergeometric, background 20,000 genes) and a 332-module gene-set collection—were seeded with the other half of the gold genes; the model received no seeds; the random arm sampled 50 genes from a 14,294-gene background pool. Because CTD annotations are literature-derived and language models are trained on that literature, curated-gold scores should be read as literature-knowledge retrieval, not experimental inference. The two random arms sample from different pools (genome background versus label-coverage pool), so random rates are comparable within a gold but not across golds. All artifacts and call logs are archived with the experiment records.

4.7. Analysis Provenance

Pre-specified in the frozen internal protocol: the stratified drug sample and reference standard; the three input tiers; the six non-LLM baselines (random order, global and class-conditional frequency, leave-one-out class transfer, and the two rarity-seeking rankings); the precision and long-tail metrics; and the primary L1 comparisons.

Added after interim results and reported as exploratory: the similar-drug transfer baseline (a seventh baseline) and its subgroup split; the routed combination; the drug-identification probe; the class-only control; the low-prevalence candidate audit; the chain probe with its negative controls; the input-perturbation consistency probe; the reasoning-model organ-directed arm; the directed-questioning negative-control arm; and the free-versus-directed classification. Each addition, its motivation, and the raw records it rests on are logged with timestamps in the project protocol record, which will be released in the project repository (see Data Availability Statement).

5. Results

5.1. L1 Pilot: Overall Precision and the Long Tail

This section asks whether a language model ranks a drug’s documented adverse effects better than simple label statistics, and especially whether it retrieves the uncommon ones.

Setup. Two language models ranked 20 adverse effects for each of the 60 SIDER drugs without examples (zero-shot), and their rankings were scored against seven baselines on the same reference lists (Section 4.2, Section 4.3 and Section 4.4). DeepSeek completed 529 of 540 planned calls (98%); Moonshot kimi-k3 completed 88, retained as an incomplete reference sample because of platform rate limits (Section 4.3). Six baselines were pre-specified; the similar-drug transfer was added later as the strongest simple alternative (Section 4.7). Table 2 gives all results; Figure 3 shows the long-tail comparison.

Overall precision: a small edge at the top of the list, none deeper down. At precision at 10, every model and input tier scored slightly higher than the strongest baseline, a leave-one-drug-out class-label transfer that ranks the most frequent terms of same-class drugs while excluding the scored drug’s own label (0.800 over all 60 drugs). DeepSeek scored 0.826–0.834 and the incomplete Moonshot sample 0.821–0.884. On the 59-drug set paired with tier A (baseline 0.797), the model’s paired gain was +0.037 (95% CI +0.001 to +0.075), with the lower bound barely above zero; the intervals for tiers B and C included zero. At precision at 20 the transfer baseline outperformed the model (0.726 versus 0.619). The class-frequency variant that includes the scored drug scored 0.798 and is reported for reference. Overall precision is therefore not the main result, and at deeper ranks class-label transfer remains the stronger system.

Long-tail recall: frequency baselines cannot see the tail. The clearest difference appeared on long-tail label recall. Frequency baselines scored 0.000 and 0.005; the main model scored 0.038–0.043 and the incomplete second-provider sample 0.048–0.064 (Figure 3). Against the class-frequency baseline, the gain was statistically distinguishable from zero in all three input tiers; in tier A it was +0.038 (95% CI +0.025 to +0.054). Simply favoring rare terms did not help: ranking candidate terms by ascending prevalence across the database scored 0.000 at precision at 10 with long-tail recall 0.001, and ranking by ascending prevalence within the drug’s ATC class (leave-one-out) scored 0.000 on both. Retrieving a specific drug’s rare terms therefore cannot be achieved by reweighting label statistics toward rarity.

Against the strongest simple alternative, the model’s edge is unresolved overall. The strongest non-LLM baseline for the tail was similar-drug label transfer. It retrieved 0.029 of long-tail terms—several times the frequency baselines—confirming that drugs within a pharmacological subgroup share rare terms. Its lower precision at 10 (0.602) reflects the 39 of 60 drugs without same-prefix neighbors, for which it falls back to broader classes. Against this baseline, the model’s paired long-tail advantage was +0.013 (95% CI −0.018 to +0.039, n = 57), not distinguishable from zero.

The edge depends on whether similar drugs exist (exploratory). For the 20 paired drugs with same-prefix neighbors, transfer scored higher than the model (paired difference −0.026, CI −0.102 to +0.030); 21 of 60 drugs have such neighbors, and one falls outside the paired set, which also requires a usable tier-A call and at least one low-prevalence reference term. For the 37 drugs without such neighbors, the model led by +0.034 (CI +0.014 to +0.056). The group-by-neighborhood interaction was p = 0.038 by permutation. The two systems thus appear complementary: the model adds most where label similarity has nothing to borrow from.

A post hoc router. A router that uses transfer for drugs with neighbors and the model otherwise scored highest of all systems on long-tail recall (0.052; Figure 3). It exceeded transfer alone (+0.022, CI +0.009 to +0.037) and the reversed router (+0.031, CI +0.007 to +0.060). Its margin over the model alone was positive but not resolved (+0.009, CI −0.011 to +0.036), because the router can add value only on the 20 drugs with neighbors. The split itself is a structural feature fixed in advance, but the decision to route was made after seeing the subgroup results, and no cross-validation was performed. The tail gain also costs head-of-list precision (precision at 10, 0.728 versus 0.833 for the model), so a deployment would route only the tail scoring.

Summary of the L1 pilot. The model beats every frequency-based baseline; a simple similar-drug transfer recovers much of the tail; and the model’s edge over transfer is uncertain overall but concentrated where no similar drug exists. None of this shows that the model can predict an adverse effect that has never been reported.

Audit of low-prevalence “misses”: no fabricated terms. We audited the 40 unique candidate terms that appeared in tier-A outputs, were absent from the reference list, and are rare in the database. The author reviewed each against standard pharmacological references for whether it is a real medical term and whether a mechanistic link to the drug is defensible; per-term judgements and their bases are logged in the project record. Thirty-nine were real terms with a defensible link, one was a real term whose direction was questionable, and none were fabricated. Several matched label-listed or standard-reference warnings that the SIDER-derived reference does not map for these drugs—pulmonary fibrosis for amiodarone, thrombotic thrombocytopenic purpura for ticlopidine, and corneal deposits for chloroquine (three verified instances). These reflect reference-list incompleteness rather than model error, and suggest that the ranking task can expose gaps in the reference as well as retrieve from it. The audit had a single reviewer and no second rater, so it should be read as illustrative.

Adding the ATC code did not help. The full ATC code (tier C) did not consistently improve on the drug name alone (tier A), perhaps because the model already knows a known drug’s class. Whether a person’s molecular measurements add information beyond names, classes, and targets remains the L2 question.

Term matching understates some matches. The model’s terms (main model, all tiers pooled) matched the reference vocabulary 82.6% of the time, and unmatched terms counted as misses. Manual review of unmatched terms for a resampled set of ten drugs found that all 27 described real medical concepts but differed in spelling (“Diarrhea” versus “Diarrhoea”), word order (“Blurred vision” versus “Vision blurred”), or MedDRA wording (“QT prolongation” versus “Electrocardiogram QT prolonged”). Whether each is a true adverse effect of the drug was not assessed. Exact matching therefore understates some concept matches, and the reference vocabulary constrains the score.

Training-data exposure limits what the scores mean. The reference lists come from package inserts that the models may have seen during training, so the scores measure retrieval of documented information, not prediction of unknown harms. A prospective test was not possible. The index cluster was observed in summer 2023 and first publicly reported in July 2024 [1]. The V4 generation underlying deepseek-chat was trained on data through about May 2025, after that report [27]; the V3 generation underlying the reasoning model was trained on data through about July 2024 [28], essentially coinciding with it. Retrieval therefore cannot be separated from prior exposure for the main model, and exposure for the reasoning model is at most marginal. A time-frozen or future-signal design remains the clean test; the full benchmark plan includes newer drugs and a control that alters or removes mechanism information.

5.2. Target-Tier Probe: What Do Target Lists Add?

This section asks whether a drug’s protein targets help the model rank adverse effects when the name is withheld. A target is a protein that a drug acts on; a target list is still text about a drug, not a measurement of a person’s molecular state, which is reserved for L2.

Targets alone carry a generic prior. Curated human targets were available for 46 of the 60 drugs (77%). On this subset, the name-only input scored 0.838 at precision at 10, close to its all-drug score of 0.833. With targets alone (D1), precision at 10 was 0.604, versus 0.017 for random ranking (Table 3), and 0.233 lower than name only (paired difference; 95% CI −0.287 to −0.183). Perturbation controls showed that this score does not reflect mechanism-based inference. With target lists shuffled across drugs (0.497) or replaced by random genes (0.635), precision at 10 stayed in the same range as with real targets (0.61, drug-level macro; paired differences versus real targets: +0.109 over shuffled, −0.028 versus random genes). Given almost any target list, the model produces a plausible generic adverse-reaction profile, and overall precision is dominated by that prior.

The tail shows a small target increment. On long-tail recall, real targets scored 0.014, versus 0.001–0.004 for both perturbation arms (paired differences +0.010 to +0.014; 95% CI +0.003 to +0.028 versus shuffled targets and +0.002 to +0.024 versus random genes; n = 45). Target content thus carries little head-of-list information beyond the generic prior but contributes to retrieving low-prevalence terms.

That increment cannot be separated from class knowledge. Two controls bound the reading. In a drug-identification probe, the model was given only the 46 target lists and asked to name the drug; it identified 13% at family level (11% exactly), typically naming another member of the same family (for example, “isoproterenol” for two beta-blockers). Targets therefore appear to carry class information rather than drug identity, making identity leakage an unlikely explanation. In a class-only control, the model received the full ATC code with neither name nor targets; long-tail recall was 0.033 (CI 0.021 to 0.047) over 58 drugs and 0.034 on the 46-drug subset—indistinguishable from the name tier on those drugs (0.034) and well above targets alone (0.014). Class identity alone matches the name tier’s long-tail signal. We therefore withdraw the stronger reading of the target-only result: it is not evidence of mechanism-based ranking, only of a small long-tail increment whose intervals exclude zero, whose leakage via drug identity is unlikely at this recognition rate, and which is not separable from class-level information.

Mechanistic explanations are fluent but not drug-specific. In a chain probe (76 drug–term pairs: 43 retrieved, 33 missed), the model justified each long-tail reference term with an explicit target-to-effect chain. Chains were well formed in both groups: most started from one of the drug’s curated targets (79% for retrieved terms, 85% for missed ones) and reached the stated effect (98% versus 94%; difference +0.037, CI −0.047 to +0.128). The model could thus articulate a plausible mechanism for terms it had failed to rank. A negative-control arm tempers this finding. For terms that are adverse effects of other drugs but not listed for the drug in question (23 pairs), the model produced equally fluent chains (endpoint reach 96%); only for non-medical everyday words (23 terms) did it mostly decline (70% refusals; 13% reach). Chain quality therefore certifies that a term is medically plausible, not that it is specific to the drug, and cannot by itself show what the model knows about a particular drug’s risks.

The aggregate measure is stable under input perturbation. In a consistency probe (15 drugs drawn with a fixed seed from the target drugs with low-prevalence terms), reordering the target list, dropping one target, or adding one irrelevant gene changed about half of the top-20 terms (median Jaccard 0.48–0.54) but moved long-tail recall little (mean absolute change at most 0.012, about a quarter of the reported gains). Individual rankings drift, but the aggregate measure is robust.

Targets add nothing to the name. Name plus targets (D2) scored 0.825 versus 0.838 for name alone (paired difference −0.017, 95% CI −0.033 to −0.001). For known drugs, the name may already supply most of the information available in text.

5.3. Retrospective Probe: Free Ranking Hides the Tail

This section asks whether the models surfaced the eye risk for semaglutide and tirzepatide, and, if not, whether the failure reflects absent knowledge or a failure to bring that knowledge forward.

Positive control: amiodarone. Amiodarone is an older drug known to cause optic-nerve and other eye effects; it checks whether the models can name a well-documented eye toxicity. In all three name-tier runs, the general-purpose model ranked corneal deposits at positions 4–6 and optic neuropathy at positions 15–16, alongside lung inflammation and abnormal heart rhythm. In its one usable run, the reasoning model ranked corneal deposits first and optic neuropathy twentieth. In the target-profile runs, amiodarone retained one eye term (corneal microdeposits, position 18, in one of three runs). Both models thus produced a recognizable eye-toxicity profile, though the target-only evidence is thin.

Free ranking: no eye term for either case drug. For semaglutide and tirzepatide, neither model listed any eye-related harm in any usable run, whether given the name or a target profile. Each case drug had 12 planned calls (2 inputs × 2 models × 3 repetitions); 8 and 10 were usable, and the rest failed because responses did not parse as valid JSON or timed out. All calls are logged in the archived records. Mapping the rankings onto organ systems shows the contrast (Figure 4 and Figure 5): amiodarone’s profile spans cardiac, nervous-system, skin, endocrine, respiratory, and gastrointestinal categories together with six eye terms, whereas both case drugs are dominated by gastrointestinal terms (36 and 33 of 60 term slots) and contain no eye terms. This is a negative result from a small probe, not proof that language models lack this information. It also illustrates the limits of drug-level inputs: they do not supply individual anatomy, treatment-time trajectories, or a way to separate drug effects from other causes.

Organ-directed questioning: the eye terms appear. When asked directly for eye and visual-system adverse effects (three repetitions per drug), the general-purpose model produced a rich ocular profile for all three drugs. For amiodarone, corneal deposits ranked first and optic neuropathy fourth in every repetition. For semaglutide, diabetic retinopathy ranked second (with macular edema and retinal hemorrhage also listed) and “optic neuropathy” appeared at position ten. For tirzepatide, the same profile appeared, with “optic neuropathy” at position sixteen. The reasoning model (deepseek-r1, V3-based), given the same question, listed the full term “non-arteritic anterior ischaemic optic neuropathy” at positions 9–15 for both case drugs (4 of 6 runs parsed; two responses were truncated by the output-length limit and not rerun), while producing no eye terms in free ranking. Diabetic retinopathy complications—a labeled warning for semaglutide based on the SUSTAIN-6 trial [29] and the current prescribing information [30]—also appeared only under directed questioning.

Classification: the eye surfaced only when directed in four of four case-drug cells. Applying the classification in Section 4.5 (Table 4), amiodarone’s eye terms surfaced freely with both models. For semaglutide and tirzepatide, eye terms were absent from free ranking but present under directed questioning with both models, so the pattern is not an artifact of one model generation. The earlier miss is therefore, at least in part, a failure of free ranking to surface a tail direction. Whether it also reflects drug-specific ocular knowledge cannot be read from the directed list alone: as the negative controls below show, the same question returns eye terms for drugs with no known ocular toxicity.

Negative controls: directed eye lists are not drug-specific. The same eye-directed question, put to olmesartan and mebeverine—drugs with no known ocular toxicity—produced rich ocular lists in every run: olmesartan returned 14 eye terms of 20 (including optic neuropathy at position 17 in all three runs and blindness at position 20 in one run), and mebeverine returned 10 of 20 (optic neuropathy at 14), six runs out of six. A directed positive therefore carries little drug-specific information by itself. Two qualifications remain. First, none of the six negative-control runs named NAION: the generic eye list is inducible, the specific syndrome term is not. Second, the free-versus-directed contrast itself stands—the case drugs’ eye terms were invisible in free ranking, which is the finding that matters for evaluation design. What the controls add is the interpretation rule: a directed sweep is informative only against the baseline of the same question put to other drugs.

What the directed result does not show. Two readings remain open. For the reasoning model, whose training cutoff essentially coincides with the first public report, naming NAION is consistent with either marginal early exposure to that report or a learned diabetes–NAION prior. For the chat model, whose cutoff postdates the report, post-report knowledge cannot be excluded. The design cannot distinguish these explanations, and the result is not evidence of prospective signal detection or that any LLM system could have predicted NAION before it was reported. Either way, the practical conclusion holds: under free ranking, the direction that mattered was invisible. Models from other providers could not be tested because the execution environment could not connect to the model-aggregation service; those comparisons remain for the full benchmark.

5.4. Preliminary Gene-Level Nomination: the Gold Standard Decides the Winner

This section asks whether language-model gene nomination could help fill the sparse reference map that the readout layer needs (Section 3.1), and how such nomination should be validated.

The map is mostly empty. Of organ-mapped preferred terms, 55.1% cannot be assigned even five genes by pharmacological inference (Figure 6b).

Against curated annotations, the model leads. Against the curated CTD gold [31], zero-shot nomination of 50 genes recovered on average 25.3% of the held-out half of each curated gene set (mean precision at 50, 0.093), despite receiving no seed genes and being scored, like every arm, only on that half. Seeded expansion from Reactome pathways [32] (2022 release, via the Enrichr gene-set library) and from the 332-module collection reached 8–9% recall—far above random but far below the model (Table 5, Figure 6a).

Against co-targeting, the ranking reverses. Against the pharmacological-inference gold, the same model recovered only 5.3%, below the random rate for that frame (7.9%), while pathway expansion led (0.338). The two golds embody different notions of a “related gene”—curated mechanism versus co-targeting. They also cover partially different phenotype sets (30 MeSH diseases versus 30 MedDRA preferred terms), and because co-targeted genes cluster in shared pathways, the pharmacological frame favors pathway expansion by construction. We therefore report both and treat the choice of nomination method as an open, gold-dependent question.

Scope. The probe is small (30 diseases, one model, one temperature, nested MeSH terms included); a full benchmark with more models and experimental perturbation-profile validation is future work. What the probe establishes is that the annotation gap is large, that nomination is measurable, and that the validation frame determines the answer.

6. Discussion

The pilot asked whether a language model can rank a drug’s uncommon adverse effects. It can, measurably, and with no fabricated terms among the 40 audited candidates—although a simple similar-drug transfer recovers much of the same tail, and the model’s advantage appears to reflect class-level knowledge rather than drug-specific mechanism. The semaglutide probe adds a second answer: free ranking left the eye invisible for both case drugs, while organ-directed questioning brought eye terms forward—though, as the negative controls show, a directed eye list is not drug-specific, and only the specific term NAION stood out. Both results support the premise of long-tail pharmacovigilance—that the tail behaves differently from the head and needs its own evaluation.

6.1. Long-Tail Pharmacovigilance as a Research Program

Three findings, three design rules. Each pilot finding translates into a rule for tail-oriented evaluation.

• Evaluate the tail separately. The head is cheap: at precision at 10 the model’s edge over class-label transfer was small, and at precision at 20 transfer was better. The tail is not: frequency and rarity-seeking baselines retrieved almost none of it. Long-tail recall should therefore be reported by prevalence band, next to—not inside—overall precision, because a system can win the head and lose the tail.
• Route by neighborhood structure. Where similar drugs exist, their labels are a strong, cheap source of tail terms; where they do not, model knowledge adds most (interaction p = 0.038, exploratory). A tail system should combine the two rather than choose one.
• Sweep with controls, do not only list. For the case drugs, the eye appeared only when the question named it—but negative-control drugs returned rich eye lists under the same question, so a directed positive alone is not evidence. Directed sweeps should be run as comparisons across drugs (same question, drugs with and without known toxicity in the organ), and a harm’s absence from a free list should be treated as unknown, not as negative evidence.

The technical route. The program moves along the three faces of the tail. L1 ranks label-tail terms from drug-level information (this pilot). L2 uses within-person, before-and-after molecular profiles to reach the incidence tail in real cohorts. L3 asks whether accessible tissues can signal remote-organ harms that depend on anatomy, which is where the vulnerability tail lives. The readout layer and its reference map connect the stages. Building that map is itself part of the program: 55.1% of organ-mapped terms cannot yet be assigned even five genes by pharmacological inference (Section 5.4).

Limits of the present evidence. The pilot measures only the label tail; the incidence and vulnerability tails are untested. The free-versus-directed contrast rests on one organ system, two case drugs and one positive control, two models from one provider, and a handful of runs, and it was defined after the results were seen. The directed-questioning negative controls cover only two drugs and one organ, so the specificity baseline is itself provisional; and the chain probe shows that the model produces fluent mechanisms even for terms not listed for a drug, so directed questions may surface plausible but non-specific terms. A confirmatory test would pre-specify the classification, sweep every organ system for all 60 drugs, include drugs with no known toxicity in each organ as negative controls, and score directed outputs against the reference. Long-tail pharmacovigilance is proposed here as a research program; this paper establishes its first measurable step, not its clinical value.

6.2. What the Method Will Not Establish

Molecular measurements and health effects are not one-to-one. A group of measurements may relate to several health effects, and one effect may depend on several groups. Without an accurate reference map, a ranking could show associations without explaining how they relate to biology, and the map’s accuracy will bound the method’s accuracy.

People start from different baselines. The same measured change may mean different things for different people, which is why the proposed input is a within-person before-and-after comparison rather than a single measurement. Collecting such repeated measurements will require appropriate study procedures and standards for N-of-1 research [33].

A warning is not proof of cause. A molecular change could reflect the underlying disease, normal measurement variation, or seasonal effects rather than the drug. Estimating whether and how much the drug contributed requires a separate analysis (Section 2.3); prediction and causal attribution must be kept apart.

L3 may have a negative answer. Measurements from blood or another accessible tissue may not contain enough information to predict an effect in a distant organ whose anatomy matters. Testing that limit would show which features must be measured directly. A well-defined negative result would set the method’s boundaries; an untested assumption would not [20].

The reference data under-represent the patients at highest risk. Package-insert adverse-effect lists are conditioned on trial populations, which systematically exclude people with severe comorbidities, advanced organ dysfunction, and polypharmacy—precisely the patients in whom adverse-event rates are highest and event profiles differ. Even a system that retrieved every listed effect would therefore under-represent risk in multimorbid patients. Within-person, state-based measurement is the natural complement for that population; connecting the two is future work [34].

6.3. Implications for Clinical-Stage AI Evaluation

The task. The case points to a clinical-stage safety task that is not yet well tested: can an AI system help trial teams notice unexpected harms early enough to investigate them? RARE-Bench is our proposed framework for evaluating that task. Its current track uses drug-level information; later tracks would test whether molecular changes measured in the same person add information beyond drug names and other public text. The benchmark scores rankings for each health-effect category, with early retrieval of uncommon, unexpected effects as a key measure, and the free-versus-directed contrast as a companion measure, scored against negative-control drugs given the same directed question. Public datasets such as LINCS, SIDER, FAERS-derived signals, and HPO can support the initial track; cohorts with before-and-after measurements can support the paired-profile track [21,22].

Why language models. In the tail, frequency statistics have little to count, so any tail method must draw on prior knowledge about drugs and biology. The task also requires translating between descriptions of molecular changes and standardized names for health effects (HPO, MedDRA, and SIDER terms), and drawing on background knowledge about drugs and body systems. Language models can work with both kinds of text, rank possible effects, and explain their choices in words a trial team can review. These abilities make them a practical starting point, not necessarily the final or best tool. RARE-Bench is designed to test whether their rankings are useful, including whether they help experts choose safety questions for a Phase II trial. This paper does not estimate whether that use would increase trial success or save money.

Benefit and harm as two readings of one state. RARE-Bench could complement an existing effort to evaluate how language models rank interventions for a given molecular state [34]. In the paired-profile stages, neither task requires a diagnosis as input or output: reading a state map toward interventions (benefit) and toward phenotypic risks (harm) would be two directions of the same representation. Clinical decisions need both. The semaglutide–NAION case motivates the safety direction, but this pilot evaluates harm rankings only and does not combine benefit and risk into a treatment recommendation.

From evidence to action. The timeline in Figure 1 shows the sequence behind this framing. Within roughly three years of the index cluster, three regulators updated labels while the U.S. label remained unchanged. Regulators reached different decisions from a largely shared evidence base. This illustrates that evidence is not translated into action by a common, explicit method, although different decisions may also reflect different evidentiary standards.

A broader lesson for this task. Averages hid the tail, and in this task the tail was where the safety signal lived. A finding that an evaluation “found no problem” is therefore a statement about the evaluation, not about the drug. Every coverage claim should state the gold standard, the prevalence band, and the query framing, because the framing decides what counts as covered. The same principle may apply to AI systems more generally, but the present study tests only this drug-safety task.

7. Conclusion

The semaglutide–NAION case raises the question of whether current drug-safety systems collect the information needed to detect uncommon harms. We describe three gaps that may contribute: individual vulnerability is poorly represented, short-term changes are hard to capture, and evidence from different sources is not consistently turned into testable clinical questions. We frame the response as long-tail pharmacovigilance and propose RARE-Bench, its first benchmark, to test whether LLM systems can rank uncommon adverse effects and help identify safety questions for clinical review. The case motivates the benchmark; it does not show that the current system would have predicted NAION before the association was reported.

The pilot results, stated plainly: global frequency ranking scored zero on the long tail and class-conditional frequency scored near zero (0.005), while language models retrieved some low-prevalence terms in every tier. Against the strongest simple alternative, similar-drug label transfer, the margin was not resolved at this sample size; instead the two split the drugs—transfer led where same-class neighbors exist, the model where they do not (interaction p = 0.038)—suggesting complements rather than rivals. Under target-only input, overall precision was dominated by a generic adverse-reaction prior: random genes scored as well as real targets. On the gene side, the ranking of methods reversed with the gold standard.

The case also shows why the tail needs its own methods. Asked freely, neither model mentioned the eye for semaglutide or tirzepatide. Asked about the eye, both returned eye terms—but so did two negative-control drugs with no known ocular toxicity, and only the specific term NAION, named by one model, stood out. A tail-oriented evaluation must therefore treat absence from a free list as unknown, not negative, and read a directed result only against the same question asked of control drugs. These findings are exploratory and limited to one organ, two case drugs, two control drugs, and one provider.

None of these results establishes prediction of a new harm. They establish where each kind of signal survives and where it does not. Whether before-and-after molecular measurements from the same person improve risk rankings remains untested, as does the clinical value of using such rankings to choose Phase II endpoints or improve trial success; both require separate, prospective evaluation. The broader proposal is long-tail pharmacovigilance: evaluating drug safety at the level of the rare questions clinical teams need to answer, rather than by average performance alone. This work is related to broader efforts to reason about interventions from a person’s molecular state, including work in longevity medicine [23,33,34].