From Sight to Insight: A Comprehensive Survey of Large Models in Medicine

1. Introduction

The use of Large Language Models (LLMs) in healthcare has moved from exploratory chat interfaces toward increasingly capable clinical decision-support prototypes [1,2,3]. In the recent wave of “medical large models”, used here as an umbrella term for large-scale foundation models that operate over medical text, images, and other clinical signals, progress has moved beyond surface-level description and retrieval [4,5]. A practical inflection point is now visible: systems are being designed not only to perceive medical data, but also to reason over multi-step clinical problems and to act within constrained workflows under human oversight.

This transition is shaped by three converging trends. First, reasoning-centric models such as OpenAI o1 and DeepSeek R1 have renewed interest in the distinction between “fast thinking” and “slow thinking” [6]. Compared with earlier instruction-following systems, these models allocate more inference-time compute and use reinforcement learning signals, including RLVR, to encourage multi-step deliberation. In medicine, such “Logic Engine” behavior is useful because it can reduce, although not eliminate, hallucinations and can make error-checking more explicit in complex cases [7].

Second, multimodality is moving from “stitched-together” late fusion toward more integrated omni-modal designs. Earlier systems commonly paired a frozen vision encoder with a language model through a projector. This design can work well, but it may lose fine-grained spatial or cross-modal correlations. More recent architectures, including early-fusion designs with Mixture of Experts (MoE) and modality-aware positional schemes such as iRoPE, treat images, audio, and text as a more unified token stream [8,9,10,11]. For medical imaging and long-form electronic health records (EHR), this direction is promising because it supports joint grounding and long-context integration, while also raising new questions about training stability, calibration, and evaluation.

Figure 1. Overview of the capability stack reviewed in this article. Medical large models are organized around four connected capabilities: sense and ground, which converts heterogeneous patient data into evidence-grounded states; unify, which moves from image–text alignment and connector-based systems toward native omni-models; reason, which allocates deliberation, verification, bias auditing, and calibration according to case difficulty; and act and discover, which links verified decisions to agentic clinical workflows and biomedical discovery.

Figure 1. Overview of the capability stack reviewed in this article. Medical large models are organized around four connected capabilities: sense and ground, which converts heterogeneous patient data into evidence-grounded states; unify, which moves from image–text alignment and connector-based systems toward native omni-models; reason, which allocates deliberation, verification, bias auditing, and calibration according to case difficulty; and act and discover, which links verified decisions to agentic clinical workflows and biomedical discovery.

Third, the field is expanding from passive assistants to active medical agents: systems that plan, call tools, maintain state, and collaborate across roles. Projects such as “Agent Hospital” study how agentic behavior can be improved in simulated environments through self-play and evolutionary training mechanisms [12]. Such simulations cannot replace real-world clinical validation, but they provide a controlled setting for studying long-horizon workflows under privacy constraints and for probing safety failures before deployment.

Despite these advances, evaluation remains a central bottleneck. Conventional benchmarks, such as exam-style multiple-choice tests, are often insufficient for separating memorization from robust reasoning [13,14,15,16]. They also provide limited evidence about reliability under distribution shift, uncertainty calibration, or workflow utility [17]. This has motivated evaluation frameworks such as audit-style checklists and dynamic, clinician-facing arenas that emphasize process validity, robustness, and bedside relevance rather than static accuracy alone [18].

To organize this literature, we provide a roadmap of medical large models in the era of reasoning and agency. Rather than classifying the field only by data modality, we adopt a capability-based taxonomy that tracks how foundation models reconstruct clinical competencies from perception to action. Specifically, we:

• define the transition from fast pattern matching to more deliberative, verifier- and search-assisted “Logic Engine” behaviors, and summarize mechanisms such as test-time scaling and process supervision;
• survey the shift toward native omni-modality and the development of medical agents that plan, use tools, and maintain state in long-horizon workflows; and
• synthesize evaluation, trustworthiness, and governance practices that can close the gap between benchmark performance and bedside reliability.

Methodologically, we emphasize recurring design patterns, evaluation practices, and failure modes that matter for translation. Section 2 reviews multimodal perception and grounding; Section 2.2 discusses knowledge integration through KGs, advanced RAG, and memory; Section 3 surveys omni-modal architectures and fusion; Section 4 covers reasoning mechanisms; and Section 5 examines agentic workflows.

Figure 2. Timeline of representative advances in medical large models from 2021 to 2026. The figure groups developments into general AI substrates, medical perception models, reasoning and reinforcement-learning methods, agents and evaluation frameworks, and knowledge or biomedical foundations, highlighting a transition from alignment and scaling toward long-context perception, reasoning, deployment, and governance.

Figure 2. Timeline of representative advances in medical large models from 2021 to 2026. The figure groups developments into general AI substrates, medical perception models, reasoning and reinforcement-learning methods, agents and evaluation frameworks, and knowledge or biomedical foundations, highlighting a transition from alignment and scaling toward long-context perception, reasoning, deployment, and governance.

2. Perceptual Foundations: Multimodal Acquisition and Knowledge Grounding

Medical AI is moving from specialized, single-task discriminative models toward generalist foundation models that support native, or near-native, multimodal perception [5,19,20]. Earlier pipelines often paired separate encoders across modalities, such as CNNs for imaging and Transformers for text, with task-specific heads. Current systems aim to map clinical concepts more directly onto raw sensory inputs, from gigapixel pathology slides to long surgical videos and 3D volumetric scans. From a survey perspective, we organize this progress by modality (2D, temporal streams, and 3D) while using a consistent comparison lens: (i) input scale, including resolution and context length; (ii) representation and fusion, including patch tokens, resamplers, and tool-based exploration; (iii) supervision and data, including paired reports, self-supervision, and synthetic instruction; and (iv) clinical evaluation, including task metrics, robustness, and failure modes under domain shift. This lens is important because richer perception is useful only when it can be connected to grounded knowledge, calibrated uncertainty, and clinically meaningful action.

Figure 3. Representative architectures for medical perception and multimodal grounding. The figure summarizes BioMedCLIP, CONCH, CT-Agent, HeAR, LLaVA-Med, Med-Gemini, MedSAM-2, and Prov-GigaPath, illustrating how contrastive learning, caption-aware representation learning, tool-assisted 3D exploration, acoustic self-supervision, connector-based instruction tuning, native multimodality, promptable segmentation, and whole-slide aggregation support medical perception before downstream reasoning or action.

Figure 3. Representative architectures for medical perception and multimodal grounding. The figure summarizes BioMedCLIP, CONCH, CT-Agent, HeAR, LLaVA-Med, Med-Gemini, MedSAM-2, and Prov-GigaPath, illustrating how contrastive learning, caption-aware representation learning, tool-assisted 3D exploration, acoustic self-supervision, connector-based instruction tuning, native multimodality, promptable segmentation, and whole-slide aggregation support medical perception before downstream reasoning or action.

2.1. Native Perception Across Static and Omni-Sensory Modalities

2.1.1. Large Vision Models for Pathology and Radiology in the Gigapixel Era

The processing of 2D medical imagery has long been constrained by the trade-off between resolution and context. This limitation is most visible in digital pathology, where a single Whole Slide Image (WSI) can contain billions of pixels and far exceed the context window of traditional Vision Transformers (ViTs). Earlier multiple-instance learning and weak-supervision pipelines already showed the value of slide-level aggregation at clinical scale [21,22]. Recent computational pathology foundation models address this issue through large-scale pretraining and slide-level representation learning, reducing the need for aggressive downsampling [23,24].

Models such as Prov-GigaPath [25] and Virchow [26] mark a shift from local patch classification toward holistic slide encoding. Prov-GigaPath, for example, uses a LongNet-style design with dilated attention to process tens of thousands of image tokens at once. This allows the model to capture both cellular-scale morphology and broader tissue context, which is important for tasks such as predicting gene mutations from morphology or identifying dispersed micro-metastases. Similarly, the UNI model [27] uses a ViT-Huge backbone pretrained on more than 100 million tissue patches, suggesting that model capacity and data diversity can improve zero-shot performance in histopathology tasks such as cancer subtyping.

In radiology, the focus has also moved beyond coarse disease classification toward fine-grained, promptable segmentation. The adaptation of Segment Anything Model 2 (SAM 2) [28,29] to medicine, represented by MedSAM-2 [30], illustrates this direction. MedSAM-2 supports point, bounding-box, and text prompts for segmenting anatomical structures across modalities such as X-ray, MRI, and ultrasound with a single set of weights. It also introduces a “memory bank” mechanism for treating 3D volumes as sequences of 2D slices, allowing segmentation masks to be propagated more consistently across CT scans. This design helps connect 2D vision foundation models with the spatial requirements of volumetric imaging, complementing established medical-segmentation baselines such as nnU-Net, the Medical Segmentation Decathlon, and TotalSegmentator [31,32,33].

Generalist radiology foundation models such as RadFM [34] extend this trend by interleaving text and images in a unified stream. RadFM supports multi-image inputs and visually conditioned question answering, addressing the single-image limitation of earlier radiology vision-language models such as CheXzero, BioViL, MedCLIP, and CheXagent [35,36,37,38].

2.1.2. Temporal and Streaming Perception: Surgical Video and Bioacoustics

Medical data is inherently temporal. Patient conditions evolve over time, surgeries are dynamic procedures, and physiological signals are continuous streams. The integration of time-series data into Large Models has transitioned from simple frame-averaging to sophisticated long-context reasoning.

Long-Context Surgical Understanding. Surgical video analysis has long been hindered by the computational cost of processing long-duration footage. Traditional methods often resorted to sampling disparate frames, losing the causal thread of surgical actions. Earlier systems such as EndoNet and TeCNO established strong baselines for surgical phase recognition, but they were not designed for open-ended long-context reasoning [39,40]. The introduction of models with extreme context windows, such as Gemini 1.5 Pro and the optimized Gemini 2.5 [11,41], has substantially alleviated this “memory” bottleneck. With context windows extending to 1 million and subsequently 2 million tokens, these models can, in principle, ingest an entire hour-long cholecystectomy or robotic-assisted prostatectomy video as a single input.

This “native video” capability allows the model to perform temporal grounding: retrieving specific surgical phases (e.g., “locate the moment the cystic artery was clipped”) or identifying adverse events with precise timestamps. Compared with short-context approaches that rely on heavy frame sampling (and often lose procedural continuity), long-context models better preserve the sequence of events and support timeline-based reasoning. In practice, however, clinical utility depends not only on context length but also on latency, annotation quality, and evaluation against procedure-specific safety criteria under realistic OR variability.

Acoustic Foundation Models. Beyond vision, sound remains an underutilized diagnostic modality. Early COVID-era studies showed that cough and respiratory audio can carry clinically relevant signal, while also exposing sensitivity to devices, demographics, and data-collection protocols [42,43]. The HeAR (Health Acoustic Representations) model [44] by Google Research illustrates the use of self-supervised learning in bioacoustics. Trained on over 300 million audio clips using Masked Auto-Encoders (MAE), HeAR learns to reconstruct missing segments of audio spectrograms, thereby learning a robust representation of human physiological sounds, such as coughs, breathing, and speech.

Crucially, HeAR is privacy-preserving and device-agnostic; it generalizes well across different microphones and recording environments. In downstream tasks, HeAR has achieved state-of-the-art performance in detecting tuberculosis and COVID-19 from cough sounds, and even estimating lung function parameters (like FEV1) from spirometry audio. More broadly, these results suggest that large-scale self-supervision can yield clinically useful “acoustic priors,” while still requiring careful validation across populations, devices, and recording conditions.

2.1.3. 3D and Spatial Intelligence: Volumetric Agents

While human vision is 2D, medical reality is 3D. One limitation of adapting standard Large Language Models (LLMs) to medicine is their 1D sequential structure, which struggles with the spatial complexity of volumetric data such as CT and MRI scans. Representing a 3D volume as a sequence of image tokens often leads to an explosion in sequence length, overwhelming standard attention mechanisms. Volumetric transformers and hybrid CNN-transformer models such as UNETR and Swin UNETR show how 3D spatial priors can be built into medical-image encoders before they are connected to language models [45,46].

From Slice-Scanning to Spatial Agents. To address this, recent work has explored agentic approaches to 3D perception, most notably the CT-Agent [47]. Unlike models that try to compress a whole volume into a single vector, often losing small lesions, CT-Agent operates as an autonomous system equipped with a specialized toolset. It treats the 3D volume not as a static image, but as an environment to be explored.

The CT-Agent architecture consists of a Planner (an LLM backbone), an Action Space (specialized tools for detection, segmentation, and zooming), and a Memory module. When asked to “check for pulmonary nodules in the upper right lobe,” the model does not blindly scan the whole volume. Instead, it:

1.

Plans: Decomposes the query into spatial coordinates (Right Upper Lobe).

2.

Acts: Calls a zoom-in tool to retrieve high-resolution slices of that specific region.

3.

Perceives: Uses a specialized 3D-CNN encoder to analyze the local volume.

4.

Reasons: Integrates the visual finding with clinical guidelines stored in its parametric memory to formulate a diagnosis.

This active-perception design resembles the workflow of a human radiologist, who scrolls through slices and adjusts window/level settings to investigate suspicious areas. By decoupling spatial exploration from semantic reasoning, models such as CT-Agent and M3D-LaMed [48] improve sensitivity for small anatomical abnormalities while maintaining computational efficiency. They illustrate a shift from viewing medical data to interacting with patient anatomy.

2.1.4. Summary of Perceptual Evolution

These advances extend medical perception along three practical dimensions: scale, through gigapixel and high-resolution imaging; time, through minute-to-hour temporal streams; and interaction, through active exploration of 3D anatomy rather than passive viewing. A critical lesson is that richer perception alone does not ensure clinical utility. Its value depends on whether percepts are grounded in factual knowledge, whether uncertainty is calibrated and communicated, and whether failure modes under distribution shift, including scanner differences, protocol drift, and long-tail pathology, are detected before they affect patient care. Table 1 maps representative tasks, recurring failure modes, and evaluation considerations across modalities.

2.2. Foundational Knowledge and Retrieval: Bridging Probability and Factuality

Perception alone is not enough. A model that detects a spiculated lung mass but misapplies staging criteria, or fluently describes a non-existent drug interaction, has limited clinical value. The central problem is that large language models are probabilistic generators: they predict likely tokens rather than verified facts. In medicine, hallucinations, including factual fabrication, evidence mismatch, and cross-turn inconsistency, can have direct clinical consequences [49,50]. This has led to growing interest in Knowledge Integration, which combines generative flexibility with the structure of symbolic knowledge and targeted retrieval. Three lines of work are especially relevant: Knowledge Graphs (KGs) for structured, multi-hop reasoning; Retrieval-Augmented Generation (RAG) systems that are hybrid, cross-modal, and provenance-aware; and memory mechanisms that preserve longitudinal patient context beyond a single context window.

2.2.1. Knowledge Graph Integration and Commonsense Reasoning

Structured knowledge integration brings connectionist and symbolic approaches together in MedLMs. Medical knowledge is relational by nature, linking symptoms, diseases, drugs, genetic markers, and treatment constraints. Purely parametric models often struggle to preserve consistency across these relations, especially for long-tail diseases or contraindications that are sparsely represented in training corpora [51]. Recent knowledge-augmented approaches therefore connect Medical Knowledge Graphs (MKGs), such as UMLS (Unified Medical Language System), SNOMED CT, Gene Ontology, SPOKE, and Hetionet, with the model’s prompting, retrieval, or inference loop [52,53,54].

Current methodologies for KG integration can be categorized into Training-time Injection and Inference-time Augmentation. Training-time strategies involve fusing entity embeddings from KGs with textual representations during pre-training or fine-tuning. A complementary line of work uses KG-derived paths and entities to construct prompts or retrieval contexts (e.g., Disease A $\stackrel{symptom}{\to }$ Symptom B $\stackrel{associated\_with}{\to }$ Disease C), encouraging the model to reason over the logical structure of medical ontology rather than merely memorizing co-occurrence patterns [55]. By aligning language-model inputs with structured MKG representations, MedLMs can better distinguish phenotypically similar but etiologically distinct conditions.

The static nature of training-time injection limits adaptability. Inference-time augmentation has therefore become important, particularly for tasks that require “Logic Engine” reasoning. In this setting, the MedLM can query external knowledge sources to verify intermediate reasoning steps. For example, when generating a differential diagnosis, the model may identify a possible link between a drug and an adverse effect. Rather than relying only on its internal weights, it can retrieve a KG-backed evidence path or guideline snippet to check whether the relation is supported [56]. This explicit grounding can reduce logical hallucinations and provides a degree of interpretability because the model can cite the evidence path used to reach a conclusion. It may also help incorporate forms of “medical commonsense” that are rarely stated directly in text but are encoded in ontological hierarchies.

2.2.2. Next-Generation Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation (RAG) is now widely used to reduce hallucination by grounding model outputs in external evidence [57,58]. First-generation RAG systems, which rely mainly on dense vector similarity search, are often insufficient for medical use. Current work is therefore moving toward Advanced RAG designs that combine hybrid retrieval, provenance tracking, and cross-modal evidence [59,60,61].

Hybrid and Adaptive Retrieval.

Dense retrieval can fail on precise medical nomenclature or acronyms, where semantic similarity is not the same as clinical relevance. For example, differentiating “Type 1” from “Type 2” diabetes often requires exact matching rather than semantic approximation. Contemporary MedLMs therefore increasingly use Hybrid Retrieval, combining sparse retrieval such as BM25 with dense embedding-based retrieval [62,63]. Sparse retrieval captures exact names, dosages, and abbreviations, while dense retrieval captures broader query intent. More structured RAG designs further organize retrieved evidence into graphs or self-reflective loops so that models can check whether the retrieved context is sufficient before answering [64,65,66]. This can reduce latency and limit retrieval-induced hallucination, where irrelevant context misleads the generator.

Cross-Modal Retrieval.

As MedLMs become multimodal, RAG also needs to extend beyond text. Cross-Modal RAG allows models to retrieve non-textual evidence for diagnosis. In radiology, for example, a text query about a “spiculated mass” could retrieve visually similar historical CT scans and associated biopsy reports from a hospital PACS (Picture Archiving and Communication System). This depends on joint embedding spaces, such as BioMedCLIP- or UniMed-CLIP-style projections, where images and text are aligned across medical modalities [67,68]. By exposing the model to verified visual precedents, Cross-Modal RAG can support rare-disease reasoning when textual descriptions alone do not capture subtle visual patterns.

From an evaluation standpoint, medical RAG systems should be assessed not only by end-to-end answer quality, but also by retrieval quality (coverage, relevance, recency), provenance (traceable citations and excerpt faithfulness), and robustness to missing or noisy evidence. Without such controls, improved fluency can mask retrieval-induced hallucinations or overconfident conclusions drawn from weak or non-representative context.

2.2.3. Medical Memory Mechanisms: From Context to Continuity

The clinical course of a patient is rarely a single event. It is usually a longitudinal record that may span years. Although recent LLMs support context windows exceeding one million tokens, relying only on long context to represent a patient’s history is computationally inefficient and vulnerable to the “lost-in-the-middle” problem. MedLMs therefore need explicit Memory Mechanisms that separate working, episodic, and semantic information.

Hierarchical Long-Term Memory.

To manage longitudinal data, modern MedLMs can draw on ideas from hierarchical EHR modeling. In this architecture, patient interactions are represented at multiple levels of granularity. Raw clinical notes and codes can be processed into high-level abstractions, such as “history of chronic hypertension”, while specific recent events, such as “BP 150/90 recorded yesterday”, remain available as time-stamped evidence [69,70,71,72]. When a query is received, the model should not simply scan the entire raw history; instead, it should retrieve relevant abstractions and specific details. This allows the model to maintain coherence over long-term interactions, recalling a drug allergy noted five years ago just as easily as a symptom reported five minutes ago. This persistence is fundamental for deploying AI agents in chronic disease management, where the trajectory of biomarkers over time is more indicative than any single measurement.

Working Memory and State Tracking.

In the context of autonomous medical agents, discussed further in Section 5, Working Memory is important for multi-turn diagnostic reasoning and task execution. Unlike passive RAG, which retrieves information, working memory maintains the state of a clinical investigation. For example, in a diagnostic dialogue or workflow-automation benchmark, the agent must track which symptoms have been asked about, which tests have been ordered, and which hypotheses have been ruled out [73]. Emerging architectures employ scratchpads, explicit state variables, or tool-call traces to maintain a dynamic representation of the current clinical problem. This allows the agent to engage in non-linear reasoning and revisit previous assumptions when new evidence, such as a lab result, contradicts the current working diagnosis. By decoupling the static knowledge base from the dynamic working context, these systems may reduce premature closure in diagnostic reasoning.

For longitudinal deployment, a practical survey lesson is that “memory” should be evaluated as a clinical consistency capability: whether key facts (allergies, contraindications, prior outcomes) are faithfully retained, whether contradictions are detected and surfaced, and whether updates are appropriately time-stamped and auditable.

Taken together, the knowledge-grounding mechanisms reviewed here, including structured KG reasoning, provenance-aware multimodal retrieval, and hierarchical memory, address a common failure pattern: fluent outputs that are not anchored to reliable evidence. The main evaluation question is therefore not only whether grounding improves average accuracy, but whether it reliably prevents high-consequence errors such as missed contraindications, fabricated evidence, or loss of longitudinal context.

3. Omni-Modality: The Convergence of Vision, Language, and Signal

Clinical decision-making rarely relies on a single data type. Physicians combine imaging, laboratory trends, free-text notes, and, in some settings, genomic or physiological signals. Translating this integrative practice into model architectures is a central problem in medical multimodal learning. In this section, “omni-modality” refers not simply to accepting multiple inputs, but to interleaved and unified processing across text, high-resolution imaging, genomic sequences, and continuous physiological signals, with explicit attention to integration mechanisms, failure modes, and evaluation practices.

3.1. The Landscape of Medical Multimodal Large Models: From Alignment to Unification

Medical multimodal systems face constraints that are less prominent in general-domain models: fine-grained anatomy, high-resolution modalities such as gigapixel histopathology, and limited aligned image–text supervision. Generalist biomedical AI and multimodal medical assistants have already shown that language, imaging, and structured clinical signals can be combined in a single system, but evaluation remains task- and dataset-dependent [3,74,75]. The development of Med-MLLMs can be organized into three phases: discriminative alignment through contrastive pre-training, modular projection with instruction tuning, and native omni-models trained on interleaved multimodal sequences. These phases differ in data requirements, training stability, compute cost, and clinical faithfulness, and therefore should be interpreted as deployment trade-offs rather than a simple linear progression.

3.1.1. Phase I: The Foundation of Vision-Language Alignment

Before the emergence of generative MLLMs, the field was dominated by contrastive learning paradigms inspired by CLIP (Contrastive Language-Image Pre-training) [76]. In the medical domain, models such as ConVIRT [77], GLoRIA [78], BioViL [36], MedCLIP [37], and CheXzero [35] established the baseline for aligning visual representations with medical text.

ConVIRT pioneered the bidirectional optimization of image and text encoders using paired medical data, demonstrating that learning from naturally occurring clinical reports yields representations superior to those derived from ImageNet transfer learning. Building on this, GLoria introduced a global-local representation learning mechanism, addressing the critical issue that global image embeddings often fail to capture small but clinically significant lesions. By utilizing attention mechanisms to align local image sub-regions with specific words in the medical report, GLoria significantly enhanced the granularity of cross-modal understanding.

These early architectures were limited by their discriminative objective. They performed well in zero-shot classification and retrieval tasks, such as matching an X-ray to a diagnostic label, but they could not explain why a diagnosis was made or participate in multi-turn clinical dialogue. In practice, they provided visual representations that later systems could connect to Large Language Models for more flexible reasoning.

3.1.2. Phase II: The Era of Modular Projection and Instruction Tuning

A subsequent phase produced many Med-MLLMs based on a “connector-based” architecture. This design typically freezes a pre-trained visual encoder, such as BioMedCLIP [67] or SigLIP [79], and a pre-trained LLM, such as Llama [80], alignment-tuned LLaMA variants [81], or instruction-tuned clinical backbones [82,83], and links them through a learnable interface or projector.

Architectural Mechanisms: The primary challenge in this phase was the misalignment between the dense, continuous feature space of vision and the discrete token space of language. LLaVA-Med [84], a seminal work in this domain, adapted the general-domain LLaVA architecture to biomedicine. It employed a simple Multi-Layer Perceptron (MLP) to project visual features into the word embedding space of the LLM. The training process was bifurcated into two stages: (1) Biomedical Concept Alignment, where the projector was trained on large-scale image-caption pairs (e.g., from PMC-OA) to align visual tokens with medical terminology; and (2) End-to-End Instruction Tuning, where the entire model (or just the projector and LoRA adapters) was fine-tuned on diverse instructions to facilitate conversational capabilities.

Other architectures explored more complex projection mechanisms. For instance, models utilizing the Q-Former architecture (inspired by BLIP-2 [85]) attempted to compress visual information into a fixed number of query embeddings. This allowed for more efficient token usage but occasionally resulted in the loss of fine-grained visual details crucial for diagnosis.

The Data Bottleneck and Solutions: This phase also made clear that architecture alone was insufficient without high-quality instruction data. Medical queries often require reasoning rather than simple description. Researchers therefore developed scalable pipelines for synthetic data generation. PMC-LLaMA [86] and RadFM [34] showed that stronger models, such as GPT-4, could convert static image captions into Question-Answer (QA) pairs. This “synthesis-then-filter” strategy improved performance on benchmarks such as VQA-RAD and SLAKE, although hallucination remained a concern because the visual encoder and language model were still only loosely coupled.

3.1.3. Phase III: Native Omni-Models and Interleaved Processing

The field is now moving toward “Native Omni-models”. Unlike modular approaches that treat images as attachments to a text-based system, native models are trained from scratch, or extensively continued-pretrained, on interleaved sequences of multimodal tokens [8,9,87].

Unified Tokenization: The core innovation here is the distinct treatment of visual data. Instead of merely projecting features, new architectures utilize discrete visual tokenization or continuous streaming inputs that the LLM processes natively. Med-Gemini [74], building upon the Gemini 1.5 architecture, exemplifies this shift. By supporting a massive context window and native multimodal understanding, Med-Gemini can process not just single images but entire 3D CT volumes and long-form surgical videos as continuous inputs. This allows for temporal reasoning, such as tracking a tumor’s growth across longitudinal scans, which was previously impossible with single-image connector models.

Interleaved Reasoning: A second feature of this phase is the ability to handle interleaved text-image inputs and outputs. In clinical workflows, a physician may review patient history, laboratory trends, imaging, pathology slides, and guidelines in the same reasoning episode. Multimodal instruction-tuned and early-fusion models such as LLaVA-style systems [88] and specialized variants of Llama-4-Med are designed to ingest this heterogeneous stream as a coherent context. By avoiding overly aggressive visual-token compression, these architectures may better preserve the spatial details needed to detect subtle findings such as microcalcifications in mammography or ground-glass opacities in lung CT.

3.1.4. Challenges in High-Dimensional Medical Modalities

Although 2D imaging tasks such as X-ray, fundus photography, and dermatology have progressed quickly, high-dimensional data remain difficult for Med-MLLMs. The main examples are Whole Slide Imaging (WSI) in pathology and volumetric imaging in radiology.

The Gigapixel Challenge in Pathology: A standard WSI can exceed $100,000\times 100,000$ pixels, far surpassing the context limits of standard vision transformers which typically operate at $224\times 224$ or $512\times 512$ resolutions. To address this, current approaches utilize Hierarchical Image Pyramid Transformers (HIPT) [89], transformer-based pathology pretraining [23], or Multiple Instance Learning (MIL) adapters [21,90] to aggregate features before feeding them into the LLM. Recent large pathology models such as CONCH [91] are also building visual encoders specifically for histopathology, enabling MLLMs to perform zero-shot tumor segmentation and subtyping from tissue architecture.

Volumetric and Temporal Dimensions: For 3D data (CT/MRI), simple slice-by-slice encoding disrupts the spatial continuity of anatomical structures. Emerging architectures use 3D-aware visual encoders or slice-perceiver mechanisms to encode a volume into a sequence of embeddings, allowing the LLM to approximate 3D reconstruction [45,46]. Similarly, surgical video analysis requires models to process long temporal sequences when identifying surgical phases or adverse events [39,40]. State-space models (SSMs) such as Mamba [92] are useful in this setting because they offer linear-complexity scaling for long sequences, unlike the quadratic complexity of standard Transformers.

3.1.5. Landscape Summary

The movement from contrastive alignment to modular projection and then to native omni-models reflects deeper cross-modal integration, but no phase makes the others obsolete. Contrastive models remain useful for retrieval and representation learning; connector-based systems offer stability under data constraints; and native architectures may provide richer grounding at higher compute and data cost. Table 2 summarizes these trade-offs before the architectural comparison that follows.

3.2. From Late Fusion to Early Fusion: The Architectural Paradigm Shift

The depth of integration between vision and language is a key design choice. Late-fusion, connector-based systems treat vision as a translation problem by projecting image features into the language model’s embedding space. Early-fusion architectures instead interleave multimodal tokens throughout the backbone, allowing visual and textual representations to interact across layers. This choice affects whether fine-grained pathological features survive into the reasoning process, how long-context coherence is maintained, and how easily the system can be calibrated or governed.

3.2.1. Late Fusion: The “Connector” Bottleneck and Manifold Alignment

Late fusion, which has been common in MedQA and PubMedQA systems, is based on the modular composition of pre-trained experts. In this architecture, a visual encoder, typically a Vision Transformer such as CLIP-ViT [76] or SigLIP [79], is coupled with a Large Language Model (LLM) through a learnable interface, often referred to as a “projector” or “connector.”

Mechanics of Projection: Mathematically, let $X_v\in {\mathbb{R}}^{H\times W\times C}$ be an input medical image (e.g., a chest X-ray) and $X_t$ be the corresponding textual instruction. The visual encoder $E_v$ transforms the image into a sequence of patch features $Z_v=E_v\left(X_v\right)\in {\mathbb{R}}^{N\times D_v}$, where N is the number of patches and $D_v$ is the visual embedding dimension. The core challenge lies in the modality gap: the geometric mismatch between the visual manifold $Z_v$ and the LLM’s text embedding space $E_t$. The Projector $\mathcal{P}$ acts as a bridge function mapping $Z_v\to {\widehat{Z}}_v$ such that ${\widehat{Z}}_v\in {\mathbb{R}}^{M\times D_{llm}}$, where $D_{llm}$ matches the LLM’s input dimension.

$${\widehat{Z}}_v=\mathcal{P}\left(Z_v\right);Y=\mathrm{LLM}\left([{\widehat{Z}}_v,\mathrm{Embed}\left(X_t\right)]\right)$$

(1)

Common implementations of $\mathcal{P}$ include:

• Linear Projection: A simple matrix multiplication $W·Z_v+b$, used in LLaVA-Med [84]. While computationally practically free, it preserves the entire sequence length N, often leading to excessive token consumption for high-resolution medical images.
• Q-Former (Query Transformer): Introduced in BLIP-2 [85] and adopted by Med-Flamingo [93], this module uses a set of learnable query vectors to attend to $Z_v$ and extract a fixed number of visual tokens (e.g., 32 or 64). This performs aggressive compression.
• Perceiver Resampler: Used in Flamingo, utilizing latents to attend to the visual features, allowing for varying image resolutions to be compressed into a fixed sequence length.

The Information Bottleneck: Despite their usefulness, connector-based approaches can introduce an information bottleneck in medicine. Diagnosis often depends on high-frequency, low-semantics features: tiny calcifications, subtle texture changes in lung parenchyma, or faint ST-segment elevations in ECG plots. General vision encoders trained on natural images such as COCO or LAION are usually optimized for low-frequency, high-semantics objects, such as “dog” or “cat.” When a Q-Former or MLP projects these features into the language model, fine-grained pathological details may be smoothed out or discarded because they do not align with the pre-trained visual prior. In clinical settings, this compression risk matters because hallucinated normal findings or missed subtle abnormalities can be hidden behind fluent explanations, a failure mode repeatedly emphasized in reviews of healthcare LLMs [94].

3.2.2. Early Fusion: Native Tokenization and Omni-Perception

Early fusion, or native multimodal learning, integrates modalities inside the backbone rather than aligning two separate models after pretraining. These architectures tokenize modalities into a shared discrete vocabulary or continuous representation space and train a single transformer backbone from scratch, or through extensive continued pre-training, on mixed sequences.

Native Tokenization Strategies: Models like Gemini 1.5 [41], GPT-4o, and the open-source Chameleon [87] exemplify this approach. They employ unified tokenizers that treat visual patches identical to text subwords.

• Discrete Tokenization (VQ-VAE/VQ-GAN): Images are quantized into discrete codebook indices. An image is represented as a sequence of integers $[v_1,v_2,...,v_k]$ drawn from the same vocabulary size (e.g., 64k) as text. This allows the model to output images by simply predicting visual tokens autoregressively.
• Continuous Tokenization: Recent works like Llama-3-V suggest directly feeding continuous patch embeddings into the transformer without a separate encoder, treating pixel values as just another signal modality analogous to audio waveforms.

The clinical advantage is that transformer layers can attend to visual tokens directly rather than only to compressed summary vectors. This may preserve fine-grained features during reasoning, especially for temporal coherence in video analysis. In surgical video tasks such as Cholec80, early-fusion models can, in principle, track instrument trajectories and tissue deformation across frames with less information loss than frame-pooling adapters, which may improve adverse event detection. In broader autonomous-agent settings, such architectures also need explicit planning, tool use, and state tracking rather than perception alone [95].

3.2.3. When Late Fusion Remains Preferable in Practice

Despite the movement toward early fusion, late fusion remains attractive in many medical settings. Connector-based systems are often more stable to train because the visual encoder is already pre-trained and distributionally well behaved. They can also be more data-efficient when paired medical image–text data are limited or when institutions cannot share raw images for joint pretraining. Finally, late fusion supports modular governance: encoders can be replaced, task-specific calibration can be added, and human-in-the-loop constraints can be enforced at system boundaries. For these reasons, early fusion should be viewed as one part of the design space rather than a universal replacement.

3.2.4. Emerging Design Patterns: MoE Routing and Spatial Awareness

To understand why some early-fusion systems can support stronger multimodal reasoning, it is useful to highlight architectural patterns that are increasingly reported in modern large models. Two recurring themes are (i) Mixture of Experts (MoE) routing to scale capacity while controlling compute, and (ii) spatially-aware positional schemes that preserve 2D/3D structure when visual tokens are interleaved with text [96,97,98,99].

1. Multimodal Mixture of Experts (MoE): In a dense model (like LLaVA), every parameter is active for every token. In contrast, an MoE architecture activates only a subset of parameters (experts) for each token. For a medical omni-model, this allows for Modality Specialization within a unified backbone.

$$y={\displaystyle \sum _{i=1}^N}g{\left(x\right)}_i{E}_i\left(x\right)$$

(2)

where $E_i$ are expert networks (FFNs) and $g\left(x\right)$ is the gating function. In a multimodal setting, MoE routing can encourage specialization (e.g., experts that are more useful for visual texture tokens versus clinical-text tokens), which may improve parameter efficiency when modalities compete for capacity. Importantly, MoE does not automatically guarantee clinical reliability: routing instability, expert collapse, and calibration under domain shift remain open issues, and latency constraints can limit how much inference-time compute is acceptable in real deployments.

2. Interleaved Rotary Positional Embeddings (iRoPE): A major failure mode of standard Transformers in vision is the loss of 2D spatial relationships when flattening an image into a 1D sequence. Standard 1D RoPE (Rotary Positional Embedding) implies that patch $(i,j)$ is simply farther from patch $(i,j+1)$ in the sequence, weakening the concepts of above and below. iRoPE (or 2D-RoPE) extends the rotational concept to higher dimensions. It assigns position IDs $(row,col)$ to visual tokens and computes the rotary matrix based on Euclidean distance in the 2D grid, even when interleaved with text. For a medical report containing [Text → Image 1 → Text → Image 2], iRoPE maintains a global coordinate system. This is useful for grounding tasks such as locating a tumor relative to the carina. The model can estimate the 2D vector difference between the tumor query and the visual features, supporting bounding-box generation and anatomical localization that connector-based models may struggle to provide after spatial information has been compressed.

Table 3. Comparative Analysis of Late Fusion vs. Early Fusion Architectures in Medical AI.

Dimension	Late Fusion (Connector-based)	Early Fusion (Native Omni)
Archetypes	LLaVA-Med, Med-Flamingo, CheXagent	Gemini 1.5 Med, GPT-4o, Chameleon, Llama-4(V)
Visual Encoder	Frozen, pre-trained (CLIP, SigLIP)	None (or joint training from scratch)
Integration Point	Input Layer (via Projector)	All Layers (Native Token Interleaving)
Resolution limit	Fixed by encoder (usually ${336}^2$ or ${512}^2$)	Flexible (Any aspect ratio/resolution via tiling)
Reasoning Depth	Often shallower visual reasoning (bottleneck risk)	Potentially deeper multimodal reasoning (token-level grounding)
Training Stability	Often higher (frozen encoder stabilizes training)	Often lower (modality competition and long contexts)
Medical Hallucination	Often higher (visual/text decoupling risk)	Often lower when grounding is strong (but not eliminated)
3D/Video Capability	Often limited without additional temporal/3D adapters	Often stronger with long-context modeling (but compute-heavy)
Inference Cost	Lower (Small projector overhead)	Higher (Long context sequences)

Table 3. Comparative Analysis of Late Fusion vs. Early Fusion Architectures in Medical AI.

Dimension	Late Fusion (Connector-based)	Early Fusion (Native Omni)
Archetypes	LLaVA-Med, Med-Flamingo, CheXagent	Gemini 1.5 Med, GPT-4o, Chameleon, Llama-4(V)
Visual Encoder	Frozen, pre-trained (CLIP, SigLIP)	None (or joint training from scratch)
Integration Point	Input Layer (via Projector)	All Layers (Native Token Interleaving)
Resolution limit	Fixed by encoder (usually ${336}^2$ or ${512}^2$)	Flexible (Any aspect ratio/resolution via tiling)
Reasoning Depth	Often shallower visual reasoning (bottleneck risk)	Potentially deeper multimodal reasoning (token-level grounding)
Training Stability	Often higher (frozen encoder stabilizes training)	Often lower (modality competition and long contexts)
Medical Hallucination	Often higher (visual/text decoupling risk)	Often lower when grounding is strong (but not eliminated)
3D/Video Capability	Often limited without additional temporal/3D adapters	Often stronger with long-context modeling (but compute-heavy)
Inference Cost	Lower (Small projector overhead)	Higher (Long context sequences)

3.2.5. The Stability-Plasticity Dilemma in Multimodal Training

Although early fusion can provide richer cross-modal interaction, it also introduces a stability-plasticity dilemma, often described as modality competition. During training, gradients from visual losses, which may be high-variance and noisy, can dominate gradients from textual losses, which are often sparse and semantic. In early experiments with models such as Chameleon, this imbalance sometimes produced models that generated images well but degraded in text reasoning, which is unacceptable for medical applications.

Current Solutions: To mitigate this, researchers have introduced Modality-Specific Normalization (MoN). Instead of a shared LayerNorm, the model detects the token type (Text vs Image) and switches to a specific set of normalization parameters (scale $\gamma$ and shift $\beta$).

$$\mathrm{Norm}\left(x\right)={\displaystyle \frac{x-\mu }{\sigma }}·{\gamma }_{modality}+{\beta }_{modality}$$

(3)

This allows the distribution of image tokens (often high magnitude) to coexist with text tokens (lower magnitude) without destabilizing the shared attention weights. Furthermore, Curriculum Learning strategies (starting with text-only pre-training to establish reasoning circuits, then introducing aligned image-text pairs, and finally interleaved documents) have become the standard protocol for training robust Med-MLLMs.

3.2.6. Outlook

The shift from late to early fusion is best understood as an expansion of the design space. Late fusion remains practical when data, compute, or governance constraints dominate; early fusion becomes more attractive when a workflow requires deeper cross-modal grounding and long-context coherence. Architecture alone, however, does not ensure clinical reliability. A model may ground a tumor in a CT volume yet still hallucinate a treatment plan if its reasoning or calibration mechanisms are weak. The agentic systems discussed in Section 5, which may need to interleave vitals, imaging, speech, and text in real time, will likely benefit from richer integration only if evaluation, calibration, and governance improve in parallel.

4. From Pattern Recognition to Deliberative Reasoning in Medical AI

Passing a medical licensing exam is not the same as reasoning through a difficult case at the bedside. Models that perform well on medical question-answering benchmarks can still fabricate evidence, over-rely on misleading cues, or reach correct answers through invalid pathophysiological logic [17,100,101]. For clinical translation, the central question is not only whether a model contains enough facts, but how it applies them: whether it can search, verify, backtrack, and calibrate confidence in ways that are clinically useful. This section reviews mechanisms for eliciting such deliberative, “Logic Engine”, behavior, the supervision signals used to verify reasoning processes, and the cognitive value and operational cost of these approaches.

4.1. From Intuition to Deliberation: The Dual-Process Paradigm

The evolution of medical AI mirrors the Dual-Process Theory of human cognition, which distinguishes between Pattern Engine (fast, intuitive, heuristic-driven) and Logic Engine (slow, deliberate, logical) [7,102]. Traditional medical LLMs primarily operated as Pattern Engine agents, retrieving memorized medical facts through next-token prediction [100].

In high-stakes clinical environments, reliance on Pattern Engine behavior can lead to failures, particularly when facing atypical presentations or complex multi-step diagnostics. Recent reasoning models, most notably OpenAI’s o1/o3 and DeepSeek’s R1, suggest that more deliberative behaviors can be elicited by incentivizing step-by-step analysis characteristic of Logic Engine thinking [6,7]. In medicine, this transition is not merely a performance optimization; it is tightly linked to safety expectations and alignment with clinical guidelines [17,18].

4.2. Mechanisms of Logic Engine Medical Reasoning

The shift from Pattern Engine to Logic Engine behavior reflects a move beyond purely left-to-right token prediction toward search, verification, and adaptive computation [7,103,104,105]. Traditional medical LLMs often behave like probabilistic retrievers: they do well when answers are densely represented in training data, but they struggle with multi-step clinical deductions [100,106]. Recent reasoning work can be grouped into four mechanism families: inference-time compute scaling, tree-structured search, reinforcement-learning-driven self-correction, and adaptive thinking frameworks. These mechanisms should be evaluated not only by answer accuracy but also by faithfulness, calibration and abstention, robustness under clinical distribution shift, and latency or cost.

Figure 4. Representative mechanisms for deliberative reasoning, verification, and agentic behavior in medical large models. The figure compares AdaThink-Med, Agent Hospital/SEAL, DeepSeek-R1, DiDi, Med-MCTS, MedCoT, TumorChain, and VPRM, showing how adaptive reasoning, simulated self-evolution, reinforcement learning, dialogic roles, tree search, multimodal chain-of-thought, ROI-level grounding, and verifiable process rewards shift evaluation from final-answer accuracy toward process validity, evidence grounding, calibration, and human-verifiable decision pathways.

Figure 4. Representative mechanisms for deliberative reasoning, verification, and agentic behavior in medical large models. The figure compares AdaThink-Med, Agent Hospital/SEAL, DeepSeek-R1, DiDi, Med-MCTS, MedCoT, TumorChain, and VPRM, showing how adaptive reasoning, simulated self-evolution, reinforcement learning, dialogic roles, tree search, multimodal chain-of-thought, ROI-level grounding, and verifiable process rewards shift evaluation from final-answer accuracy toward process validity, evidence grounding, calibration, and human-verifiable decision pathways.

4.2.1. Test-Time Scaling: Compute and Accuracy

Recent work shows that reasoning performance can improve when additional computation is allocated during inference, sometimes independently of the pre-training compute budget [7,107]. While training scaling focuses on parameter count (P) and data volume (D), test-time scaling studies the relationship between generated tokens or reasoning paths (N) and final accuracy (A).

In clinical diagnostics, this idea appears in strategies such as Best-of-N Sampling and Majority Voting [104,107]. If a model generates N independent reasoning trajectories for one case, the probability of finding at least one correct path can be written as:

$$P\left(\mathrm{correct}\right)=1-{(1-p)}^N$$

(4)

where p is the probability of a single reasoning trace being correct [104]. However, simple voting is insufficient for complex pathology. Recent verification-oriented work instead scores intermediate traces with separate verifiers or reward models, so the final answer is selected by a confidence-weighted consensus rather than a raw majority [108,109,110]. Such “search-over-reasoning” approaches explicitly trade latency for diagnostic precision.

4.2.2. Tree-Structured Search and Monte Carlo Exploration in Medicine

Traditional autoregressive generation is limited by a left-to-right linear constraint, which prevents models from backtracking or exploring alternative diagnostic hypotheses once a reasoning path has begun [103,105,111]. To address this, medical reasoning architectures have begun to adapt Monte Carlo Tree Search (MCTS) and Tree of Thoughts (ToT) frameworks [111,112,113].

Recent medical slow-thinking frameworks illustrate this shift by modeling clinical diagnosis as a sequential decision-making process [113]. Unlike unconstrained search, medical reasoning systems need a clinically informed action space that can be organized into four stages:

1.

Key Symptom Extraction: Identifying pathognomonic features from clinical vignettes.

2.

Hypothesis Generation: Proposing a differential diagnosis (DDx) list based on symptoms.

3.

Evidence Verification: Traversing medical knowledge graphs (KGs) to find supporting or contradicting data for each DDx.

4.

Deductive Analysis: Re-weighting the probability of each diagnosis based on the evidence gathered.

The search process involves a cycle of Selection (choosing the most promising reasoning branch), Expansion (generating new clinical steps), Simulation (rolling out the path to a terminal diagnosis), and Backpropagation (updating the value of previous nodes based on final correctness) [111,113]. By integrating structured knowledge substrates (e.g., SPOKE or specialized medical KGs), these models can ground the expansion process in biological facts rather than statistical hallucination [53,113].

4.2.3. Reinforcement Learning: GRPO and Self-Correction

Another route to Logic Engine behavior is reinforcement learning (RL), which can elicit reasoning patterns without requiring extensive human-annotated chains of thought [6,114,115]. The Group Relative Policy Optimization (GRPO) algorithm, popularized by DeepSeek-R1, is one example of this training approach [6].

During the DeepSeek-R1-Zero training phase, the model reportedly learned forms of self-correction [6]. This behavior is often marked by reflective tokens such as “wait”, “let me re-evaluate”, or “but actually” [6,116]. In medicine, this behavior is relevant for cases with contradictory evidence. For example, during reasoning about suspected pulmonary embolism, a model might initially consider anticoagulation but revise its reasoning after processing a negative D-dimer result [6].

Research indicates that these reflective behaviors are not merely stylistic but may be linked to model-internal states before reflective tokens are generated [116]. For clinical use, the practical implication is that self-correction should be evaluated against external evidence and outcome validity, because a plausible reasoning trace can still be unfaithful to the computation that produced the final answer [101,117].

4.2.4. Adaptive Reasoning and Uncertainty

A practical challenge for reasoning models is overthinking: the tendency to apply expensive, token-heavy reasoning to simple queries that could be answered directly [107,118]. In clinical practice, efficiency matters alongside accuracy.

The AdaThink-Med framework [118] addresses this via Uncertainty-Guided Length Calibration. The mechanism employs an entropy-based difficulty estimator to evaluate the input complexity:

$$H\left(x\right)=-\sum P\left(y\right|x)logP(y\left|x\right)$$

(5)

If the entropy of the initial token distribution is low, the model triggers a “non-thinking” mode and avoids a long reasoning trace [118]. For high-entropy cases, such as rare diseases or atypical presentations, a difficulty-aware reward encourages longer reasoning [118]. This adaptive routing can reduce average reasoning length while preserving performance on complex diagnostic tasks [118].

4.2.5. Multimodal Grounding in Reasoning

Reasoning mechanisms also apply to medical vision-language models (MVLMs), where the reasoning process must remain grounded in pixel-level evidence. The Interleaved Multimodal Chain-of-Thought (M-CoT) mechanism, used in models such as TumorChain [119] and LLaVA-style step-by-step vision-language reasoning [120], asks the model to generate intermediate rationales tied to anatomical regions of interest (ROIs) before producing a pathology-level conclusion.

A key mechanism here is Look-Twice Feature Matching (LTFM), reported in TumorChain [119]. It approximates human visual inspection by first identifying suspicious abnormal tokens in an image and then re-reading those tokens with enhanced attention during the Logic Engine reasoning phase. This iterative refinement helps MVLMs address weak vision-language grounding, where models may ignore critical visual clues, such as nuclear shape in hematology, in favor of text-based heuristics [119,120].

4.2.6. Summary of Mechanistic Advancements

Across test-time scaling, tree-structured search, RL-driven self-correction, and adaptive compute allocation, the focus shifts from how much a model knows to how well it uses what it knows. A major open question is faithfulness: whether generated reasoning traces actually drive predictions or instead rationalize answers produced by other internal processes. Until this is resolved empirically, the clinical value of “slow thinking” should depend on external verification rather than on the apparent plausibility of the trace alone.

4.3. Reinforcement Learning with Verifiable Rewards (RLVR)

A critical hurdle in medical reasoning is the lack of objective, verifiable signals to guide model optimization during reinforcement learning. Unlike coding or mathematics, many clinical judgments involve nuance, incomplete information, and preference-sensitive trade-offs. In this subsection, we review RLVR primarily as a mechanism for process supervision; broader questions about clinical validation, governance, and lifecycle monitoring remain essential for deployment.

4.3.1. Reward Modeling: Outcome vs. Process

Outcome Reward Models (ORM) only provide feedback based on the final answer [6,108]. While useful for multiple-choice questions (e.g., USMLE), ORMs are prone to reward hacking and fail to penalize logical fallacies that coincidentally lead to the correct diagnosis [109,121,122]. Process Reward Models (PRM) offer fine-grained supervision at each intermediate step [109,113]. Recent frameworks like MedS³, Med-PRM, and medical RLVR variants utilize MCTS or verifiable checks to construct rule-verifiable reasoning trajectories, where steps that degrade the node value are penalized [110,113,123].

4.3.2. Verifiable Process Reward Models (VPRM)

To mitigate the bias of neural judges, recent research has introduced Verifiable Process Reward Models (VPRM) [122]. VPRMs replace subjective neural scoring with deterministic, rule-based verifiers based on clinical guidelines [122]. Applied to medical evidence synthesis and risk-of-bias assessments, VPRM-trained models achieved up to a 20% improvement in F1 scores over state-of-the-art outcome-only models, ensuring higher coherence between step-level decisions and final labels [122].

4.4. Clinical Cognitive Value: Mitigating Diagnostic Biases

Beyond raw accuracy, the primary clinical advantage of Logic Engine reasoning is its ability to counteract well-documented human cognitive biases that contribute to diagnostic error [124,125].

4.4.1. Anchoring Bias and Confirmation Bias

Anchoring bias occurs when a clinician fixates on initial information and fails to adjust to new evidence [102,124]. Multi-agent and role-based frameworks, such as “red team” or “devil’s advocate” roles, are being explored as ways to make models re-evaluate misleading initial assumptions rather than merely confirm them [12,95]. Comparative medical reasoning studies of models such as DeepSeek-R1 suggest that reasoning-centric models may retrieve and use more diagnostic clues than more pattern-centric baselines, although such gains require careful clinical validation [6].

4.4.2. Premature Closure

Premature closure: the tendency to stop considering other hypotheses once a plausible diagnosis is reached—is a major contributor to diagnostic error [124,125]. Reasoning frameworks like MedCoT and TumorChain utilize hierarchical expert verification and interleaved multimodal CoT to encourage models to rank and rule out competing hypotheses [119,126]. By forcing an analytic pathway that summarizes case features before committing to a diagnosis, these models aim to reduce premature conclusions driven by superficial pattern matching [119,126].

4.5. Slow Thinking in Clinical Practice: Clinical Value and Cost

The mechanisms above raise a practical question: what clinical value does deliberative reasoning provide beyond fast pattern matching, and at what cost? The gap between exam-level accuracy and bedside reliability is well documented [17,18,101]. Slow-thinking models may help narrow this gap by reducing some cognitive biases, supporting collaborative deliberation, and making uncertainty more visible, but they also introduce latency, compute cost, and workflow burden.

4.5.1. Addressing the Performance-Reliability Gap

The main reason for studying Logic Engine behavior in medicine is that high benchmark scores do not translate directly into clinical reliability [17,18]. Traditional models can reach correct conclusions through “stochastic retrieval” rather than causal deduction [100,106]. This gap is especially visible in evaluations that test abstention, false premises, or faithfulness, where models may fabricate justifications for incorrect premises [101,117].

Logic Engine medical AI addresses this by introducing a reasoning phase that approximates parts of hypothetico-deductive reasoning [7,102]. When a model is encouraged to generate or use reasoning traces that include self-correction and re-evaluation, it can move beyond simple knowledge retrieval toward a more analytical form of decision support [6,6].

4.5.2. Mitigating Cognitive Biases through Deliberate Reasoning

Cognitive factors contribute substantially to diagnostic error in medicine [124,125]. Artificial slow thinking may help by introducing cognitive friction and systematic exploration, although such benefits need clinical validation.

Combatting Anchoring and Confirmation Bias

Anchoring bias, the tendency to fixate on initial information, such as a patient’s chief complaint, while ignoring subsequent contradictory evidence, is a natural target for reasoning models [102,124]. Non-reasoning models may also exhibit demographic or social bias, over-weighting patient attributes in ways that are not clinically justified [127,128].

In contrast, Logic Engine frameworks utilize mechanisms like Look-Twice Feature Matching (LTFM) and stepwise multimodal re-examination [119,120]. These allow the model to:

1.

Identify initial suspicious features (the anchor).

2.

Re-read or re-examine clinical evidence in light of the anchor to seek counter-evidence.

3.

Backtrack in the reasoning trace when evidence (e.g., a negative D-dimer in a suspected pulmonary embolism) contradicts the initial hypothesis [6].

Studies on multi-agent frameworks, such as “red team” or “devil’s advocate” agents, suggest that forcing a model to challenge its own working diagnosis can surface alternative hypotheses and reduce acquiescent confirmation of the user’s first impression [12,95].

Preventing Premature Closure

Premature closure is the tendency to stop the diagnostic process once a plausible explanation is found [124]. Reasoning models counteract this by employing Tree of Thoughts (ToT) and search-based expansion to maintain a wider search space [111,113]. The TumorChain framework [119] and the MedCoT approach [126] exemplify this by requiring the model to rank and rule out relevant differentials before committing to a final diagnosis. By enforcing a structured sequence of Key Symptom Extraction → Hypothesis Generation → Evidence Verification → Deductive Analysis, these models aim to prevent the shallow convergence typical of Pattern Engine agents [113,126].

4.5.3. Collaborative Deliberation: Multi-Agent and Dialogic Frameworks

Recent work has also shifted attention from monolithic models to agentic systems that simulate parts of multidisciplinary consultation.

Role-Based Deliberation and the SEAL Mechanism

Frameworks such as Agent Hospital, which uses the SEAL mechanism, move the interaction from passive retrieval toward structured deliberation [12]. More generally, autonomous-agent surveys emphasize that role assignment, tool use, memory, and planning are core ingredients for moving from one-shot QA to coordinated problem solving [95]. In clinical deliberation, the AI system may assume or coordinate several complementary roles:

• Framework Coach: Ensures adherence to structured reasoning protocols.
• Socratic Guide: Asks probing questions to the human physician to surface missing data.
• Red Team Partner: Presents evidence-based alternative diagnoses.

The SEAL (Simulacrum-based Evolutionary Agent Learning) mechanism allows agents to refine diagnostic strategies by treating synthetic cases in a sandbox environment [12]. This gives models exposure to simulated failures that would be unethical to stage in a real hospital, and may help them learn useful reasoning trajectories for rare or complex diseases [12].

Counterfactual Reasoning and Demographic Fairness

Clinical reasoning often requires asking what might have happened if a single variable were different. Counterfactual Reasoning benchmarks such as MEDEQUALQA allow researchers to audit reasoning stability [128]. By perturbing variables such as patient gender or pronouns while holding symptoms constant, researchers have identified localized divergences in reasoning traces even when the final diagnosis remains unchanged [128]. Logic Engine models can address this problem only if their stated reliance on risk factors is faithful and externally verifiable [101,117].

4.5.4. Efficiency vs. Depth

Not every medical query requires a long reasoning chain. A question such as “What is the standard dose of aspirin?” may be better handled by a direct, grounded answer than by an extended DeepSeek-R1-style trace [6,118].

Adaptive Inference and AdaThink-Med

To balance latency and accuracy, frameworks such as AdaThink-Med use Uncertainty-Guided Length Calibration [118]. This mechanism uses an entropy-based estimator to determine problem difficulty. If the initial token distribution is confident, the model uses a non-thinking mode to conserve compute [118]. For high-uncertainty cases, it triggers an extended thinking mode. This adaptive routing achieves up to a 6.4x reduction in average reasoning length while preserving diagnostic performance [118].

Cognitive Surrender and the Role of Explainability

As AI reasoning becomes more fluent and persuasive, automation bias becomes more concerning: clinicians may adopt AI outputs with insufficient scrutiny [102]. Research on the faithfulness of reasoning traces is ongoing, and some studies suggest that models may produce post-hoc rationalizations rather than faithful accounts of their internal computation [101,117]. For clinical safety, Logic Engine AI should remain a co-pilot: its reasoning trace should be treated as a map for physician review, not as evidence by itself [17,18].

4.5.5. Summary

Slow-thinking capabilities give MedLMs a more explicit way to search, verify, and revise clinical hypotheses. These capabilities may reduce some diagnostic shortcuts, but they are not sufficient on their own. Their clinical value depends on three safeguards: faithful reasoning traces, external verification against clinical evidence, and adaptive deployment that routes simple cases to efficient responses while reserving extended reasoning for uncertain or high-stakes decisions.

4.6. Conclusions

The reasoning methods reviewed here extend medical AI beyond fact recall toward diagnosis through search, verification, and self-correction. They also expose a practical tension: deeper reasoning requires more tokens, time, and energy, while clinical workflows often demand rapid answers. Near-term progress will depend on adaptive compute routing, faithful reasoning verification, and human-in-the-loop designs that balance analytical depth with bedside urgency.

5. From Instruction Following to Autonomous Planning and Collaboration

A model that can reason through a differential diagnosis but cannot order the confirmatory test, check the patient’s allergy list, or coordinate with a pharmacist remains a sophisticated question-answering system rather than a clinical tool. Healthcare delivery is a multi-stage workflow with feedback loops, handoffs, and safety checks. It requires systems that maintain state, decompose goals, and interact with external tools under explicit constraints [12,95,129,130,131,132]. We use medical agents to denote such systems and review three threads in their development: simulation-driven skill acquisition, multi-agent collaboration, and long-horizon planning. Throughout, we use a dual evaluation lens: task success and safety, including constraint violations, error recovery, auditability, and clinician workload.

Table 4. A survey taxonomy of medical agents: representative roles, tool environments, key risks, and evaluation focus.

Agent type	Typical tools / environment	Primary risks	Evaluation focus
Documentation & coding agent	Note drafting, order summarization, ICD/CPT suggestion, guideline retrieval	Hallucinated facts, privacy leakage, over-trust by clinicians	Factuality vs source; edit distance to clinician notes; PHI leakage tests; time saved
Triage & patient-facing agent	Symptom collection, routing, patient education	Mis-triage, unsafe reassurance, biased recommendations	Safety policy adherence; calibrated uncertainty/abstention; equity stratification
Workflow orchestration agent	EHR search, scheduling, messaging, order set suggestions	Tool misuse, incorrect actions, silent failure modes	Action correctness; rollback/recovery; audit logs; human-approval gating
Perception-action clinical agent	Imaging navigation, measurement, report assistance	Missed subtle findings; spurious correlations; poor handoff to clinician	Lesion-level sensitivity; robustness across sites; handoff quality and confidence
Multi-agent team	Role-based agents (doctor/nurse/pharmacist), structured debates	Coordination failure, compounding errors, inconsistent state	Consistency across agents; disagreement resolution; error containment

Table 4. A survey taxonomy of medical agents: representative roles, tool environments, key risks, and evaluation focus.

Agent type	Typical tools / environment	Primary risks	Evaluation focus
Documentation & coding agent	Note drafting, order summarization, ICD/CPT suggestion, guideline retrieval	Hallucinated facts, privacy leakage, over-trust by clinicians	Factuality vs source; edit distance to clinician notes; PHI leakage tests; time saved
Triage & patient-facing agent	Symptom collection, routing, patient education	Mis-triage, unsafe reassurance, biased recommendations	Safety policy adherence; calibrated uncertainty/abstention; equity stratification
Workflow orchestration agent	EHR search, scheduling, messaging, order set suggestions	Tool misuse, incorrect actions, silent failure modes	Action correctness; rollback/recovery; audit logs; human-approval gating
Perception-action clinical agent	Imaging navigation, measurement, report assistance	Missed subtle findings; spurious correlations; poor handoff to clinician	Lesion-level sensitivity; robustness across sites; handoff quality and confidence
Multi-agent team	Role-based agents (doctor/nurse/pharmacist), structured debates	Coordination failure, compounding errors, inconsistent state	Consistency across agents; disagreement resolution; error containment

5.1. Embodied Simulation and Self-Evolution

Static training on textbooks and de-identified records provides knowledge but not procedural skill. Agents must learn to gather information, act, and recover from errors within workflow constraints. High-fidelity clinical simulations, or digital twins that instantiate triage, consultation, and diagnostic nodes with simulated patients, nurses, and clinicians [12,95,132,133], offer a controlled environment for iterative skill acquisition over long-horizon trajectories without exposing patients to risk. The key methodological concern is the sim-to-real gap: whether simulated rewards, patient models, and tool APIs approximate clinical variation closely enough for skills to transfer, and whether safety failures in simulation reliably predict failures in deployment.

The Self-Evolutionary Arbitrating Learning (SEAL) framework supports this type of iterative learning. Unlike traditional Supervised Fine-Tuning (SFT), which is usually a one-time training event, SEAL defines a Med-LLM-in-the-loop lifecycle [12]. The agent’s clinical performance is captured as a Diagnostic Trajectory, a step-by-step reasoning log from the initial patient inquiry to the final therapeutic recommendation. The trajectory is then evaluated through Arbitration, where a built-in arbitrator, functioning as a virtual chief resident, compares the agent’s decisions with medical gold standards such as WHO guidelines or established clinical pathways. When an error is detected, the system performs Reasoning Gap Analysis to identify the source of failure, such as a missed laboratory marker or misprioritized symptom, and converts that error into structured feedback for optimization.

5.2. Multi-Agent Social Collaboration

Medical practice is inherently collaborative, and agentic systems increasingly model this complexity by moving beyond a “single-expert” assistant toward multi-agent organizations that mimic real hospital teams [12,95,134]. This logic is often operationalized via role-playing: agents are assigned personas such as Doctor, Nurse, and Pharmacist, each governed by explicit role constraints to encourage specialization. For instance, a pharmacist agent may prioritize contraindications and dosing interactions while a physician agent focuses on differential diagnosis and care planning. A related research direction is Theory of Mind (ToM), where an agent maintains a model of other agents’ knowledge and intentions. From an evaluation viewpoint, multi-agent systems introduce new failure modes beyond single-agent hallucination: coordination breakdowns, inconsistent shared state, and compounding errors across roles. Consequently, surveys should emphasize not only task success but also disagreement handling, escalation protocols, and error containment.

5.3. Autonomous Task Execution and Long-horizon Planning

The most difficult agentic setting is long-horizon task execution, such as managing a multi-month clinical-trial workflow or coordinating recruitment and scheduling. To handle extended timelines, agents use hierarchical task decomposition, including CoT/ToT-style planning, to break broad goals such as “Conduct Phase II recruitment” into sub-tasks such as inclusion screening, EHR scanning, and subject scheduling [105,111,129]. This is typically coupled with dynamic tool interaction, including calls to EHR APIs, scheduling systems, or statistical tools such as R and Python [130,135]. From a safety perspective, long-horizon autonomy raises two concerns: compounding error, where small mistakes accumulate across steps, and action governance, where irreversible actions must require human approval, logging, and rollback options. Evaluation should therefore track not only final success but also step-level correctness, recovery behavior, audit quality, and clinician burden.

6. Conclusions

Medical large models are beginning to move clinical AI from single-purpose predictors toward a capability stack that spans perception, knowledge grounding, reasoning, and workflow action. Across modalities, progress is driven not only by scale but also by mechanisms: long-context and native multimodal integration for richer grounding; retrieval, knowledge graphs, and memory for factual continuity; and test-time search and verifiable reward learning for more deliberate reasoning. At the same time, these advances expose a translational tension: fluency and benchmark accuracy can improve faster than reliability, calibration, and safe integration into real clinical processes.

Three priorities follow from this review. First, evaluation should move beyond static exam-style scores toward stress-tested, workflow-aware, and lifecycle-maintained assessments that measure robustness, abstention, provenance, and action safety [136,137,138]. Second, deployment should treat agents as bounded systems, with least-privilege tools, human approval gates, auditable memory, and explicit rollback pathways. Third, governance should be implemented as engineering practice, including data stewardship, decontamination, monitoring for drift, and incident reporting, rather than appended after model development [139,140,141,142]. If these technical and socio-technical constraints are addressed, medical foundation models may move from impressive demonstrations toward dependable clinical infrastructure.