Orchestrating LLMs with Specialized Models: A Survey on Heterogeneous Multi-Model Agents

1. Introduction

Large language models (LLMs) have shown strong abilities in instruction following [110,196], reasoning [69,80], and planning [156,163]. Modern multimodal LLMs extend this ability to images [13,101], audio [42,57], and videos [40,176]. For many tasks, a single LLM can now read the input, reason over it, and produce a useful answer.

However, many real tasks still require abilities that a single LLM cannot reliably handle. A vision-language model may describe a scene but miss small objects. A language model may write a prompt but cannot by itself render a high-resolution image, generate audio, or build a 3D scene. A robot planner may list steps but cannot directly produce low-level control signals.

This has led to a fast-growing design pattern in which an LLM is connected with specialized learned models. The specialists include perception models such as GroundingDINO [106] and Depth Anything [96], generative models for image and speech synthesis [103,138,180], and action policies such as GSB [22] and Diffusion Policy [16]. The LLM reads the task and coordinates suitable specialists for answering, media generation, or action execution. These specialists are not deterministic tools. They are learned probabilistic models with their own errors and uncertainty.

We call this paradigm Heterogeneous Multi-Model Agent (HMMA). In an HMMA, the LLM acts as the central reasoning module, while specialized non-LLM models handle perception, generation, and action execution. The system is heterogeneous because the modules differ in architecture, training objective, and modality. It is agentic because the LLM provides high-level reasoning and coordination for the overall process.

These systems are promising because they combine flexible LLM reasoning with the strengths of specialized models. This modular design also introduces system-level challenges. Interfaces between models can lose information. Errors can cascade through the system. Uncertainty is difficult to track across the full pipeline. Evaluation also requires end-to-end analysis because a strong LLM and a strong specialist do not always form a reliable system. These issues make HMMA design a system-level problem rather than merely a matter of model selection.

Several related surveys cover nearby areas, including compound AI systems [10,35,85,131], LLM agents [79,185], multi-agent collaboration [155,198], multimodal LLMs [20,70,188], and embodied AI [108]. However, the core difficulty of HMMAs is not tool use or multimodality alone. It is the orchestration of heterogeneous learned models whose errors, uncertainty, and interface losses interact across the pipeline. HMMAs make this interaction explicit. The errors, uncertainty, and interface losses of distinct learned specialists propagate through explicit connections between separate models, and the system’s reliability depends on how this propagation is managed. Our survey provides the first unified account of such systems. We study systems that keep multiple learned models as separate components and connect them through explicit interfaces.

Our main contributions are as follows.

• We establish a formal definition for Heterogeneous Multi-Model Agents, where a central LLM coordinates probabilistic, specialized non-LLM models. We strictly distinguish HMMAs from deterministic tool-augmented LLMs, homogeneous multi-LLM systems, and unified multimodal LLMs by emphasizing their modularity and heterogeneous architectures.
• We review papers from major AI venues between 2023 and 2026, and identify 572 HMMA papers that meet our criteria. To organize the field, we categorize existing systems into five core interaction patterns based on their use of perception, generation, and action models. Building on this taxonomy, we map the literature across distinct application domains.
• We analyze HMMA design choices along five dimensions: information flow, interfaces, coupling, feedback, and uncertainty handling. We further discuss evaluation principles and summarize open challenges such as error cascading, interface bottlenecks, shallow alignment, and evaluation gaps.

The rest of the paper is organized as follows. Section 2 defines HMMA and contrasts it with adjacent paradigms. Section 3 presents the five interaction patterns. Section 4 analyzes architectural design choices. Section 5 reviews the largest application domains. Section 6 discusses evaluation and benchmarking. Section 7 discusses open challenges and future directions. Section 8 concludes the survey.

2. Background

2.1. Definition

We define an HMMA as a system in which a central LLM coordinates one or more specialized non-LLM deep learning models to accomplish tasks that no single model can solve alone. The definition has three requirements. First, there is one central LLM that provides high-level reasoning or coordination for the system. Second, the system uses at least one non-LLM specialist that is itself a learned model. Third, the task is one that the LLM alone cannot finish, so the cooperation is not optional. Systems that satisfy all three points are HMMAs in our sense, and the rest of this paper studies how the cooperation is structured in practice.

Two properties are essential to this definition. First, the specialized models must be heterogeneous. They differ from the central LLM and from each other in architecture, training objective, or input and output modalities. For instance, an object detector and a denoising diffusion model differ from the central LLM in all these respects. A system composed entirely of LLMs communicating in natural language does not qualify as an HMMA because all the models involved share the same basic foundation. Even when those LLMs play different roles, the underlying representation and training paradigm are too similar to count as a heterogeneous mix in the sense we care about.

Second, each specialized model must be a learned probabilistic model with its own internal representations, such as an object detector or a diffusion model. The output of such a model carries uncertainty that depends on the data the model has seen and the loss it was trained with. We exclude deterministic rule-based programs (e.g., calculators, search engines, code interpreters) because their outputs do not carry the same kind of statistical risk and their failures look very different. A calculator either returns the right number or throws a domain error, while a detector may return a high-scoring box on the wrong object.

The specialized models can be grouped into three functional roles based on their relationship to the external environment, though an HMMA may use only a subset of these roles:

• Perception models extract structured information from raw sensory inputs. Examples include object detectors [106], segmentation models [83], depth estimators [96], OCR engines [46,143], and speech recognition models [66,134]. They turn pixels, waveforms, or screens into labels, boxes, masks, or text.
• Generation models synthesize new content in various modalities. Examples include image diffusion models [138], text-to-speech systems [43,82], video generators [32,182,202], and 3D asset creation models [81]. They turn a textual or structured plan into a concrete artifact such as an image, a clip of audio, or a 3D mesh.
• Action models produce executable actions that affect the environment. Examples include robot manipulation controllers [16] and autonomous driving controllers [139]. They turn a high-level goal into low-level commands that hardware or software can run.

During execution, the central LLM receives a task description along with optional environmental observations and coordinates specialized models through explicit interfaces. This coordination may appear as a fixed pipeline, a planned sequence of model calls, or an iterative loop over intermediate results. In iterative settings, the LLM may read intermediate replies, assess whether they are useful, and continue or revise the workflow. The inter-model interface governs how information flows between the LLM and the specialized models, ranging from discrete symbolic representations (natural language descriptions, bounding box coordinates, structured JSON) to continuous representations (embeddings, soft prompts, shared hidden states).

2.2. Scope and Distinctions

The HMMA paradigm intersects with several related research areas. As summarized in Figure 1, we contrast it with four adjacent paradigms below. For each one we state the criterion that decides whether a borderline system qualifies as an HMMA, so that the same criterion may be applied to systems not covered here.

2.2.0.1. Compound AI Systems.

Compound AI systems [35,85] and tool-augmented agents [10,109] equip LLMs with external capabilities such as calculators, web search APIs, code interpreters, and visual tools. Such tools obey fixed rules and return the same output for the same input. The specialized models in an HMMA are instead probabilistic, so their outputs carry inherent uncertainty. An object detector may miss occluded objects, a diffusion model may generate artifacts, and a reinforcement learning (RL) policy may select suboptimal actions. When such modules are chained, their uncertainties compound through the pipeline, creating error-propagation dynamics that are absent in tool-augmented systems. For instance, a failure in a calculator stops the chain with a clear error, while a failure in a perception model often slips through, since the LLM has no clean way to know that the box it just received is wrong. This is one reason why the design space of HMMAs is broader and richer than that of compound AI systems even when the surface architecture looks similar.

2.2.0.2. Multi-agent Systems.

Multi-agent systems [27,79,155,198,203] deploy multiple LLM instances that collaborate through debate, negotiation, or role-playing. They are homogeneous in modality and communicate through natural language. A doctor agent [153] and a lawyer agent [29] in such a system share the same backbone and differ only in prompt. By contrast, HMMAs are heterogeneous. The LLM serves as the “brain” for high-level reasoning, while specialized models act as “sensory organs” and “limbs” for modalities the LLM cannot process natively. Inter-model communication extends beyond natural language to include modality-specific data such as boxes, masks, and continuous embeddings. A multi-agent system can be lifted into an HMMA when at least one of its agents is replaced by a non-LLM specialist that handles a different modality.

2.2.0.3. Multimodal LLMs (MLLMs).

MLLMs [15,20,70,188] integrate multiple modalities into a unified LLM-centered inference framework. A single forward pass takes images and text and returns text. HMMAs instead keep specialized models as external components with separate parameters, inference processes, and input-output interfaces. This preserves modularity and often yields human-interpretable intermediate outputs. A human inspector can examine the generated bounding boxes, text plan, and image patch individually. However, the cost is information loss when dense perceptual features are compressed into discrete textual or symbolic forms, as discussed in Section 4.7.

Whether an MLLM-equipped system qualifies as an HMMA depends on the functional roles of its components. We adopt a role-based criterion to decide how MLLMs are classified in this survey. A system is included when an MLLM serves as a specialized module coordinated by a separate central LLM, such as a vision-language model that produces structured scene descriptions for a downstream planner. A system is excluded when multiple MLLMs act as functionally homogeneous agents that communicate only through natural language, since this collapses the architecture into a multi-LLM debate without heterogeneous components. A system in which a single MLLM invokes a fixed set of vision modules counts as an HMMA when at least one of those modules is itself a learned probabilistic model rather than a deterministic tool. This criterion follows the heterogeneity property in Section 2.1 and keeps the survey focused on systems whose components differ in architecture, training objective, or modality.

2.2.0.4. Model Routing.

Model routing systems [98,120] select among a pool of alternative LLMs or specialist models for each query, typically to optimize the cost-quality trade-off. The routing module asks which model should handle a query and forwards the query to that single model. The key distinction is that routing selects one model from a set of substitutes, whereas an HMMA composes multiple complementary models to accomplish a task that none of them could handle individually. While conceptually distinct, the two mechanisms can be combined. For instance, a system might route an input to the optimal detector from a pool of candidates, and then feed the resulting bounding boxes into an LLM planner. We treat such hybrid systems as HMMAs that utilize routing internally.

2.3. Survey Protocol

To ensure a systematic review, we collected 82,416 main-conference papers published between 2023 and 2026 across major AI venues. The 2026 data covers ICLR 2026 only, as other venues were not yet published at the time of collection.

We implemented a two-stage screening pipeline. First, we extracted the full text from all PDFs and employed an LLM (Kimi-k2.5-thinking) to evaluate each paper with our HMMA criteria in Section 2.1 and Section 2.2. This automated screening identified 591 candidate papers.

Subsequently, the authors manually reviewed all candidates to verify the LLM’s decisions. Any disagreements regarding a paper’s inclusion were resolved through rigorous discussion until a consensus was reached. This validation filtered out 19 candidates, establishing our final literature base of 572 HMMA papers. The detailed venue and year distribution of these selected papers is illustrated in Figure 2.

3. Taxonomy of Heterogeneous Multi-Model Agents

3.1. Overview

We organize the 572 HMMA papers identified in Section 2.3 by interaction pattern, defined by the combination of functional roles (perception, generation, action) that non-LLM models play alongside the LLM. Figure 3 illustrates the yearly trends and application co-occurrences of the five interaction patterns.

The five patterns differ in how much the system changes the world. Patterns with only perception read the environment and write back text. Patterns with generation also produce new content such as images or audio. Patterns with action go further and emit commands that move a robot, steer a vehicle, or manipulate an object. The more roles a pattern includes, the more complex the engineering becomes. Each new role adds a new boundary that the LLM must cross and a new kind of error that can enter the chain. This functional spectrum is the main purpose of the taxonomy. It should not be read as a domain taxonomy. The same domain may contain several patterns, and the same pattern may appear in many domains.

3.2. LLM with Perception

This pattern represents the largest category. In this pattern, perception models extract structured information from raw inputs, and the LLM reasons over these structured representations to produce textual results such as answers, analyses, or descriptions. The system reads images or other sensory data, but it never writes back a new image or a new action. This makes the pattern a natural fit for question answering and content analysis, where a textual answer is the only thing the user wants.

A prominent paradigm within this pattern is visual programming, where the LLM writes short programs that call a library of vision models to answer compositional visual questions. ViperGPT [149] pioneered this idea by having a code LLM generate Python programs that integrate object detectors, captioners, and question-answering models on demand. Each function in the generated program is a thin wrapper around a specialist, and the program itself acts as the explicit plan. Chameleon [111] generalizes this approach into a plug-and-play framework where GPT-4 picks from a broader set of modules. In Chameleon, the LLM does not write a long program. It writes a short list of module calls that are easy to inspect. Beyond visual programming, the pattern also covers modular pipelines where perception models feed structured information into the LLM for downstream reasoning. The important common point is that the specialist side extracts evidence, while the LLM side composes that evidence into an answer.

Recent systems make this pattern more interactive. Transductive Visual Programming [169] uses GroundingDINO [107] and UniDepth [128] to support spatial reasoning programs, and then admits only verified programs into a growing tool library. InSight-o3 [91] follows a similar idea for high-density images. A reasoning agent describes a desired region in natural language to a visual search agent, receives the cropped patch as evidence, and continues multi-hop reasoning over it. This example shows that the perception-only pattern is not limited to one-pass feature extraction. It can also be an iterative search process in which the LLM decides what evidence it still needs. In all of these cases, the output remains textual, which is what separates this pattern from generation and action patterns.

3.3. LLM with Perception and Action

This pattern is the second-largest category. Perception models extract structured representations from sensory inputs, the LLM reasons over them and formulates plans, and action models execute physical or digital commands. The three-stage perceive-plan-act loop fits naturally with tasks that require closed-loop interaction with an environment, where a visual recognition model on the input side and a learned controller on the output side bracket an LLM that decomposes goals, selects tools, and issues structured commands. Compared with the pure perception pattern, the system now affects the external state. A wrong action moves the robot in the wrong direction or clicks the wrong button, and the next step has to be planned from a new state. This action-state feedback loop makes the pattern both powerful and challenging.

Several representative systems illustrate the core logic of this pattern. SG-Nav [187] builds a structured map of the environment from visual observations and asks GPT-4 to reason over it to decide where to go next, leaving low-level movement to a fixed controller. TANGO [210] has GPT-4o write short programs that chain together off-the-shelf perception and action modules, so that the same agent can handle many embodied tasks without retraining. Agent S2 [4] separates long-range planning from short-range execution while relying on specialized grounding models for screen elements. R2C [14] gives a compact example of the same pattern. It maps an RGB-D room into a chessboard state using instance segmentation, lets the LLM choose grid positions, and then converts those positions into executable commands. Across these systems, the shared feature is not the environment. It is the closed-loop path from perception to LLM planning to action. The LLM must keep its plan in sync with a state that changes because of its own previous commands.

3.4. LLM with Perception and Generation

This category follows a perceive-plan-generate pipeline. Perception models analyze existing content, the LLM plans the creative process, and generation models synthesize new artifacts. The output is no longer a textual answer but a new image, a new audio track, a new 3D scene, or a new video. The perception side gives the system a sense of where it currently stands. The generation side is where the system spends most of its compute and where most of its perceived quality comes from. In a typical workflow, a perception model interprets the current state, an LLM refines or rewrites the instructions, and a generative model produces the output.

Self-correcting LLM-controlled Diffusion [170,171] exemplifies the closed-loop variant. The LLM extracts key objects from the prompt, the image generator renders an initial result, and an open-vocabulary detector helps the LLM verify object presence and spatial placement, triggering another editing round when mismatches are detected. GenArtist [161] takes a more general view, letting GPT-4V break a complex creation or editing request into sub-problems, pick suitable modules from a library of generators, and verify each step before moving on. The generated artifact thus accumulates correctness step by step rather than appearing in one shot.

The same template extends naturally to 3D scene synthesis [206], video generation [68], and other synthesis settings. PhyT2V [179] applies this loop to physics-grounded video generation. The LLM extracts physical rules from the prompt, a text-to-video model generates a clip, and a captioning model describes the result so that the LLM can detect mismatches and revise the prompt. SceneTAP [26] uses segmentation maps to decide where adversarial text should be inserted, and then sends a detailed prompt to a diffusion model for image editing. LogiStory [118] extends the idea to story visualization by checking generated panels against local and global causal constraints. These examples differ in modality, but they share the same pattern. A perception module inspects either the input or the generated output, the LLM turns that inspection into a plan or revision, and a generator produces the next artifact.

3.5. LLM with Perception, Generation, and Action

This is the most comprehensive pattern, involving all three types of specialized models, but it remains relatively uncommon. The full perceive-generate-act loop typically appears in tasks that require understanding the current state, synthesizing intermediate artifacts such as plans or visual predictions, and then executing physical actions. The system not only thinks and acts, it also imagines. The imagined artifact gives the LLM something to evaluate before the action is committed. Robotic settings illustrate the pattern most clearly.

Video Language Planning [47] imagines what future frames would look like for different action plans and lets the LLM pick the most promising one before the robot moves. The video model serves as a visual simulator that lets the planner see the future without paying the cost of running the real robot. RoboGen [160] has an LLM to invent new training tasks, build the corresponding simulation scenes, and then learn the required skills inside the simulator through RL or motion planning. The LLM writes the curriculum, the simulator builds the world, and the policy learns the skill that the LLM later calls. PERIA [123] offers a tighter manipulation example, where a multimodal LLM reasons over language and vision, a latent diffusion model imagines subgoal images, and a low-level policy executes the resulting plan. RoboTwin [121] shows the benchmark side of the same pattern. It uses visual perception to describe real objects, generative models to build digital twins and textures, and trajectory planners or policies to create expert trajectories. Execution failures are fed back to the LLM, which revises code and constraints until the simulated rollout succeeds.

3.6. LLM with Generation

This lightweight pattern skips the perception stage entirely. The LLM receives textual instructions directly and orchestrates generation models to produce content. There is no detector, no segmenter, and no scene parser between the user and the generator. The LLM does the entire understanding step inside its own forward pass. It is most commonly instantiated in text-to-image workflows where the LLM serves as a prompt writer, layout planner, or iterative reviser for diffusion models. Idea2Img [183] is a typical case. GPT-4V writes a prompt, inspects the generated image, and gradually improves the prompt until the output matches the user’s idea. Although Idea2Img inspects images, that inspection is done by the same multimodal LLM rather than by a separate perception specialist, which is what keeps the system inside this pattern in our scheme. Multi-agent variants such as GenPilot [186], BannerAgency [157], T2I-Copilot [28], and Twin Co-Adaptive Dialogue [158] extend the same template to test-time prompt optimization, structured graphic design, and dialogue-based image generation. This paradigm also applies to video [97] and audio generation [173], where LLMs plan scripts before handing the synthesis off to specialized generators.

4. Architectural Design and Analysis

This section provides a cross-cutting architectural analysis. Rather than asking which specialized models are used, we focus on how these models are connected, what information they exchange, and how tightly the modules are integrated. Two systems with the same set of specialists can still differ significantly. One may run them in a fixed order while the other lets the LLM choose at run time. One may chain them through plain text while the other glues them with a learned adapter. These structural choices often matter more for end-to-end behavior than the choice of any individual specialist.

Table 1 reports the distribution of 572 surveyed systems along the five architectural dimensions covered in this section. Feedback structure and uncertainty handling are analyzed below as standalone design dimensions. We also use reported limitations and error cases in the literature as a lens on architecture. The goal is not to rank systems by how often they fail. It is to understand where a composed system becomes brittle. The same pattern appears across domains. A model misses a small object, a planner turns a vague goal into the wrong subtask, or a generator follows the words of a prompt while missing its intent. These failures are often ordinary. They matter because each one becomes another module’s input.

4.1. LLM Roles in HMMA Orchestration

The LLM in an HMMA does not act as a single black-box reasoner. Its functional roles can be described using common patterns in the surveyed papers, as summarized in Table 2. Almost every paper exhibits the planning, coordinating, and reasoning roles [59,112,140,150], since these together correspond to the basic LLM-centered coordination process. The reflector and verifier roles [24,39,73,161], which monitor and second-guess intermediate outputs, appear in roughly half of the reviewed papers and are concentrated in iterative and feedback-driven systems. The translator role [24,61,146,193] captures cases where the LLM converts between modality-specific representations, for example by rendering scene graphs or detection outputs back into natural language. The router role [24,112,126,140] applies when the LLM selects among interchangeable specialized models for the same subtask. A single LLM can play several roles inside the same system, often switching between them as the task progresses. For example, the LLM may begin as a planner, then act as a verifier after the first specialist call, then act as a translator when handing structured detections to a generator that prefers free-form text.

4.2. Frequently Used Specialized Models

Table 3 lists the most common specialized models in the surveyed systems, separated by functional role. The perception side is dominated by general-purpose vision encoders such as CLIP [133] and a small number of open-vocabulary detectors [107] and segmentation models [83]. Generation is concentrated around the Stable Diffusion family [138] and its conditioning extensions [197], and to a lesser extent DALL-E [135,136] and Flux [84]. The action side is markedly more fragmented, as learned manipulation policies such as Diffusion Policy [16], CLIPort [142], and PPO-based controllers [44,105,127] are typically confined to a few specific works. This long tail on the action side is a useful signal. It suggests that grounded action remains the least standardized component of HMMAs, and that progress in this area may depend more on shared interfaces than on a single dominant model. The contrast between the perception and action sides also explains why many embodied HMMAs feel more brittle than their visual reasoning counterparts. A perception failure can be patched by swapping in a stronger detector that everybody else also uses, while an action failure usually requires retraining a custom policy.

4.3. Information Flow Topology

Information flow topology specifies the computational structure through which the LLM and specialized models exchange intermediate states. Topology specifies which modules may exchange information and whose outputs trigger whose inputs. As shown in Table 1, the field has shifted from one-pass pipelines toward architectures that either let the LLM dynamically select models or repeatedly revise decisions using environmental feedback.

Iterative Closed-loop Topology.

Iterative topologies, the most common pattern among the surveyed systems, repeatedly cycle through perception, reasoning, and action or generation. The system perceives, the LLM thinks, the system acts, and the cycle starts again with a new observation. In robotics, SayCan [6] combines language-model scores with affordance functions to select feasible skills, while Inner Monologue [73] feeds scene descriptions and execution feedback back into the LLM planner. Both systems treat the world as a partner that has to be heard between every move. In content generation, LLM-directed diffusion workflows alternate between prompt planning, image synthesis, visual evaluation, and prompt refinement. Recent systems instantiate several concrete forms of this loop. R2C [14] uses instance segmentation to update a chessboard state after each robot action. PhyT2V [179] captions the generated video and sends the caption back to the LLM for prompt repair. RoboTwin [121] feeds planning or simulation errors back to the LLM so that it can regenerate code or constraints. These examples share the same structure even though they live in different domains. A specialist changes or reads the world, the LLM checks the new state, and the next call is conditioned on that check. The dominance of this topology indicates that feedback has become a core design principle rather than a special feature of a few embodied agents.

Star-shaped Orchestration.

In star or hub-and-spoke topologies, the LLM serves as a central controller that selects specialized models on demand. The other models do not talk to each other. They each reply only to the LLM. Chameleon [112] uses an LLM planner to choose among modules for image captioning, text detection, and knowledge retrieval. HuggingGPT [140], Gorilla [126], ControlLLM [109], and Olympus [98] similarly decompose user requests into subtasks, select suitable external models, execute them, and summarize outputs. Star-shaped architectures are more flexible than sequential pipelines, but they place a heavier burden on the LLM’s model-selection and state-management capabilities. The LLM has to remember which calls it has already made, what each call returned, and what to do when an unexpected output arrives.

Sequential Pipelines.

Sequential pipelines execute modules in a fixed order. The order is decided at design time, not at run time, so the same sequence runs for every input. Visual programming systems such as VisProg [59] and Recursive Visual Programming [56] are representative. The LLM generates executable programs that call vision modules such as object detectors, depth estimators, and captioners, and then aggregates their outputs into an answer. Such pipelines are transparent and easy to debug, but they suffer from error propagation. Once an upstream detector or parser fails, downstream reasoning is conditioned on a corrupted state, and there is no later stage that can repair the damage. Reported error cases make this limitation concrete. In document, GUI, and 3D reasoning systems, an early OCR, grounding, or localization error often becomes a confident text statement. The LLM then reasons over that statement as if it were true. A later module may still look competent, but it is solving the wrong problem. This is one reason why iterative topologies have grown faster among the papers reviewed in this survey.

DAG and Convergence Topologies.

Directed acyclic graph (DAG) and convergence topologies extend the linear pipeline and the iterative loop with parallel branches that gather evidence from multiple sources before the LLM synthesizes a final decision. They are less frequent than the three topologies discussed above, but they appear in domains that need to combine heterogeneous evidence streams. Medical HMMAs offer typical examples: MMedAgent-RL [172] routes a case from a triage agent to a panel of specialist Med-LVLMs whose judgments are then integrated by an attending agent, MedAgentSim [9] has multiple doctor agents independently assess each case and aggregates their diagnoses through majority-vote ensembling, and medical visual agents such as VILA-M3 [122] add domain-specific knowledge to the same convergence pattern. DAG topologies make the resulting dependency structure explicit, while convergence topologies merge multiple specialized model outputs into a single intermediate representation that the LLM then consumes. Both structures are more expressive than linear pipelines, but they require explicit dependency management and stronger mechanisms for reconciling inconsistent intermediate results, which is one reason why current HMMAs still favor simpler topologies. A practical workaround is to use a DAG only for the stages where parallel evidence is most useful and to fall back to a sequential structure for the rest of the pipeline.

4.4. Model Coupling

Model coupling describes how tightly the LLM and specialized models are bound together. Coupling can be understood as a spectrum that runs from no shared training to full joint training, with several useful midpoints in between. Table 1 and Table 4 confirm that HMMA research is still primarily driven by modular assembly rather than end-to-end co-training, with loose coupling accounting for the majority of systems.

Loose Coupling.

Loosely coupled systems compose pretrained models without any joint training. A new specialist can be plugged in by writing a thin wrapper, and an outdated one can be removed by deleting that wrapper. Socratic Models [193] compose pretrained foundation models through language alone. VisProg [59], Chameleon [112], and LLaVA-Plus [104] similarly rely on prompting, code generation, API calls, or model descriptions to coordinate external modules. This approach is cost-effective, modular, and easy to extend. Its weakness is shallow alignment. The LLM understands specialized models solely via text descriptions or few-shot examples, so it may invoke the wrong module when a module’s behavior drifts from its description.

Medium Coupling.

Medium-coupled systems train lightweight connectors while keeping the major backbones frozen. The backbones are still off-the-shelf, but a small bridge layer is trained to translate between them. PathAsst [146] trains a projection layer that maps PathCLIP visual features into the Vicuna embedding space so that pathology images can be consumed by the diagnostic LLM. Muses [45] fine-tunes CLIP to align 3D object orientations with the layout planner’s natural-language specifications. These designs preserve more information than symbolic APIs while avoiding the cost of full joint training, and they are especially useful when the interface must carry dense visual or spatial information.

Tight Coupling.

A small number of systems, notably RT-2 [18], adopt end-to-end training for both the LLM and specialized modules, while recent robot agents [94,114] explore intermediate designs that ground video or language models into action policies. Here the boundary between the language model and the specialist is partly erased, since both share gradients during training. While this paradigm improves task performance, it incurs significant computational and maintenance overheads. Replacing one component often forces a partial retraining of the others. Consistent with these costs, tight integration accounts for only a tiny fraction of the surveyed systems. When it does appear, it is concentrated in robotics and autonomous driving, where the gap between high-level language and low-level control is largest and the benefit of joint optimization is most direct. A pragmatic reading is that tight integration is reserved for the cases where loose composition simply does not work well enough.

Coupling by interaction pattern.

Table 4 breaks down model coupling by interaction pattern. Loose composition dominates every pattern, but its share varies. The LLM with perception pattern is the most loosely coupled, because most visual reasoning systems treat perception modules as zero-shot APIs. Patterns that include a generation module use medium coupling more often, because generators frequently rely on adapters or projector layers (Q-Former [90], ControlNet [197]) to receive structured guidance from the LLM. Tight integration remains marginal across all patterns, indicating that joint training across LLMs and specialized modules is still the exception rather than the rule. The relative absence of tight integration in action-heavy patterns is worthy of note, since these are the patterns where the gap between language and low-level control is largest. The likely explanation is that joint training in such settings is expensive and that most groups can only afford to train one stage at a time.

4.5. Feedback Structure

Feedback structures govern how LLMs iteratively refine decisions using signals from specialized models, environments, or other agents. A feedback signal is whatever the LLM can read after a call that tells it whether the call worked. The prevalence of these mechanisms across surveyed systems shows that closed-loop control has become the dominant paradigm in HMMA design. Feedback is useful because many reported failures are not exotic model failures. They are small mismatches between a subgoal and the next model call. A retry loop can catch some of them if the signal is readable by the LLM.

Iterative Refinement.

The most common pattern, where the LLM observes the output of a specialized model, checks it against the task, and re-issues a revised plan or prompt. This is the standard observe-evaluate-retry loop. LLM-directed embodied planners such as Inner Monologue [73] fall in this category. Most LLM-driven image editors also belong here, since the LLM looks at the latest render before deciding the next prompt. GenArtist [161] makes this loop explicit through a planning tree and step-by-step verification. If one tool call fails, the agent can backtrack to a sibling tool or launch a correction sub-tree. LogiStory [118] uses local and global causal verifiers to decide whether a generated panel should be accepted, regenerated, or edited. Transductive Visual Programming [169] uses a VLM judge to decide which generated programs are reliable enough to enter the example library. These designs show that feedback can improve not only the current output but also the future tool library of the agent.

No Explicit Feedback.

A second and still substantial group of systems [56,59,140] execute modules in a single forward pass, with no mechanism for the LLM to revise its decisions based on downstream outputs. These systems trade robustness for simplicity. They are easy to deploy and easy to reason about, but they are vulnerable to errors from upstream specialist.

Other Feedback Patterns.

Cross-model feedback routes signals between specialized modules without going through the LLM, for example, when a verifier model directly filters or reranks outputs of a generator [117]. Self-correction arises when the LLM critiques its own outputs through reflection or self-consistency [3], which can catch logic errors that no specialist would notice. Human-in-the-loop feedback remains a specialized but useful mechanism, mostly for medical or scientific HMMAs where a domain expert validates intermediate results [141]. The expert may approve, reject, or rewrite a single step, and the LLM then continues from the corrected state. Human-in-the-loop feedback is especially valuable in safety-critical settings such as medicine, robotics, and autonomous driving. KnowNo [137] shows that uncertainty-aware language grounding can trigger clarification rather than forcing the robot to act under ambiguity. This style of feedback is useful when wrong actions are costly to undo, since it lets the system pause and ask rather than commit.

4.6. Uncertainty Handling

Uncertainty handling is the least mature dimension. Specialized models in an HMMA are probabilistic and produce errors that can compound across the pipeline, yet most surveyed systems include no mechanism to detect or contain such errors. Object detectors return uncertain boxes, segmentation models may over- or under-segment objects, diffusion models can produce artifacts, and action policies may fail under distribution shift. The LLM that sits at the center is asked to do an enormous amount of implicit error correction, and it does not always succeed.

Among the systems that do handle uncertainty, the most common strategies are LLM or VLM verification, confidence thresholding, retry or fallback, cascaded filtering, and voting or ensemble methods. The fact that these strategies are mostly local rather than global is one of the main weaknesses of current HMMA design. Reported error cases make this pattern clear. Current systems often know that a step failed only after visible damage appears. By then the system has lost the chance to make a cheap correction at the source.

No Uncertainty Handling.

The prevalence of systems in this category highlights an open problem. HMMA research has relied on the LLM’s reasoning to absorb downstream errors rather than treating uncertainty as a first-class architectural concern. A wrong detection or a noisy plan is silently passed forward, and the user only sees the final mistake.

LLM-based Verification.

The predominant strategy for addressing uncertainty employs a central LLM or an auxiliary VLM to evaluate the plausibility of intermediate outputs [73,117,169]. This works well when the LLM can spot the error from text alone, for example when the output mentions an object that the prompt did not ask for. It is less effective when the error is fine-grained, such as a slightly off bounding box that still names the right object.

Confidence Thresholding.

Confidence thresholds use scores produced by specialized models themselves, gating downstream computation when scores are low. LogiStory [118], for example, uses causal plausibility thresholds to choose among accepting, regenerating, or editing a story panel. A common lightweight strategy is to use confidence scores from perception models. Open-vocabulary detectors such as GroundingDINO [107] return scored detections that can be filtered before they are passed to the LLM. This is simple and computationally cheap, but confidence scores are not always calibrated across domains or model families. A score of 0.7 from one detector may not mean the same thing as a score of 0.7 from another, which makes thresholds hard to set in a principled way.

Retry and Fallback.

Retry or fallback strategies invoke an alternative module or prompt when the primary one fails. RoboTwin [121] applies this idea when poor 3D assets or failed trajectories trigger regeneration or code revision. Fallback strategies also include switching to a stronger specialist, asking for human clarification, or abstaining. KnowNo [137] is a representative example of uncertainty-aware clarification in embodied agents. Such mechanisms remain underused relative to their importance. A practical reason is that designing a clean abstain step requires the system to know when it does not know, which is exactly the capability that current HMMAs lack.

Other Uncertainty Mechanisms.

Cascaded filtering stacks multiple lightweight checks before a heavier downstream module is triggered [164]. Voting and ensemble approaches aggregate decisions across several models or sampled runs [77,204]. Verifier-based correction uses another model to inspect outputs and trigger correction, as in Woodpecker’s hallucination correction loop [189]. Each of these methods is local, in the sense that it inspects one stage at a time. None of them tracks how earlier errors keep affecting later stages, which we discuss in Section 7. Across these systems, the weak point is rarely one module in isolation. It is the handoff. A detector returns boxes without calibrated context. A planner turns them into text. A generator or policy then treats the text as complete. The missing piece is a shared state that keeps the original evidence, the compressed message, and the confidence of the conversion together.

Table 5 consolidates the five architectural dimensions discussed in this section, listing representative designs, representative papers, and the main tension associated with each dimension.

4.7. Inter-Model Interface Design

The inter-model interface defines how heterogeneous modules expose their internal results to the LLM and to each other. This is a central distinction between HMMAs and multi-LLM systems. Communication is not limited to natural language, but also includes boxes, masks, scene graphs, executable code, embeddings, and control signals. Each format brings a different trade-off between expressiveness, interpretability, and replaceability.

Discrete Symbolic Interfaces.

Most HMMAs adopt discrete symbolic interfaces. Perception modules return object labels, bounding boxes, segmentation masks, OCR text, or structured scene descriptions. GroundingDINO [107] outputs open-set grounded boxes, and SAM [83] returns segmentation masks that can be consumed by downstream reasoning or planning modules. De-fine [54] and Chameleon [112] further convert model use into executable code or structured module calls. The main advantage is modularity. Components can often be swapped without retraining the whole system, and a human can read the intermediate state at any time. The cost is information loss, because dense perceptual signals are compressed into sparse symbolic summaries. A scene with one hundred relevant pixels per object becomes a single label and four box coordinates. This loss shows up in concrete errors. GUI agents may return only a pair of coordinates when the planner also needs the reason for the click. Robot agents may turn a rich map into a short object list. Generation agents may pass a caption when layout, timing, or style is the actual control signal. These interface choices are not just format decisions. They decide which mistakes the next model can still notice.

Continuous Representation Interfaces.

A much smaller group of HMMAs uses continuous representations to reduce this information bottleneck. LMDrive [139] compresses multi-view camera and LiDAR features through a Q-Former into visual tokens that are consumed directly by the driving LLM. ModaVerse [159] projects ImageBind embeddings of images, video, or audio into the LLM token space via trainable linear layers. These interfaces preserve richer modality-specific information, but they reduce plug-and-play flexibility and make intermediate reasoning less interpretable. As Table 1 shows, purely continuous interfaces remain rare. Instead, more than a third of surveyed systems [72,154,207] combine symbolic and continuous channels in a single architecture, for example by routing dense features through an adapter while still passing high-level decisions as text or structured module calls. This mixed style appears to be the optimal balance for many designers, since it gives them the readability of text where humans need to inspect and the bandwidth of embeddings where the model needs to be precise.

5. Application Domains

HMMA is now deployed across a wide range of application domains, with each domain favoring a particular combination of interaction patterns and model choices. Section 3 classifies systems by functional roles. This section uses the opposite lens. It asks what each application environment needs from those roles. The choice of pattern is rarely arbitrary. A domain that needs a textual answer pulls toward the perception-only pattern. A domain that needs to act on a screen or in a room pulls toward the perception-and-action pattern. A domain that needs new artifacts pulls toward patterns that include a generator. Table 6 summarizes the application domains among the surveyed systems. We highlight four mature areas below, then briefly cover the long tail.

5.1. Visual Reasoning and Understanding

Visual question answering and reasoning is the single largest application of HMMA, dominated by the LLM with perception pattern. The domain pressure is evidence selection. A visual reasoning system must decide which parts of an image, chart, page, or video frame matter before the LLM can reason over them. A typical system pairs CLIP-style encoders [133], grounded detectors [107], segmentation models [83], and captioners [90] on the perception side with an LLM that performs compositional reasoning over the extracted scene information. The perception side answers low-level questions such as “which objects are in the image” and “where are they”. The LLM then chains those answers into a longer line of reasoning that maps to the user’s question. Multimodal agents such as LLaVA-Plus [104], TextBind [89], MEXA [191], Omni-R1 [204], and Affective-CoT [74] extend the same setting to interactive multimodal dialogue, where the user can keep asking follow-up questions and the assistant keeps requesting evidence as needed. The main domain risk is that a missing box, wrong referent, or OCR error may be turned into a confident answer. This risk is why active evidence search has become important in visual reasoning systems.

5.2. Embodied and GUI Agents

Embodied navigation, robot manipulation, and GUI or web agents collectively form the second largest cluster. They share the LLM with perception and action pattern but differ in how the environment is grounded and how costly a wrong action is. Navigation agents must maintain a memory of places and goals while moving through partial observations [49,181,184].

Manipulation agents must turn language goals into short skills that a learned policy can execute [31,94,130]. GUI agents replace rooms and objects with screens, DOM nodes, OCR text, and clickable regions [52,86,100,152]. The shared issue is long-horizon state tracking. The agent must remember what has already changed, decide whether the current observation matches the plan, and avoid acting on outdated state. This makes recovery more central here than in static visual reasoning. A wrong visual answer can often be corrected later. A wrong click, grasp, or navigation step may change the next state and make the original plan invalid.

Thus, embodied HMMAs often introduce a compact state between perception and control. This state may take the form of a topological map [33], a grid [14], a symbolic object list or scene graph [187], a subgoal image [47], or a simulated scene [160]. The details vary by environment, but the pressure is the same. Embodied systems need a state that is abstract enough for reasoning and precise enough for execution. Bridging these two needs is one of the main design problems in embodied HMMAs.

5.3. Multimodal Content Generation

Multimodal content generation spans image, 3D, video, and audio output. It is most often realized through the LLM with perception and generation pattern; when the input is purely textual, it uses the LLM with generation pattern instead. The domain pressure is controllability. Users usually care about layout, style, identity, timing, and safety, not only whether an output looks plausible. The LLM acts as a creative director that translates abstract user requests into precise specifications, and then hands synthesis off to modality-specific generators such as Stable Diffusion [138], ControlNet [197], and text-to-speech or video diffusion backbones. Perception models are often reused as evaluators, comparing the generated artifact against the prompt or layout. WordArt Designer [63] and MetaDesigner [62] illustrate the pattern for artistic typography. Beyond static images, the same domain pressure appears in image restoration [30,211], adapter selection for diffusion backbones [113], programmatic editing [21,60,67,76], and 3D-controllable generation [45]. A growing subset focuses on trustworthy generation [7,25,38,75,124,178], where the LLM also acts as a guardian that screens the generator’s output for unsafe, biased, or low-quality content before it reaches the user.

The surveyed systems also show that generation HMMAs are becoming more structured, with explicit layout, layer, and design-control representations appearing in recent image, diagram, SVG, and slide-generation agents [23,88,99,132,165,168,200,209]. Recent video and image-editing systems [97,147,179] add similar structure through physical constraints, segmentation masks, or causal consistency checks. These systems suggest that the strongest generation HMMAs do not only write better prompts. They expose intermediate controls that the LLM can inspect and revise. That is what separates domain-level controllability from a generic prompt-improvement loop.

5.4. Scientific and Medical Assistants

Scientific and medical applications are smaller in volume but structurally diverse, and they are often where the full LLM with perception, generation, and action pattern shows up. Medical systems typically pair image analysis models for X-rays [48,65], pathology slides [146], or EHR signals [12] with an LLM clinical reasoner that coordinates a set of medical tools [87,122]. The LLM integrates findings from imaging, lab tests, and patient history, much as a clinician would combine evidence during a consultation. When therapy planning is also involved, the LLM additionally coordinates with treatment or design modules [11]. Scientific discovery systems follow a similar template in domains such as catalyst and molecule design [145], scientific visual discovery [116], schematic generation [162], and causal hypothesis generation [1]. These domains place strong emphasis on verifiability and human-in-the-loop feedback, since downstream decisions carry real-world consequences. A wrong dosage or a wrong reaction prediction may harm a patient or waste a costly experiment, so the systems are usually designed to flag uncertain steps for expert review rather than to act on them automatically.

5.5. Additional Application Environments

Table 6 also includes smaller application environments covered by the survey. These tail domains are individually smaller but together account for a non-trivial portion of the reviewed papers.

Autonomous driving.

Autonomous driving HMMAs combine perception modules for lane detection, object tracking, and traffic-sign recognition with an LLM that proposes high-level driving decisions, while a learned controller turns those decisions into actuator commands [50,61]. The pattern is similar to embodied navigation, but the safety bar is much higher and the time budget per decision is much tighter.

Video and audio generation.

Video and audio generation systems usually separate the planning step from the synthesis step [64]. The LLM writes a beat-by-beat script or a shot list, and a video diffusion model or a text-to-speech model fills in the audio-visual content. Perception models often serve as evaluators that decide whether the generated clip matches the script [71].

Document understanding and retrieval-augmented systems.

Document understanding HMMAs run OCR, table extraction, and layout parsing before the LLM reads the content, which lets them handle long PDFs and forms that would not fit in a single LLM context [115,148]. Retrieval-augmented HMMAs add a multimodal retrieval module that pulls relevant images, tables, charts, or video clips from a database before the LLM answers a question [2,41,175,192,199].

6. Evaluation and Benchmarking

Evaluation is a central problem for HMMA research. A composed system may use a strong LLM and strong specialists, yet still fail because the parts do not work well together. A detector may return the right object but in the wrong format. A generator may follow the prompt but violate a spatial constraint. A controller may execute a plan that was already outdated. For this reason, HMMA evaluation should not only ask whether each component is good. It should also ask whether the model chain stays coherent from the user request to the final answer, generated media, or action.

Current evaluation practice shows a clear tension. Most studies [10,64,70,185] inherit metrics and benchmarks from their application domains, such as VQA accuracy, navigation success rate, GUI task completion, generation preference, clinical correctness, or robot manipulation success. These metrics are necessary because they keep HMMA systems comparable with prior non-agentic and single-model baselines. However, they are often insufficient for explaining why a heterogeneous model chain succeeds or fails [79]. Recent HMMA papers commonly report both component-level and end-to-end results, yet coordination failures, interface losses, recovery behavior, and runtime cost are far less consistently documented [185,208]. This section therefore focuses on evaluation as a diagnostic problem rather than only a leaderboard problem.

6.1. Evaluation Scope and Axes

Current evaluation practice can be organized around five axes. The first axis is component competence. Each specialist should still be tested on its own task, such as detection, segmentation, speech recognition, image synthesis, or robot control. This is visible in systems that evaluate specialist modules before reporting whole-system performance, such as multimodal assistants, medical agents, and robot systems [93,94,104,146].

The second axis is coordination quality. The LLM must choose suitable specialists, order their calls, translate outputs across interfaces, and stop when the goal is met. Tool-learning benchmarks and agent studies already evaluate parts of this behavior [5,126,177,180,208], but HMMA coordination is harder because the tools are learned models with uncertain outputs.

The third axis is end-to-end task success [5,139]. The whole system must answer the question, produce the requested media, or complete the action. The fourth axis is reliability and recovery [137,195,205]. The system should detect bad intermediate states, retry when needed, and abstain when the task is unsafe or underspecified. The fifth axis is cost and deployability. The number of model calls, latency, memory, energy, and human intervention all matter in deployed systems. Recent cost-aware, performance-aware, and deployment-oriented agents make this dimension explicit, but it is still far from universal [24,38,151,167,184].

Taken together, these five axes show that HMMA evaluation has to separate local quality from system quality. A specialist can be locally correct while still causing a downstream error if its output is too coarse, too late, or expressed in the wrong interface. Likewise, an LLM can generate a plausible plan that fails because the selected specialist cannot execute one of the steps. This gap is why direct evaluation of the composed system is needed.

6.2. Benchmark Settings and Metrics

The benchmark landscape follows the application structure of the field. Visual and document reasoning papers often reuse VQA, grounding, document QA, or multimodal retrieval benchmarks. For example, VisDoM introduces a multi-document QA benchmark with visually rich elements, while active VQA, visual programming, and dynamic visual reasoning systems reuse established VQA and spatial reasoning datasets [95,144,148,169,174]. Embodied navigation and manipulation papers rely on simulators and robot benchmark suites, such as Habitat-style object navigation, dual-arm manipulation, open-world robot manipulation, and vision-language-action evaluation [51,94,121,130,166,187]. GUI and web agents use task-completion benchmarks for computer interaction, mobile control, and cross-environment execution [3,4,5,53,177,208]. Generation papers combine automatic metrics with human preference, especially for image, diagram, editing, layout, and 3D-controlled generation [34,36,37,45,157,165,194]. Medical, scientific, and safety-oriented assistants add domain benchmarks and expert-oriented correctness checks [17,48,87,93,116,145,201].

Across these settings, the most common metrics are still inherited from the target task: accuracy for question answering and diagnosis, success rate for navigation or manipulation, task score for agents, quality scores and human preference for generation, and expert judgment for specialized domains. These signals are useful because they connect HMMAs to mature evaluation traditions in each domain. They also make results easier to compare with non-agentic baselines. However, they are often too coarse for diagnosing a heterogeneous model chain. A visual reasoning system may answer incorrectly because the detector missed an object, the LLM ignored a correct detection, or the final reasoning step used the wrong relation. A robot may fail because the language plan was wrong, the subgoal was impossible, or the controller drifted. The same final score can hide very different causes.

Table 7 gives a domain-level view. The table is not meant to replace domain benchmarks. It shows what extra information is needed once the system becomes a heterogeneous model chain. A benchmark should not only report whether the final answer, generated media, or action is correct. It should also record whether the system used the right evidence, selected the right specialist, preserved the needed information across interfaces, and recovered from bad intermediate states. These additions make evaluation more diagnostic without forcing every domain into the same metric.

6.3. Ablation and Counterfactual Evaluation

Ablation is now common in HMMA papers, but it is unevenly distributed across design dimensions. Many studies remove or replace detectors, generators, retrievers, planners, verifiers, memory modules, or agent roles. For example, generation and editing systems ablate layout planning, visual feedback, prompt optimization, or evaluation agents [34,36,37,45,183,186,194]. Embodied systems ablate perception, memory, map construction, planning representations, or hierarchical action models [51,78,94,130,187,205]. GUI, web, and general computer-control agents compare planners, critics, GUI parsers, function-calling interfaces, and different agent organizations [5,53,151,177]. Such studies are valuable because they show which component contributes to final performance.

Feedback and interface ablations are less common but more diagnostic for HMMAs. This is a gap because feedback and interfaces are often where heterogeneous composition succeeds or fails. Some works do make these factors visible. TextBind [89] compares continuous hidden-state communication with discrete text-style interfaces for multimodal instruction following. SG-Nav [187] compares different scene-graph prompting interfaces for zero-shot object navigation. ORION [50] studies alternative interfaces between vision-language reasoning and action generation in autonomous driving. Agent S [3] evaluates the effect of an agent-computer interface for GUI control. Recent benchmarks and systems also compare structured UI representations, function calling, symbolic workflow representations, text-only versus multimodal evidence passing, and alternative VLA planning representations [17,36,51,53,177]. These examples suggest a broader counterfactual protocol: start with a working HMMA, then change one boundary in the chain and measure how the whole system changes. One can replace a detector with a weaker detector, remove a verifier, switch from structured output to text, disable feedback, or force the LLM to use a fixed model order. If the final score drops sharply, that part is a bottleneck. If the score barely changes, the component may be over-engineered or poorly used.

Counterfactual tests also help compare modular and end-to-end designs. A modular HMMA should benefit from replacing one specialist with a better one. If it does not, the interface or orchestrator may be the limiting factor. An end-to-end MLLM may perform better on a closed benchmark, but it is harder to isolate which internal skill caused an error [70]. For HMMAs, the advantage is that the model boundary is visible. Evaluation should use that visibility rather than only report final accuracy.

6.4. Failure Analysis, Human Evaluation, and Deployment

A second gap is the limited use of trace-level failure analysis. Many papers include qualitative examples or failure cases, but most analyses remain final-output oriented. This is useful but not enough for heterogeneous systems. A failed run should ideally be attributed to the first visible error in the chain: perception, retrieval, planning, interface translation, generation, action execution, verification, or recovery. Some papers [24,65,94,130,199,201,205] move in this direction by discussing how visual evidence, memory, action grounding, rechecking, or safety-oriented perception fails in concrete cases. However, few benchmarks require all systems to report the same trace-level error categories. This makes failures hard to compare across papers.

Human evaluation is also uneven. It is common in generation tasks, where human preference remains important, and in high-stakes domains, where expert review is necessary. Interactive image generation and design systems often include user studies or preference tests [34,157,158,183,194]. Medical, scientific, safety, and fact-checking systems include more domain-specific validation, but still rarely evaluate expert handoff, abstention, and calibration as first-class outcomes [17,48,93,145]. This pattern suggests that many HMMAs are still evaluated as offline systems even when their intended use involves interaction.

Deployment evidence remains thinner than benchmark evidence. Most studies are evaluated on offline benchmarks, simulators, or curated task suites rather than sustained real-world use. Robotics, GUI automation, autonomous driving, and general computer-control systems are beginning to report more realistic settings, but even these studies often emphasize success rate more than trace logging, wall-clock latency, model-call cost, or human intervention burden [5,53,151,166,201]. Some systems explicitly target low-cost or efficient operation, such as cost-aware tool planning [167], on-device navigation [184], performance-aware generation [38], fast-slow editing agents [58], phishing detection [24], and multimodal agent tuning [55]. These papers show that cost can be evaluated without replacing task quality metrics. For deployed HMMAs, a slower or more expensive model chain is only useful if the extra calls bring a clear reliability gain.

6.5. A Diagnostic Evaluation Protocol

A practical HMMA evaluation can follow a short protocol on top of existing domain benchmarks. First, for each run, define the expected trace of evidence, model calls, and actions, and log every model interface with its inputs, outputs, and confidence signals. Second, judge why a run failed, not only whether it failed. Evaluate the final output with the domain metric, and attribute a failed run to the first visible error in the chain, using the coarse categories from Section 6.4. For example, replace a detector with a weaker one, remove a verifier, or turn off a feedback channel, and see how the final score moves. Third, report cost together with quality, and report it as a ratio when possible, such as success per model call or per unit of latency. Since the system already passes information through explicit interfaces, those interfaces can become evaluation checkpoints.

7. Challenges and Future Directions

We classify the recurring limitations of current HMMA systems and their corresponding research directions into four themes. The themes are not isolated. A weakness in one theme often shows up as a symptom in another. For example, poor uncertainty handling makes interface bottlenecks more dangerous, since wrong intermediate outputs travel further before they are caught.

Error Cascading and System-level Uncertainty.

HMMAs chain probabilistic modules, so errors accumulate through the pipeline. A small mistake at the perception stage can grow into a large mistake at the action or generation stage, because each module conditions on the output of the one before it. Many systems rely on local confidence scores from a single module. A local score tells the system whether one module is unsure, but it does not tell the system whether the chain as a whole is unsure. System-level approaches [15,137,189,195] that track and correct errors across perception, reasoning, and action remain rare. These recurring failure cases are often easy to describe. For example, a small object is missed, a spatial relation is described too coarsely, a subgoal looks reasonable in text but cannot be executed, and the final answer or action may look fluent even when the hidden state is already wrong. The common thread is that each step can look locally plausible while the chain as a whole is already wrong.

Therefore, a useful direction is to treat the LLM not only as a task planner but also as an uncertainty aggregator, reasoning jointly over the provenance, confidence, and downstream impact of each intermediate prediction. The LLM would then decide whether to trust a step, retry it, or ask the user for clarification. Existing examples suggest the first step toward this goal. TVP [169] verifies programs before adding them to a library. LogiStory [118] keeps causal scores for generated panels. RoboTwin [121] retries failed assets and trajectories. These methods are useful, but they still record uncertainty locally. Future HMMAs need a shared trace that records which specialist produced each intermediate state, what confidence or verifier score it had, and which later decisions depended on it. Such a trace would make it possible to explain a final error by walking back through the chain.

Interface Bottleneck.

Most HMMAs communicate through discrete symbolic interfaces such as text, JSON, or bounding boxes. This keeps the system modular but loses a large amount of perceptual detail. A diffusion model that could in principle condition on dense visual features instead receives a short caption, and a planner that could in principle condition on a continuous map instead receives a list of object names. The error cases show a common pattern. The interface keeps the information that is easy to print, not always the information that is needed to act. Layout, timing, style, uncertainty, and spatial context are often squeezed into short text. Once that happens, the next module has little chance to recover what was dropped. Adapter-bridged designs [8,90,102] preserve more information but weaken plug-and-play flexibility. A key open problem is to design intermediate interfaces that retain modularity, allow components to be swapped without retraining, and still transmit dense visual, spatial, or temporal signals when the task demands it. Standardized schemas for scene graphs, layouts, action spaces, and confidence metadata would make such interfaces easier to reuse across systems, in the spirit of recent efforts on standardizing tool and model APIs [10,85,109,126]. A shared schema would also make it possible to mix and match models from different research groups without writing custom integration code each time.

Modularity vs. Alignment.

The distribution of coupling strategies is dominated by loose training-free composition [56,59,112,193]. This reflects the community’s preference for rapid integration, but it also means that most HMMAs are only shallowly aligned. The LLM interacts with specialized models through prompts and descriptions rather than through joint objectives. When the specialist is updated to a new version, the LLM may keep using outdated assumptions about its behavior. Recent work on passing agent-level reward signals back to perception or generation modules [18] shows a path toward tighter alignment without sacrificing modularity. Exploring when such co-training pays off, and when loose prompting is sufficient, is an important direction. A useful starting point is to study which kinds of tasks see the largest gain from co-training, since the answer probably depends on how often the LLM disagrees with the specialist.

Coexistence with End-to-End Multimodal LLMs.

End-to-end multimodal LLMs continue to improve rapidly [13,20,70,176,188], raising a natural question of whether HMMAs will remain relevant as backbones become more capable. Our analysis suggest that the two paradigms are complementary rather than competing. HMMA research continues to grow at all major venues even as MLLMs scale up, indicating that the modular paradigm fills a different niche. HMMA retains advantages in modularity, interpretability, domain adaptation, and cost control, since individual specialists can be swapped, audited, or quantized without retraining the whole system. End-to-end MLLMs, in turn, provide a strong general-purpose backbone that can itself be plugged into HMMA systems as either the central orchestrator or a specialized perception module. We therefore expect future HMMA architectures to increasingly use MLLMs as components, while continuing to rely on dedicated detectors, generators, and policies wherever fine-grained control or domain-specific accuracy is required. Cross-modal routing and cascading techniques [120] provide a complementary mechanism for choosing between MLLM-internal and HMMA-external paths at inference time. A practical pattern is to start with a strong MLLM as the orchestrator, fall back to dedicated specialists for the few subtasks where the MLLM is weak, and revise the split as the MLLM improves over time.

Benchmark Infrastructure.

Section 6 discussed how HMMAs should be evaluated. The remaining challenge is infrastructure. Most current benchmarks are still built around a single input and a single output. They rarely store the model-call trace, the intermediate artifacts, or the first visible error in a failed run. This makes it hard to compare two HMMAs that reach the same final score through very different paths. It also makes it hard to know whether progress comes from a better LLM, a better specialist, a better interface, or a better recovery policy. Future benchmarks should therefore expose trace logging as a standard part of evaluation. They should support counterfactual swaps of individual modules, controlled corruption of intermediate states, and replay of failed runs under different specialists. In high-stakes settings such as medicine, robotics, and autonomous driving, these benchmarks should also include abstention, expert handoff, and uncertainty-aware clarification [137]. The goal is not to make one universal benchmark for all HMMAs. It is to make domain benchmarks record enough structure that composed systems can be diagnosed rather than only ranked.

HMMA as Model-Oriented Agent Harness.

The study of HMMAs focuses on the model-oriented layer of an agent harness. The harness is the runtime that exposes specialists to a central LLM, schedules calls, and tracks intermediate state. Future systems should move from ad-hoc model calls to reusable harness abstractions for interfaces, uncertainty tracking, and execution monitoring [58,208]. A reusable harness lets a researcher who builds a new specialist plug it into many existing HMMAs at once, rather than rewriting glue code for each one. It also gives a common surface where uncertainty handling and evaluation can be improved without modifying the LLM or the specialists themselves.

8. Conclusions

We introduced Heterogeneous Multi-Model Agents as a distinct paradigm in which a central LLM orchestrates heterogeneous probabilistic specialist models. This paradigm has its own set of design choices and failure modes. Through a systematic survey of recent HMMA work, we proposed a taxonomy of five interaction patterns, analyzed five architectural dimensions, and surveyed applications across different domains. We also discussed evaluation as a system-level problem that should combine final task scores with trace-level diagnosis, recovery behavior, and cost. HMMA research is shifting from static pipelines toward iterative closed-loop designs, but it still relies heavily on loose training-free composition and symbolic interfaces, and it lacks mature system-level mechanisms for uncertainty and evaluation. We hope this survey offers a clear map of the current landscape and a useful starting point for the next wave of HMMA research. We also hope it makes clear that the most promising open problems lie at the system level, where the LLM, the specialists, and the harness must be considered together.