Orchestrating LLMs with Specialized Models: A Survey on Heterogeneous Multi-Model Agents

Large language models (LLMs) have become a common center for building intelligent agents, yet many real tasks still need abilities that a single LLM cannot provide. A growing line of work therefore connects LLMs with specialized non-LLM models, including object detectors, segmentation models, diffusion generators, and robot policies. We call this emerging paradigm Heterogeneous Multi-Model Agents (HMMAs). This survey provides a systematic review of 572 HMMA papers published at major AI venues between 2023 and 2026. We first define the scope of HMMAs and distinguish them from adjacent paradigms. We then organize existing systems into five interaction patterns according to the roles played by perception, generation, and action models. Building on this taxonomy, we analyze architectural choices across information flow, interface design, feedback structure, uncertainty handling, and model coupling, and review major application domains. Finally, we summarize open challenges and outline future directions for building reliable HMMAs.