MedAgent-Pro: Towards Evidence-Based Multi-Modal Medical Diagnosis via Reasoning Agentic Workflow

1. Introduction

Medical diagnosis is a fundamental aspect of the clinical process, requiring doctors to perform step-by-step reasoning based on medical guidelines while integrating various patient information to analyze different indicators and reach accurate and well-informed conclusions [57]. This process is time-consuming, and the growing demand for medical diagnosis is increasingly surpassing the capacity of existing healthcare resources, placing a significant burden on medical services. To alleviate this burden, efforts have been made to develop computer-aided diagnosis (CAD) techniques that assist with specific diagnostic tasks, such as tumor segmentation and cancer grading. However, these models function merely as tools, with the overall diagnostic process still relying heavily on human doctors. As a result, the development of AI systems capable of autonomously handling the entire diagnostic workflow has become a key research focus.

Recently, multi-modal large language models (MLLMs) [1,13,21,33,42] have garnered significant attention for their strong interaction abilities. They can perform a variety of tasks based on user prompts, including Medical Vision Question Answering (MedVQA) [51], Medical Report Generation (MRG), and clinical decision support [9,20]. These advancements underscore the potential of MLLMs in enhancing medical decision-making. However, despite these advancements, MLLMs still fall short of meeting the stringent standards required for disease diagnosis in clinical practice. Firstly, they often exhibit hallucinations and inconsistencies in reasoning, making it difficult to strictly adhere to established medical criteria. Moreover, many disease diagnoses clinically rely on quantitative evidence. For example, the vertical optic cup-to-disc ratio is crucial for glaucoma diagnosis, and left ventricular ejection fraction plays a key role in heart function evaluation. However, MLLMs struggle to perform quantitative analysis on images, thus hard to be applied in clinical practice. As a result, current MLLMs primarily function in one-hop medical visual question answering for diagnostic tasks, which falls short of the expert-driven, evidence-based analysis required in real-world medical diagnosis. These limitations raise significant safety concerns, highlighting the need for further advancements to improve their reliability and clinical applicability.

Figure 1. Comparison of existing MLLMs and our MedAgent-Pro framework on two disease diagnoses. Red text highlights the limitations of MLLMs, while green text represents the evidence-based diagnosis provided by MedAgent-Pro. Our approach enhances diagnostic accuracy while offering comprehensive literature support and visual evidence.

Figure 1. Comparison of existing MLLMs and our MedAgent-Pro framework on two disease diagnoses. Red text highlights the limitations of MLLMs, while green text represents the evidence-based diagnosis provided by MedAgent-Pro. Our approach enhances diagnostic accuracy while offering comprehensive literature support and visual evidence.

Due to the extensive computational resources and high-quality data required for fine-tuning [33], enabling MLLMs with quantitative analysis capabilities and domain-specific expertise remains challenging, limiting their applicability in complex medical scenarios. Agentic systems [12,34,56,62,72] offer a potential solution for assisting MLLMs in medical diagnosis. These systems, originally developed for general-purpose applications, integrate MLLMs with external tools to extend their capabilities beyond simple conversational interactions. Meanwhile, in the medical domain, specialized AI models such as classification [74], grounding [6], and segmentation [45,66,78] have achieved great success while existing medical literature provides essential clinical criteria for disease diagnosis. By leveraging these resources, agentic systems can help compensate for the limitations of MLLMs, enhancing their effectiveness in medical applications. Several studies have explored the development of medical agentic systems [16,28,32]. However, most existing approaches adapt general-purpose agentic frameworks to improve diagnostic accuracy without enhancing interpretability by providing supporting evidence. These frameworks incorporate debate or collaboration mechanisms between MLLMs to refine answers, but they still rely on empirical black-box decision-making and overlook the importance of expert-driven evidence, which is essential in modern medicine. In contrast, medical diagnosis requires structured reasoning based on guidelines, integration of multi-modal patient data, and evidence-based analysis to ensure reliability and safety. As a result, existing medical agentic systems often oversimplify diagnostic tasks, merely focusing on empiricism-based question answering rather than the in-depth, evidence-based analysis needed for real-world applications. These limitations highlight the need for approaches tailored to the stringent requirements of medical decision-making.

To align AI system with evidence-cased modern medical protocol, we propose a reasoning agentic workflow for evidence-based multi-modal Medical diagnosis, termed MedAgent-Pro, to enable comprehensive, evidence-based medical diagnosis. MedAgent-Pro achieves this by incentivizing the reasoning capabilities of MLLMs by integrating the retrieved medical guideline and medical expert tools. It adopts a hierarchical structure, where the task level formulates unified diagnostic plans for specific diseases, while the case level executes the plan and makes diagnostic decisions for suspected patients. At the task level, MLLMs act as planner agents, performing knowledge-based reasoning to generate reliable diagnostic plans by integrating retrieved clinical criteria. At the case level, medical expert models act as tool agents, analyzing both quantitative and qualitative indicators. Finally, a decider agent ensures a comprehensive and accurate diagnosis for each patient, supported by clinical literature and visual evidence. Our key contributions are listed below:

• We propose MedAgent-Pro, a reasoning agentic workflow that can provide accurate and explainable medical diagnoses supported by visual evidence and clinical guidelines.
• At the task level, MLLMs perform knowledge-based reasoning to generate well-founded diagnostic plans. Without fine-tuning, this is accomplished by integrating retrieved clinical criteria, ensuring greater reliability in diagnosis.
• At the case level, various medical expert tools execute the corresponding steps in the plan to process multi-modal patient information, providing interpretable qualitative and quantitative analyses to support evidence-based decision-making.
• We evaluate MedAgent-Pro on both 2D and 3D multi-modal medical diagnosis, where it achieves state-of-the-art performance, surpassing both general MLLMs and task-specific solutions. Additionally, case studies highlight MedAgent-Pro’s superior interpretability and reliability beyond quantitative results.

2. Related work

2.1. AI-Driven Multi-Modal Medical Diagnosis

Medical diagnosis is one of the most critical and complex tasks in clinical practice, demanding both expertise and precision. The development of artificial intelligence (AI) systems capable of assisting human clinicians in diagnostic decision-making has emerged as a key research objective [5,29,47,58]. However, multi-modal diagnosis presents significant challenges, as it requires the integration of both qualitative and quantitative assessments of patients’ visual examinations alongside a comprehensive analysis of diagnostic findings and patient-specific information. To address these challenges, prior research has primarily focused on developing visual models to support medical diagnosis, encompassing tasks such as classification [2,4,14,73,74], detection [7,8,15,22,61], and segmentation [3,25,45,55,64,66,76]. These models extract critical diagnostic indicators, which are subsequently synthesized by medical professionals to formulate clinical decisions.

In parallel, efforts have been made to develop end-to-end AI architectures for medical diagnosis with minimal human intervention, particularly through multi-modal approaches like Vision Question Answering (VQA) [27,71,75]. The emergence of MLLMs [1,13,42,69] has further expanded the potential of AI-driven diagnosis, as these models exhibit advanced reasoning capabilities approaching human-level performance across various domains. Several studies have explored fine-tuning general-purpose MLLMs for medical applications [33,52], achieving competitive performance in Medical VQA tasks. Despite these advancements, most existing Medical VQA datasets [23,31,41] remain overly simplistic compared to real-world diagnostic scenarios. They primarily focus on classification tasks, lacking comprehensive analytical depth and explanatory reasoning, making them insufficient for revealing the interpretability limitations of MLLM-based diagnoses. While some methods improve explainability by linking text outputs to specific image regions [30,39,40,70], they still fall short of structured reasoning necessary for rigorous diagnosis. Consequently, further research is required to advance the application of MLLMs in real-world medical diagnosis, with particular emphasis on their reliability, interpretability, and clinical integration.

2.2. LLM-Based AI Agents

Developing reliable autonomous intelligent systems has long been regarded as a highly promising research avenue. With the advent of the agent concept, which refers to an entity capable of perceiving its environment and taking action, agent-based intelligent systems have garnered considerable attention in recent years. Large language models (LLMs) have shown remarkable potential in reasoning and planning, closely aligning with human expectations for agents that can perceive their surroundings, make informed decisions, and take actions within interactive environments. Driven by this potential, LLM-based agents have made significant advancements in navigating complex environments and tackling intricate tasks across diverse applications such as industrial engineering [37,49,53,67], scientific experimentation [10,19,44], embodied agents [11,24,65], gaming [18,36,54], and societal simulation [26,46,68].

With these advancements, LLM-based agents surpass traditional task-specific models by dynamically adapting to diverse applications without additional training. However, their capabilities remain prone to hallucinations, often leading to inconsistent answers for the same case. Moreover, LLMs excel at qualitative analysis but struggle with generating precise quantitative results. As a result, an agentic system relying solely on LLMs is insufficient for medical diagnosis [35,43,59,79]. Instead, a more structured, tool-based system is needed to ensure accuracy and reliability in clinical applications.

3. Methods

Our MedAgent-Pro framework consists of two main stages: task-level reasoning and case-level diagnosis, as shown in Figure 2. At the task level, the planner agent utilizes retrieved medical pipelines to formulate a diagnostic plan that aligns with established medical principles for each disease. At the case level, specialized models follow the plan’s steps to analyze multi-modal patient data, extracting diverse biomarkers and indicators. Finally, the decider agent integrates all information to generate an explainable diagnosis supported by visual evidence. In the following sections, we introduce the agents and tools involved and provide a detailed explanation of the two-step process.

3.1. Agents Involved in MedAgent-Pro

In this section, we first introduce the various agents in our MedAgent-Pro framework and specify the models that function as each agent.

3.1.1. Agents in Task-Lavel Reasoning

• Retrieve-Augumented Generation (RAG) Agent We utilize RAG to retrieve relevant medical documents, ensuring the development of reliable diagnostic processes that follow clinical criteria. For retrieval, we employ the built-in functionality of LangChain [60], and retrieve from medical library [48,50].
• Planner Agent The planner agent generates a diagnostic plan based on the retrieved guidelines and available tools. We employ GPT-4o [1] as the planner agent due to its strong reasoning capabilities.

3.1.2. Agents in Case-Lavel Diagnosis

• Orchestrator Agent responsible for conducting a preliminary analysis of the patient’s multi-modal information and determining which steps of the diagnostic plan will be executed. We employ GPT-4o [1] since it can recognize different input effectively.
• Tool Agents We utilize various tool agents to complete different tasks in the diagnostic plans, including:

  -

  Classification Models We use image classification models for analyzing some quantitative indicators. We use some general classification models like BioMedclip [74], as well as some task-specific models like RetiZero [63] for Ophthalmic.

  -

  Segmentation Models We use the Medical SAM Adapter [66] as the segmentation model due to its ability to achieve strong performance on the target task with only a small amount of data. To further optimize its effectiveness, we trained task-specific adapters for the target task like optic cup/disc segmentation.

  -

  Vision Question Answering (VQA) Models We utilize VQA models to generate visual descriptions of medical images rather than performing complex reasoning. We employ general VQA models such as LLaVa-Med [33], alongside task-specific models like VisionUnite [38] for ophthalmic applications.

• Coding Agent The coding module is designed to generate simple code for computing additional metrics from the raw outputs of vision models (i.e. segmentation masks). We use GPT-o1 for its strong coding ability.
• Summary Agent: Since LLM outputs are often lengthy, we introduce a summary agent to refine the LLM decider’s response into a simple "yes" or "no" for accuracy evaluation. Additionally, the summary agent condenses the VQA tool’s output into "yes," "no," or "uncertain." We employ GPT-4o [1] for its strong summarization capabilities.
• Decider Agent: In charge of making the final diagnosis based on the indicators obtained from previous steps. The implementation includes two approaches, which will be introduced in the following sections.

3.2. Knowledge-Based Task-Level Reasoning

The goal of this phase is to devise a diagnostic process that aligns with medical guidelines for specific diseases. The planning process is generally divided into two steps. First, we gather information on the clinical diagnosis process for the disease. In previous studies, this step is generally handled by LLMs themselves. However, due to the inherent hallucination phenomenon and inconsistencies in reasoning, the information provided by these models is often highly unpredictable, which contradicts the structured nature of clinical diagnosis.

To address this issue, we introduce a Retrieval-Augmented Generation (RAG) agent to ensure the reliability of the information by retrieving medical resources, including clinical guidelines and hospital protocols. When processing queries about specific diseases, the agent consults medical texts, guidelines, and protocols to summarize relevant information [48,50], rather than relying on unsupported LLM-generated content. This approach enables evidence-based planning, enhancing the reliability and accuracy of the diagnostic process.

Once the diagnostic information is generated, it needs to be structured into an actionable diagnostic workflow. Specifically, the planning agent assigns tasks to different tool agents based on the available tools for the task. The result is a multi-step diagnostic process, where each step follows a triplet format: (object, tool, action). This format specifies which tools and operations should be applied to the input object or intermediate results of each modality at each stage. By structuring the process in this way, the system provides clear and practical diagnostic steps tailored to each specific case.

3.3. Evidence-Based Case-Level Diagnosis

Existing MLLMs approach medical diagnosis through end-to-end VQA. However, this method often lacks effective visual explanations and primarily relies on qualitative analysis without quantitative results to support the diagnosis. Although some methods enhance explainability by linking outputs to specific regions in input images, they merely highlight attended areas without providing underlying reasoning. In contrast, our proposed MedAgent-Pro system leverages advanced vision models as specialized tools, utilizing their fine-grained visual perception capabilities to perform detailed analyses rather than relying on the limited visual processing abilities of MLLMs.

Specifically, when given multi-modal patient inputs, the orchestrator agent automatically identifies the input modality, matches it with the object field in the planned triplet, and executes the corresponding diagnostic steps. Based on these steps, various specialized models are involved as tool agents to conduct detailed analyses. Additionally, the coding agent generates code to process visual data for computing quantitative indicators or producing intermediate results as inputs for subsequent steps. Finally, the summary agent consolidates the analysis of each indicator into "yes," "no," or "uncertain" options, while the decider agent integrates all indicators, formulates the final diagnosis, and provides supporting evidence. In this paper, we explore two approaches for the final diagnosis. The first, LLM decider, involves feeding all obtained indicators into an LLM (GPT-4o in our case), allowing it to autonomously integrate the information and generate a diagnosis. The second approach, MOE (mixture-of-experts) decider, utilizes an LLM to assign weights to different indicators with a threshold-based $\theta$, and compute a final risk score s for the decision-making process:

$$s=\sum _{i=1}^n{w}_i{x}_i,x_i=\left\{\begin{array}{cc}1,\hfill & \mathrm{abnormal}\hfill \\ 0.5,\hfill & \mathrm{uncertain}\hfill \\ 0,\hfill & \mathrm{normal}\hfill \end{array}\right.$$

(1)

where n is the total number of indicators, $w_i$ represents the weight assigned to the i-th indicator, and $x_i$ encodes the indicator’s status. The final decision is determined by:

$$\mathrm{decision}=\left\{\begin{array}{cc}\mathrm{sick},\hfill & s\ge \theta \hfill \\ \mathrm{healthy},\hfill & s<\theta \hfill \end{array}\right.$$

(2)

For example, diagnosing glaucoma requires a comprehensive analysis of a patient’s multi-modal data, including fundus images, OCT scans, and IOP measurements. When provided with only a fundus image, the orchestrator agent queries the diagnostic plan and identifies four indicators: The VQA tools first scan the entire image to detect potential disc hemorrhages, while the segmentation tools identify the optic cup and disc. The coding agent then calculates the vertical cup-to-disc ratio by calculating their vertical diameter, and then extracts the region around the optic disc for further analysis. The VQA module examines this region to identify signs of peripapillary atrophy, while the coding agent computes rim thickness in different directions. Similarly, when provided with 3D echocardiography and basic patient information (height, weight, etc.), the orchestrator agent queries the diagnostic plan and identifies four key indicators. The segmentation tools first detect the myocardium and left ventricle at both start-diastolic and end-diastolic phases. The coding agent then calculates the left ventricular ejection fraction, start-diastolic diameter, and end-diastolic diameter. By incorporating the patient’s height and weight, the left ventricular mass index is further derived. Once all indicators are obtained, the summary agent consolidates the findings, and the decider agent formulates the final diagnosis. A detailed implementation and the results of each agent are provided in the case study within the experimental section.

4. Experiment

We evaluated the performance of MedAgent-Pro in two essential and complex diagnostic tasks: glaucoma and heart disease. Diagnosing both diseases requires a comprehensive evaluation of multiple factors, including both qualitative and quantitative analysis. While current MLLMs struggle in most cases without the aid of external tools.

4.1. Datasets and Evaluation Metrics

For glaucoma diagnosis, we use the REFUGE2 dataset [17], which is a 2D retinal fundus image dataset containing 1200 RGB images at a resolution of $2124\times 2056$ annotated by experts. Each image includes segmentation masks for the optic disc and optic cup, along with a classification label indicating whether the patient has glaucoma.

For heart disease diagnosis, we utilize the MITEA dataset [77], a 3D echocardiography dataset comprising 536 images from 143 human subjects. Each image contains segmentation masks for the left ventricular myocardium and cavity, along with a classification label identifying the patient’s heart condition across seven categories: healthy, aortic regurgitation, dilated cardiomyopathy, amyloidosis, hypertrophic cardiomyopathy, hypertrophy, and transplant. Due to the limited number of samples for each specific heart condition, we simplify the task to a binary classification: determining whether the patient has heart disease.

While the greatest advantage of our MedAgent-Pro system lies in the interpretability of medical diagnosis, we also conducted quantitative experiments to validate the superiority of our approach. For evaluation, we take two common metrics for classification: mACC and F1 score. The mACC represents the average accuracy across all classes; the F1 score provides robustness against class imbalance; Throughout all experiments, the best results are highlighted in bold, while the second-best results are underlined.

4.2. Comparison with Multi-Modal Foundation Models

We conduct comprehensive experiments to demonstrate the superiority of our MedAgent-Pro framework. We have included comparisons with advanced multi-modal LLM methods such as BioMedClip [74], GPT-4o [1], LLaVa-Med [33], and Janus [13]. Notably, since existing MLLMs cannot process 3D images, we randomly select three slices from the 3D echocardiography as visual input and repeat this process ten times to calculate the mean value to minimize randomness.

When comparing with general LLMs, we use mACC and F1 scores as evaluation metrics and evaluate both ways of the decider agent. For comparison with the REFUGE2 challenge method, since only AUC results are available for these methods and LLMs provide binary classifications without probability scores like traditional methods, we only include the comparison with the MOE decider.

As shown in Table 1, our MedAgent-Pro framework significantly outperforms the current state-of-the-art multi-modal foundation models across both diagnostic tasks. The mACC metric improves by 32.3 and 19.8%, and the F1 score increases by 55.1% and 14.8%, demonstrating the effectiveness of our design on multi-agentic workflow. When faced with complex diagnoses such as glaucoma and heart disease, LLMs struggle to make accurate judgments based solely on the general appearance of fundus or echocardiography images, as they lack the ability to analyze visual features effectively. Particularly, GPT-4o often refuses to provide a clear diagnosis, while LLaVa-Med, despite being fine-tuned with medical knowledge, tends to classify all patients as healthy. The proposed MedAgent-Pro framework integrates various specialized models as tool agents to provide both analysis of both qualitative and quantitative indicators, thus approach enables precise and comprehensive diagnosis.

It can also be observed that within our MedAgent-Pro framework, the MOE decider consistently outperforms the LLM decider, achieving improvements of 14.5% and 3.0% in the mAcc metric for glaucoma and heart disease diagnosis, respectively. This is because the final diagnosis relies on multiple indicators, making it difficult for a single LLM to effectively integrate and analyze all relevant information. As a result, the LLM decider often focuses on only a subset of indicators while overlooking others, leading to reduced diagnostic accuracy. Meanwhile, the MOE decider can alleviate this issue by assigning weight for various indicators, facilitating a comprehensive decision-making.

4.3. Case Study

Figure 3 illustrates the workflow of a typical case within our MedAgent-Pro framework. At the task level, the retrieval agent gathers relevant information to establish clinical criteria, while the planner agent formulates a diagnostic plan based on the given criteria and available tools. At the case level, the orchestrator agent selects the appropriate steps from the plan based on the input data, engaging various tool agents for indicator analysis. Finally, the summary agent consolidates the findings, and the decider agent delivers the final diagnosis.

4.4. Comparison with Task-specific Models

Additionally, we compare MedAgent-Pro with domain-specific approaches, including fine-tuned MLLMs for ophthalmology [38,63] and the top-performing methods from the REFUGE2 challenge leaderboard [17]. Since the REFUGE2 leaderboard only provides the AUC metric, which measures the area under the ROC curve, and the LLM decider outputs only categorical decisions without probabilistic scores, we limit the comparison to the MOE decider.

As shown in Table 2, MedAgent-Pro maintains its superiority in comparison with the REFUGE2 challenge winners and state-of-the-art MLLMs in ophthalmology. The AUC metric has improved by 6.8%, while the mACC and F1 scores have increased by 4.6% and 3.3%, respectively. The improvement is particularly impressive, given that the challenge winners employed numerous techniques and that expert MLLMs were trained on millions of ophthalmology data, whereas the MLLMs in our MedAgent-Pro framework maintain a zero-shot setting. This finding further demonstrates that leveraging MLLMs for a specific domain does not necessarily require fine-tuning but can instead be achieved through domain-specific tools, highlighting the potential of our MedAgent-Pro framework.

4.5. Ablation Study

In the diagnosis of glaucoma, four key indicators are analyzed in fundus images: vertical Cup-to-Disc Ratio (vCDR), Rim Thickness (RT), Peripapillary Atrophy (PPA), and Disc Hemorrhages (DH). To further analyze the accuracy of the indicators themselves and their impact on the final diagnosis, we conduct comprehensive ablation studies as shown in Table 3.

Columns 5 and 6 of Table 3 analyze the impact of individual indicators. Each indicator’s positive or negative result is directly taked as the final outcome without passing through the decider agent. The accuracy of indicators varies significantly, with vCDR and PPA being notably more accurate than the other two. On one hand, different indicators vary in their relevance to glaucoma. For example, vCDR is the gold standard for glaucoma diagnosis in clinical practice, whereas the others are not. On the other hand, some indicators are inherently more challenging to analyze. For example, when analyzing DH, the lack of effective detection tools necessitates the use of VQA and classification tools. However, these tools lack fine-grained perception of image details, thereby reducing accuracy.

When multiple indicators are combined for the final diagnosis, we can get the below findings: 1) The MOE decider effectively leverages multiple indicators, leading to a significant improvement: when using only one indicator, the highest mACC achieved is 81.7%, while the highest F1 score is 74.6%. However, with the MOE decider incorporating multiple indicators, the mACC increases to 90.4% and the F1 score reaches 81.5%. By compensating for the limitations of relying on a single indicator, the MOE decider achieves a more holistic and reliable diagnosis. 2) However, the LLM decider experiences a performance decline for multiple indicators, particularly in the absence of vCDR, where the F1 score drops to just 14.3% (7th row). This is because LLMs tend to adopt a conservative diagnostic approach—when critical information (vCDR) is missing, they often fail to make a definitive diagnosis. Furthermore, when multiple indicators are provided, LLMs may hesitate to confirm glaucoma even if only one appears normal and others are clearly abnormal. 3) Less accurate indicators can negatively impact the correctness of the final diagnosis. Comparing the last two rows, the F1 score drops by 5.1% and 16.5%, respectively. This decline is due to the low accuracy of DH, which has a high false positive rate, leading to a significant number of misdiagnoses.

5. Conclusion and Future Work

This paper introduces MedAgent-Pro, a reasoning-based agentic system designed to deliver accurate, evidence-based medical diagnoses. The system adopts a hierarchical structure: at the task level, knowledge-based reasoning generates reliable diagnostic plans by integrating retrieved clinical criteria, while at the case level, evidence-based diagnosis conducts step-by-step indicator analysis using multiple tool agents. Finally, a decider agent synthesizes all indicators and provides a final diagnosis supported by visual evidence. Comprehensive experiments show that MedAgent-Pro surpasses both general MLLMs and expert methods in diagnosing two diseases, demonstrating its exceptional accuracy and interpretability.

Future works will involve the following aspects: First, we aim to expand the datasets to further demonstrate the generalization capability of MedAgent-Pro. This includes increasing data scale, diversifying tasks, and incorporating more modalities to validate the framework’s performance in complex scenarios and multi-modal inputs. Additionally, human-in-the-loop validation with expert doctors will help assess clinical applicability and provide detailed analyzes to generate qualitative results, demonstrating the superior interpretability of our method compared to MLLMs. These advancements will further enhance the reliability and impact in computer-aided diagnosis.