Multimodal Large Language Model for Intelligent Diagnosis and Management of Crop Nutrient Deficiencies and Environmental Stresses

1. Introduction

The escalating global population and the pervasive impacts of climate change pose unprecedented challenges to agricultural productivity and food security. Crop health stands as a cornerstone for ensuring a stable food supply, yet it is frequently undermined not only by pests and diseases but also significantly by crop nutrient deficiencies and environmental stresses such as drought, high temperatures, and salinity. These factors are primary contributors to reduced crop yields and diminished quality. Traditional methods for diagnosing crop health issues typically rely on expert knowledge, time-consuming laboratory analyses, or manual visual inspection. Such approaches are often inefficient, labor-intensive, costly, and lack the scalability and precision required for modern, large-scale agricultural management [1].

In recent years, artificial intelligence (AI), particularly large language models (LLMs) and multimodal large language models (VLMs), has demonstrated remarkable capabilities in areas such as image recognition, natural language processing, and human-computer interaction [2]. The emergence of generative AI, for instance, has significantly elevated tasks like machine translation [3], while advancements in vision representation compression contribute to efficient video generation with LLMs [4]. Despite this potential, applying existing general-purpose vision-language models to the highly specialized domain of agriculture presents several significant challenges:

• Insufficient Domain Knowledge: Generic models often lack a deep understanding of crop physiology, plant nutrition, and environmental science, which are critical for accurate agricultural diagnostics.
• Difficulty in Fine-grained Recognition: Symptoms of crop nutrient deficiencies and environmental stresses can be subtle and often mimic those of other issues, such as diseases, demanding highly specialized visual discernment capabilities.
• Multimodal Information Integration: Farmers and agricultural technicians typically provide multi-modal inputs, combining crop images with textual descriptions (e.g., "Why are the leaves turning yellow?"). Existing models struggle to effectively integrate these diverse data sources for accurate diagnosis.
• Generation of Professional Advice: Beyond diagnosis, there is a crucial need for models to generate professional, actionable, and contextually coherent management recommendations that are easily understood and implemented by users.

Figure 1. Addressing the Challenges in Agricultural AI Diagnosis with AgriHealth-LLM.

Figure 1. Addressing the Challenges in Agricultural AI Diagnosis with AgriHealth-LLM.

To address these limitations and bridge the existing gaps, this research aims to design and develop a specialized multimodal large language model for crop nutrient deficiency and environmental stress, facilitating more precise and efficient intelligent diagnosis and management advice, thereby enhancing agricultural productivity and sustainability.

We propose a novel multimodal large language model, termed AgriHealth-LLM, specifically engineered to integrate visual and linguistic information for the intelligent identification of crop nutrient deficiencies and environmental stresses, and to provide expert management recommendations. The core mission of AgriHealth-LLM is to process multimodal inputs, comprising crop images and user queries, to first identify the specific type of nutrient deficiency or environmental stress (e.g., nitrogen deficiency, potassium deficiency, drought stress, heat stress). Subsequently, based on the identification results and further user inquiries, the model offers personalized, professional, and actionable management and prevention advice through a multi-turn conversational interface, drawing inspiration from methods for aligning LLMs with implicit user feedback in conversational systems [5]. The architecture of AgriHealth-LLM mirrors that of advanced VLM structures like VisualGLM [6], incorporating three key modules: a Vision Encoder (based on a pre-trained Vision Transformer (ViT) [7] adapted for agricultural nuances), a Modality Aligner (inspired by Q-Former [8] to bridge visual and linguistic feature spaces), and a Language Model (utilizing ChatGLM-6B [9] for its robust Chinese comprehension and generation capabilities). For efficient adaptation to the specialized agricultural domain, we employ the Low-Rank Adaptation (LoRA) [10] method for fine-tuning. This approach, coupled with research into model and data parallelism optimization methods for large language models [11], significantly reduces computational overhead while maintaining high performance.

To validate the efficacy of AgriHealth-LLM, we constructed a custom Chinese agricultural multimodal dataset called AgriDiagnose-MVD. This dataset focuses exclusively on crop nutrient deficiencies and environmental stresses, comprising 3,560 color images covering 185 distinct categories across 20 common crop types (e.g., rice, corn, wheat, tomato, cucumber). Each image is meticulously annotated with detailed descriptions of symptoms and corresponding deficiency/stress types, complemented by 3-5 relevant question-answer pairs per image to train the model’s diagnostic and advice generation capabilities. Our experiments involved fine-tuning AgriHealth-LLM using LoRA, with a learning rate of 0.0001, batch size of 4, and 12,000 training steps, utilizing data augmentation techniques for images and standard tokenization for text. The model’s performance was rigorously evaluated using Accuracy and F1-Score for identification tasks, and a hybrid approach involving GPT-4 assisted evaluation alongside agricultural expert manual scoring for the quality of generated management advice, assessing professionalism, accuracy, practicality, and fluency.

Experimental results demonstrate the superior performance of AgriHealth-LLM compared to several mainstream vision-language models, including MiniGPT4 [12], Qwen-VL [13], VisCPM [14], and VisualGLM [15]. In the "Nutrient Deficiency Identification" task, AgriHealth-LLM achieved an accuracy of 83.5% and an F1-Score of 82.1%, outperforming the closest competitor, VisualGLM, by a significant margin. Similarly, for the "Environmental Stress Identification" task, our model reached an accuracy of 80.9% and an F1-Score of 79.5%, again setting a new benchmark. Furthermore, in the "Question Answering (Comprehensive Management Advice)" task, AgriHealth-LLM attained an average score of 87.2 out of 100, as evaluated by GPT-4 and agricultural experts, indicating its exceptional ability to generate professional and practical advice. These results collectively underscore the effectiveness and robustness of AgriHealth-LLM in intelligent agricultural diagnosis and management.

Our primary contributions are summarized as follows:

• We propose AgriHealth-LLM, a novel multimodal large language model specifically tailored for the intelligent diagnosis and management of crop nutrient deficiencies and environmental stresses, addressing critical challenges in agricultural AI.
• We introduce AgriDiagnose-MVD, a comprehensive, self-curated Chinese agricultural multimodal dataset focusing on fine-grained identification of nutrient deficiencies and environmental stresses, which is crucial for training specialized agricultural VLMs.
• We demonstrate that AgriHealth-LLM significantly outperforms existing general-purpose vision-language models across various tasks, including nutrient deficiency identification, environmental stress identification, and the generation of professional management advice, showcasing its potential for real-world agricultural applications.

1.1. Large Vision-Language Models

Large Vision-Language Models (LVLMs) draw from diverse fields such as augmented cognition [2]. Their vision component is enhanced by foundational research in computer vision, including Simultaneous Localization and Mapping (SLAM) for better spatial understanding [16,17]. Principles from systems like SurveyMan help engineer robust, structured outputs [18], while models such as Video-ChatGPT advance video-based conversational AI [19]. Key research areas include leveraging generative imagination [3], vision representation compression for efficiency [4], and multi-teacher distillation [20]. For multimodal learning, graph-based frameworks like MTAG [21] and continuous-time dynamic graphs [22] are significant. Deeper semantic challenges are addressed by research into implicit meaning [23]. Studies also investigate the few-shot capabilities of models like CLIP [24], fine-tuning for alignment with user feedback [5], and the mechanisms of in-context learning [25]. Efficiency is improved through parameter-efficient methods like LN-tuning [26] and parallelism optimization [11]. Advances in visual and cross-modal understanding stem from improved feature learning [27], multimodal fusion [28], and novel retrieval techniques [29,30]. Important considerations include ethical fairness [31], out-of-distribution robustness [32], and security against backdoor attacks [33]. System stability and robustness also draw parallels from control theory [34,35,36]. Finally, models like the Knowledge Augmented Transformer (KAT) focus on integrating external knowledge for tasks such as VQA [37].

1.2. AI for Crop Health Diagnosis and Management

Applying AI to crop health benefits from research in other domains, such as using LLMs for interpretable decision-making to provide farmers with explainable diagnostics [38]. Advanced NLP techniques, including cross-lingual knowledge graph question answering [39], robust text retrieval [40], and evidence validation methods [41], can enhance agricultural AI systems’ ability to process diverse information. Language understanding benchmarks like CBLUE are vital for adapting NLP models to specialized agricultural data [42]. Furthermore, predictive modeling methodologies from other fields, such as finance [43], can be adapted for tasks like yield forecasting. Practical deployment relies on robust infrastructure, including advanced communication hardware for data transmission [44,45,46] and precise motor control for agricultural robotics [47,48,49]. Rigorous evaluation is also crucial, drawing from benchmarking practices in other fields [50] and multilingual assessments to ensure global applicability of generative models for tasks like phenotyping data interpretation [51].

2. Method

In this section, we present the details of our proposed multimodal large language model, AgriHealth-LLM, designed for the intelligent diagnosis and management of crop nutrient deficiencies and environmental stresses. AgriHealth-LLM aims to effectively integrate visual and linguistic information to provide precise identification and actionable agricultural recommendations, thereby enhancing agricultural productivity and sustainability.

Figure 2. Overview of the AgriHealth-LLM architecture for intelligent crop health diagnosis and management. It integrates visual and textual inputs through a Vision Encoder, Modality Aligner, and a LoRA fine-tuned Language Model to provide precise identification and actionable advice.

Figure 2. Overview of the AgriHealth-LLM architecture for intelligent crop health diagnosis and management. It integrates visual and textual inputs through a Vision Encoder, Modality Aligner, and a LoRA fine-tuned Language Model to provide precise identification and actionable advice.

The core mission of AgriHealth-LLM is to process multimodal inputs, which consist of a crop image and a user’s textual query. Upon receiving these inputs, the model performs two primary tasks. Firstly, for Identification, it discerns the specific type of crop nutrient deficiency or environmental stress present, such as nitrogen deficiency, potassium deficiency, drought stress, or heat stress. This precise identification is crucial for targeted intervention. Secondly, for Management Recommendation, based on the identified issue and any subsequent user inquiries, the model generates personalized, professional, and actionable management and prevention advice. This is facilitated through a multi-turn conversational interface, allowing for dynamic interaction and refinement of recommendations based on user feedback.

2.1. Model Architecture

AgriHealth-LLM adopts a modular architecture inspired by advanced vision-language models, comprising three interconnected modules: a Vision Encoder, a Modality Aligner, and a Language Model. This modular design facilitates robust processing of diverse agricultural data by allowing specialized components to handle different data types and then seamlessly integrate their outputs.

2.1.1. Vision Encoder

The Vision Encoder is responsible for extracting rich, high-dimensional visual features from the input crop images. We utilize a pre-trained Vision Transformer (ViT) as the backbone for this module. The ViT is adapted to better capture the subtle visual cues indicative of crop health status, such as discoloration patterns, leaf deformation, growth abnormalities, and lesion characteristics, which are critical for accurate diagnosis in agricultural contexts. Given an input image I, the Vision Encoder processes it to generate a sequence of visual feature embeddings $V=\{v_1,v_2,\dots ,v_N\}$, where N represents the number of visual tokens or patches extracted from the image. Each $v_i$ encapsulates localized visual information.

$$\begin{array}{c}\hfill V=\mathrm{VisionEncoder}\left(I\right)\end{array}$$

(1)

The use of a pre-trained ViT allows us to leverage extensive knowledge learned from large-scale image datasets, providing a strong foundation for agricultural image understanding.

2.1.2. Modality Aligner

The Modality Aligner serves as a crucial bridge between the visual and linguistic domains. Drawing inspiration from query-based transformer architectures, we construct a lightweight module specifically designed to align the extracted visual features with the embedding space of the language model. This alignment ensures that the semantic content embedded within the image, such as signs of disease or deficiency, can be effectively understood and leveraged by the language model for reasoning and generation. The aligner takes the sequence of visual features V and a set of learnable query embeddings $Q=\{q_1,q_2,\dots ,q_M\}$ as input. These queries act as a mechanism to selectively extract the most relevant information from the visual features. The aligner then processes these inputs to produce a set of aligned visual-linguistic features $F_{\mathrm{aligned}}$.

$$\begin{array}{c}\hfill F_{\mathrm{aligned}}=\mathrm{ModalityAligner}(V,Q)\end{array}$$

(2)

These aligned features encapsulate the most relevant visual information in a compressed and semantically coherent format, making it directly consumable by the subsequent language model. This process is vital for translating raw pixel information into a conceptual representation that the language model can integrate with textual queries.

2.1.3. Language Model

The Language Model forms the core of AgriHealth-LLM’s reasoning and generation capabilities. We employ ChatGLM-6B as the foundational language model due to its robust capabilities in Chinese language understanding, generation, and conversational interaction, which are essential for serving agricultural users in relevant regions. The aligned visual-linguistic features $F_{\mathrm{aligned}}$ are concatenated with the token embeddings of the user’s textual query $T_{\mathrm{query}}$. The textual query is first processed by an embedding layer to convert it into a sequence of dense vector representations, $\mathrm{Embed}\left(T_{\mathrm{query}}\right)$. This concatenation forms a comprehensive input representation X, which combines both visual and textual information into a unified sequence. The Language Model then processes this integrated input X to generate the output sequence Y, which includes the precise diagnosis of the crop issue and detailed management advice.

$$\begin{array}{cc}\hfill X& =\mathrm{Concatenate}(\mathrm{Embed}\left(T_{\mathrm{query}}\right),F_{\mathrm{aligned}})\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill Y& =\mathrm{LanguageModel}\left(X\right)\hfill \end{array}$$

(4)

This architecture allows AgriHealth-LLM to perform complex multimodal reasoning by integrating information from both modalities and generating coherent, contextually relevant, and actionable responses in a conversational manner. The output Y can range from a direct identification of the problem to a multi-turn dialogue providing step-by-step guidance.

2.2. Training Strategy

To efficiently adapt the general-purpose large models to the highly specialized agricultural domain, we adopt the Low-Rank Adaptation (LoRA) method for fine-tuning AgriHealth-LLM. LoRA significantly reduces the computational resources and time required for training while maintaining high model performance, making it an ideal choice for domain-specific adaptation of large pre-trained models.

During the fine-tuning process, the majority of the parameters within the pre-trained Vision Encoder and Language Model are frozen. LoRA achieves efficient adaptation by injecting small, trainable low-rank matrices into the attention layers of the pre-trained model. Specifically, for a pre-trained weight matrix $W_0\in {\mathbb{R}}^{d\times k}$ in a large model, LoRA introduces two low-rank matrices, $B\in {\mathbb{R}}^{d\times r}$ and $A\in {\mathbb{R}}^{r\times k}$, where the rank r is significantly smaller than $min(d,k)$. The updated weight matrix W used during fine-tuning becomes the sum of the original weight matrix and the product of these low-rank matrices:

$$\begin{array}{c}\hfill W=W_0+BA\end{array}$$

(5)

In this formulation, only the parameters within matrices B and A are trained, while the original pre-trained weights $W_0$ remain fixed. This approach drastically reduces the number of trainable parameters, leading to faster training and lower memory consumption. In our implementation, we primarily train the Modality Aligner from scratch and fine-tune the Language Model by injecting LoRA matrices into its attention mechanisms. This strategy enables AgriHealth-LLM to acquire specialized agricultural knowledge and adapt to the specific nuances of crop health diagnosis and management without catastrophically forgetting its general capabilities and language understanding.

3. Experiments

In this section, we detail the experimental setup, dataset construction, evaluation metrics, and comprehensive results demonstrating the efficacy of our proposed AgriHealth-LLM in diagnosing crop nutrient deficiencies and environmental stresses, as well as providing actionable management advice.

3.1. Dataset

To facilitate the development and evaluation of AgriHealth-LLM, we constructed a novel, specialized Chinese agricultural multimodal dataset named AgriDiagnose-MVD. This dataset is meticulously curated to focus on the fine-grained identification of crop nutrient deficiencies and environmental stresses, addressing the scarcity of such specialized resources in the agricultural domain.

AgriDiagnose-MVD comprises a rich collection of multimodal data:

• Crop Images: The dataset includes 3,560 high-resolution color images depicting various common crops, such as rice, corn, wheat, tomato, and cucumber, at different growth stages. These images capture diverse visual symptoms associated with nutrient deficiencies and environmental stresses.
• Image Descriptions and Labels: Each image is accompanied by detailed annotations, including the precise type of nutrient deficiency (e.g., nitrogen deficiency, potassium deficiency) or environmental stress (e.g., drought stress, high temperature stress) it represents. Furthermore, comprehensive textual descriptions of the observed symptoms are provided to enrich the visual information.
• Question-Answer Pairs: To train the model’s diagnostic and advice generation capabilities, 3 to 5 relevant question-answer pairs are associated with each image. These pairs cover a range of inquiries, such as "What is this problem?", "How can I identify it?", and "What management strategies should I apply?". This structured Q&A format is crucial for developing a conversational AI assistant.

In total, AgriDiagnose-MVD encompasses 3,560 color images, covers 185 distinct categories of nutrient deficiencies and environmental stresses, and involves 20 common crop types, providing a robust foundation for domain-specific model training.

3.2. Experimental Setup

Our experimental methodology involved fine-tuning the AgriHealth-LLM on the bespoke AgriDiagnose-MVD dataset. The training process leveraged the Low-Rank Adaptation (LoRA) [10] method, which is highly efficient for adapting large pre-trained models to specific domains.

3.2.1. Training and Fine-tuning Details

During fine-tuning, the majority of parameters in the pre-trained Vision Encoder and Language Model components of AgriHealth-LLM were frozen. LoRA was applied by injecting low-rank matrices into the attention mechanisms of the Language Model, allowing for efficient adaptation with a significantly reduced number of trainable parameters. The Modality Aligner was trained from scratch to ensure optimal alignment between visual and linguistic features. The specific training parameters were configured as follows: the learning rate was set to 0.0001, the batch size was 4, and the model was trained for 12,000 steps. The rank for LoRA matrices was set to 16.

3.2.2. Data Preprocessing

For image data, standard data augmentation techniques were applied, including normalization, random cropping, and random horizontal flipping, to enhance the model’s generalization capabilities and robustness to variations in input images. Textual data underwent tokenization and encoding processes to convert natural language into a numerical format suitable for the language model. Crucially, during the training of the Modality Aligner, we constructed positive and negative sample pairs for an image-text matching binary classification task. This approach ensures that the visual features are effectively aligned with their corresponding textual semantics, improving the model’s ability to integrate multimodal information.

3.3. Evaluation Metrics

To thoroughly assess the performance of AgriHealth-LLM, we employed a combination of quantitative metrics for identification tasks and a hybrid evaluation approach for the quality of generated advice.

3.3.1. Identification Tasks

For the "Nutrient Deficiency Identification" and "Environmental Stress Identification" tasks, where the model is required to classify the specific type of issue from an input image, we utilized two widely accepted metrics:

• Accuracy: Measures the proportion of correctly identified instances out of the total number of instances.
• F1-Score: Represents the harmonic mean of precision and recall, providing a balanced measure of the model’s performance, especially valuable in scenarios with imbalanced class distributions.

3.3.2. Management Advice Generation Task

For the "Question Answering (Comprehensive Management Advice)" task, which evaluates the quality and practicality of the model’s generated recommendations, a more nuanced evaluation approach was necessary. We combined automated and human-centric assessments:

• GPT-4 Assisted Evaluation: We leveraged the advanced capabilities of GPT-4 to provide an initial automated assessment of the generated advice, focusing on aspects such as coherence, relevance, and factual correctness.
• Agricultural Expert Manual Scoring: A panel of agricultural experts independently reviewed and scored the model’s generated management advice. The scoring was based on a comprehensive set of dimensions, including professionalism, accuracy, practical utility, and language fluency. Each piece of advice was rated on a scale of 0 to 100, with higher scores indicating superior quality.

3.4. Baseline Methods

To contextualize the performance of our proposed AgriHealth-LLM, we conducted comparative experiments against several state-of-the-art general-purpose vision-language models. These baselines represent diverse architectural designs and pre-training strategies in the multimodal AI landscape:

• MiniGPT4 [12]: A lightweight yet powerful vision-language model known for its efficient training and strong multimodal capabilities.
• Qwen-VL [13]: A prominent large vision-language model developed by Alibaba Cloud, recognized for its comprehensive understanding of visual and textual information.
• VisCPM [14]: Another powerful multimodal model, often cited for its performance in various vision-language benchmarks.
• VisualGLM [15]: A multimodal model built upon the GLM architecture, demonstrating strong performance in vision-language tasks, particularly with Chinese language support.

These models were evaluated on the same AgriDiagnose-MVD dataset under comparable conditions to provide a fair assessment of AgriHealth-LLM’s specialized domain adaptation.

3.5. Results and Discussion

Our experimental results, presented in Table 1, Table 2, and Table 3, demonstrate the superior performance of AgriHealth-LLM across all evaluated tasks compared to the baseline models.

3.5.1. Nutrient Deficiency Identification

Table 1 summarizes the performance of AgriHealth-LLM and baseline models on the "Nutrient Deficiency Identification" task. This task requires the model to accurately identify the specific type of nutrient deficiency from an input crop image.

As shown in Table 1, AgriHealth-LLM achieved an accuracy of 83.5% and an F1-Score of 82.1%. This significantly surpasses the performance of general-purpose models like MiniGPT4 (65.2% Accuracy, 62.8% F1-Score) and even specialized VLM-like VisualGLM (79.3% Accuracy, 77.5% F1-Score). This indicates that our domain-specific adaptation, coupled with the tailored dataset, enables AgriHealth-LLM to better capture the subtle visual cues associated with various nutrient deficiencies.

3.5.2. Environmental Stress Identification

Table 2 presents the results for the "Environmental Stress Identification" task, where the models were evaluated on their ability to identify different types of environmental stresses from crop images.

For environmental stress identification, AgriHealth-LLM again demonstrated superior performance, achieving an accuracy of 80.9% and an F1-Score of 79.5%. This consistent outperformance over baselines (e.g., VisualGLM at 76.8% Accuracy and 74.9% F1-Score) underscores the effectiveness of AgriHealth-LLM’s specialized architecture and fine-tuning strategy in handling the complexities of environmental stress symptoms, which can often be subtle and overlap.

3.5.3. Comprehensive Management Advice Generation (Question Answering)

Table 3 illustrates the models’ capabilities in generating comprehensive management advice, evaluated through a combined GPT-4 assisted assessment and agricultural expert manual scoring. This task assesses not only diagnostic accuracy but also the professional, practical, and fluent generation of actionable recommendations.

In the crucial task of generating comprehensive management advice, AgriHealth-LLM achieved an average score of 87.2 out of 100. This score, derived from a rigorous evaluation by GPT-4 and agricultural experts, significantly exceeds that of the best baseline, VisualGLM (81.0). This demonstrates AgriHealth-LLM’s exceptional ability to not only accurately diagnose issues but also to translate these diagnoses into professional, practical, and contextually relevant advice, presented in a natural and fluent conversational style. The high score is a testament to the model’s deep integration of specialized agricultural knowledge, allowing it to provide actionable insights vital for real-world agricultural applications.

The consistent superior performance of AgriHealth-LLM across all three evaluation tasks highlights the critical importance of domain-specific adaptation for large multimodal models in specialized fields like agriculture. By tailoring the model architecture, training strategy, and dataset to the nuances of crop health diagnosis, AgriHealth-LLM effectively addresses the challenges of insufficient domain knowledge, fine-grained recognition, and the generation of professional advice that general-purpose models often face.

3.6. Ablation Studies

To systematically understand the contribution of each key component within the AgriHealth-LLM architecture, we conducted a series of ablation studies. These experiments isolate the impact of the Modality Aligner and the LoRA-based fine-tuning of the Language Model on overall performance. The results are summarized in Table 4.

• Base (Frozen LM): In this configuration, visual features from the Vision Encoder are directly concatenated with textual embeddings and fed into the Language Model, whose parameters are largely frozen. This setup serves as a minimal baseline, demonstrating the difficulty of direct multimodal integration without specialized alignment or adaptation. As shown, its performance is significantly lower across all tasks.
• Aligned-FrozenLM: Here, we introduce the Modality Aligner but keep the Language Model parameters frozen. The improvement in NDI Accuracy (from 70.1% to 75.8%), ESI Accuracy (from 67.5% to 72.9%), and QA Score (from 71.3 to 76.5) highlights the critical role of the Modality Aligner. It effectively transforms raw visual features into a semantically richer representation that the frozen Language Model can better utilize, even without fine-tuning its core weights. This validates the design choice of a dedicated alignment module.
• Direct-LoRALM: This variant omits the Modality Aligner, directly concatenating visual features, but applies LoRA fine-tuning to the Language Model. While this configuration shows an improvement over the Base model (NDI Acc. 78.4%, ESI Acc. 75.1%, QA Score 79.8), it still underperforms the full AgriHealth-LLM. This suggests that while adapting the Language Model is beneficial, effective alignment of visual features is equally, if not more, important for nuanced multimodal understanding in our agricultural context. The raw visual features, even with an adapted LM, are not as readily interpretable as the aligned features.
• AgriHealth-LLM (Full): Our proposed model, combining both the Modality Aligner and LoRA-based Language Model fine-tuning, consistently achieves the best performance across all metrics. This synergistic effect underscores that both specialized modality alignment and efficient domain adaptation of the language model are indispensable for achieving state-of-the-art results in complex agricultural diagnosis and recommendation tasks. The Modality Aligner ensures that the visual information is presented in an optimal format for the Language Model, while LoRA enables the Language Model to deeply integrate this new domain knowledge without extensive computational overhead.

4. Conclusions

This study introduced AgriHealth-LLM, a novel multimodal large language model designed to address critical agricultural challenges arising from crop nutrient deficiencies and environmental stresses, overcoming the limitations of traditional and general-purpose diagnostic methods. Featuring an adapted Vision Encoder, a specialized Modality Aligner, and a LoRA-tuned ChatGLM-6B, and trained on our meticulously constructed AgriDiagnose-MVD dataset, AgriHealth-LLM demonstrated superior performance. It consistently outperformed state-of-the-art general vision-language models across all evaluated tasks, achieving high accuracy in identifying nutrient deficiencies (83.5%) and environmental stresses (80.9%), and generating expert-validated comprehensive management advice (87.2 average score). Ablation studies confirmed the indispensable contributions of its specialized architecture and domain-specific fine-tuning. While future work will focus on refining fine-grained distinctions, handling ambiguities, and integrating real-time data for proactive strategies, AgriHealth-LLM represents a significant advancement towards leveraging AI for sustainable agriculture, empowering farmers, and ensuring global food security.