Maximizing Scoring Divergence in Automated Essay Assessment with LLaMA-Based Meta-Attention Networks

1. Introduction

With the rise of educational technology, AES systems have gained traction, yet traditional methods struggle with objectivity and consistency across varied writing styles and topics. Their limited sensitivity to stylistic diversity often leads to biased and constrained evaluations that overlook the full spectrum of essay quality.

This study proposes a Meta-Learning framework, DGMAN, which exploits LLaMA’s inherent biases to amplify divergence among scoring models. By generating essays that trigger discrepancies, the system enhances evaluation objectivity. DGMAN integrates LLaMA ensembles, dynamic attention, and adversarial perturbations to guide and diversify the scoring process.

DGMAN employs LLaMA-7B, 13B, and 30B, each trained on specific domains—web, academic, and informal corpora. A dynamic attention mechanism adjusts their contextual influence for style-aware scoring. Adversarial perturbations increase response divergence without sacrificing fluency, while meta-learning optimizes input generation for maximal scoring inconsistency. This strategy enables more objective and fair AES by incorporating diverse evaluative perspectives.

Li et al. [1] introduced a dual-agent approach for strategic reasoning in LLMs, enhancing their capabilities but not focusing on knowledge modification. Dai et al. [1] explored contrastive augmentation for speech technology, showing improvements in noisy data but not addressing essay scoring. Chen’s coarse-to-fine SLAM-Transformer framework for multi-view 3D reconstruction [2] directly influenced our hybrid ensemble design by demonstrating how attention-guided geometric refinement can improve robustness under viewpoint diversity, which inspired the integration of our dynamic attention mechanism for LLaMA-based scoring divergence.

2. Related Work

Recent research in automated essay scoring (AES) has used machine learning techniques to improve evaluation accuracy and fairness. However, most AES systems still face problems with consistency and objectivity, especially when evaluating diverse writing styles and topics.

One key area in AES research is the use of large language models (LLMs) for essay scoring. Wang [3] proposed the EAIN framework integrating attention and high-order interaction modules, which informed our ensemble design by demonstrating how feature-specific attention and selective masking can enhance representational diversity under sparse conditions. Jin’s work on retail sales forecasting [4] demonstrates the power of ensemble models like LightGBM, XGBoost, and deep neural networks, similar to the ensemble approach used in our research to maximize divergence in scoring.The entropy-attention-based feature selection strategy proposed by Wang et al. [5] inspired our model’s meta-perturbation component by demonstrating how adaptive attention mechanisms can be dynamically tuned for multiple objectives under coupling constraints, aligning closely with our goal of maximizing inter-model divergence while preserving textual fidelity. Guan [6] applies machine learning and network analysis to breast cancer risk prediction using NHIS 2023 data, uncovering latent disease-patient relationships often overlooked in traditional models.

Wang et al. [7] propose a deep learning-based sensor selection framework to enhance failure mode recognition and RUL prediction under time-varying conditions, addressing the overlooked heterogeneity in multi-sensor data relevance for improved prognostic accuracy. Jin [8] applied integrated machine learning techniques to improve supply chain risk prediction, offering insights into how different learning strategies can be combined to improve model performance.The Bayesian signal-inference strategy introduced by Zhang and Hart [9] influenced our design of the disagreement-driven objective by illustrating how prior parameter tuning can guide convergence under noisy observational conditions, which parallels our adaptive loss formulation in high-variance LLM scoring environments.

Wang [10] demonstrated that combining FM, GCN, and attention enhances feature discrimination, which inspired our use of structure-aware attention in LLaMA ensemble scoring. Zhang and Bhattacharya’s surrogate modeling approach [11] informed our meta-learning design by illustrating how neural approximators can efficiently capture complex dependencies with minimal computational overhead.

3. Methodology

The use of large language models (LLMs) for essay scoring has become a cornerstone in automated education systems. However, these models often suffer from intrinsic biases that can skew their evaluations. This paper proposes a novel, complex multi-level meta-learning approach to manipulate the outputs of multiple LLaMA-based LLMs in order to maximize disagreement between judges. Our model, which we call Dynamically Guided Meta-Attention Network (DGMAN), integrates a hybrid ensemble of LLaMA models, a dynamic attention mechanism, adversarial input perturbations, and meta-learning to explore and exploit model-specific biases. This framework is designed to generate input essays that systematically exploit these biases, ensuring high variance in the evaluation scores, while simultaneously maintaining linguistic integrity. The proposed approach is shown to outperform traditional models in terms of horizontal and vertical variance, demonstrating significant improvements in exploiting LLaMA model vulnerabilities to maximize model disagreement. The pipline of approach is shown in Figure 1.

3.1. Hybrid LLaMA Ensemble

We utilize an ensemble of three distinct LLaMA models with varying training data distributions and architectures. These models are:

• ${\mathrm{LLaMA}}_1$ (based on LLaMA-7B): Trained on general-purpose web data.
• ${\mathrm{LLaMA}}_2$ (based on LLaMA-13B): Fine-tuned on academic writing and research papers.
• ${\mathrm{LLaMA}}_3$ (based on LLaMA-30B): Specializes in informal language and conversational data.

Each model produces a quality score, denoted by $q_1,q_2,q_3\in [0,9]$, for a given essay x. The main objective is to create essays that maximize disagreement between these judges while maintaining the coherence and quality of the text.

Let ${\mathcal{L}}_i$ represent the loss function for each model i, which is defined as:

$${\mathcal{L}}_i\left(x\right)=\mathbb{E}\left[{\left(q_i\left(x\right)-{\widehat{q}}_i\right)}^2\right],$$

(1)

where ${\widehat{q}}_i$ is the expected score for model i, and $q_i\left(x\right)$ is the actual quality score output for essay x.

3.2. Dynamic Attention Mechanism

The DGMAN incorporates a dynamic attention mechanism, which adapts the importance of each model’s score depending on the context of the essay. The attention weights ${\alpha }_1,{\alpha }_2,{\alpha }_3$ for the three models are dynamically learned based on the input essay x and the collective performance of the models.

The attention mechanism is formally represented as:

$${\alpha }_i={\displaystyle \frac{exp\left({\mathrm{score}}_i\left(x\right)\right)}{{\sum }_{j=1}^{3}exp\left({\mathrm{score}}_j\left(x\right)\right)}},i\in \{1,2,3\}$$

(2)

where ${\mathrm{score}}_i\left(x\right)$ is the predicted quality score of model i, and ${\alpha }_i$ is the weight that model i contributes to the final disagreement measure.

The final quality score $q_{\mathrm{final}}$ is a weighted sum of the individual scores:

$$q_{\mathrm{final}}=\sum _{i=1}^3{\alpha }_i·q_i\left(x\right).$$

(3)

3.3. Adversarial Perturbation Network

To maximize disagreement, we employ adversarial input perturbations. The goal of this network is to find an optimal perturbation $\Delta x$ to the essay x such that the outputs of the models are maximally divergent, while maintaining linguistic quality.

The perturbation is generated using the following optimization problem:

$$\Delta x=arg\underset{\Delta x}{max}\left(\sum _{i=1}^3\left|q_i(x+\Delta x)-\overline{q}\left(x\right)\right|\right),$$

(4)

where $\overline{q}\left(x\right)={\displaystyle \frac{1}{3}}{\sum }_{i=1}^3{q}_i\left(x\right)$ is the mean quality score of the three models.

3.4. Meta-Learning Strategy

A meta-learning layer is introduced to optimize perturbation strategies per essay topic by learning a mapping $\Phi$ from topics t to perturbations $\Delta x$. The objective minimizes:

$${\mathcal{L}}_{\mathrm{meta}}={\mathbb{E}}_t\left[\sum _{i=1}^3\left|q_i(x+\Phi \left(t\right))-\overline{q}\left(x\right)\right|\right],$$

(5)

where $\Phi \left(t\right)$ denotes the learned strategy for topic t. This enables adaptive control over model disagreement. The meta-learning pipeline is illustrated in Figure 2.

3.5. Final Output Generation

After learning adversarial perturbations and attention weights, essays are generated to exploit model biases. The hybrid ensemble, guided by dynamic attention, emphasizes models with the greatest disagreement to amplify horizontal variance ${\mathrm{avg}}_h$ while preserving linguistic quality.

The final objective for essay generation is:

$$\begin{array}{cc}\hfill L_{\mathrm{final}}=& {\lambda }_1·\mathrm{Disagreement}(q_1,q_2,q_3)+{\lambda }_2·\mathrm{Penalty}\left(x\right)\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill & -{\lambda }_3·\mathrm{Quality}\left(x\right),\hfill \end{array}$$

(7)

where $\mathrm{Penalty}\left(x\right)$ discourages repetitive or non-English content, and $\mathrm{Quality}\left(x\right)$ promotes fluency and correctness.

3.6. Loss Function

The loss function is designed to maximize disagreement among multiple LLaMA models while ensuring the essays remain of high quality and linguistically sound.

3.6.1. Disagreement Loss

The disagreement loss encourages the models to produce divergent scores. Let $q_1,q_2,q_3$ represent the quality scores from three LLaMA models. The mean score $\overline{q}\left(x\right)$ is:

$$\overline{q}\left(x\right)={\displaystyle \frac{1}{3}}\sum _{i=1}^3{q}_i\left(x\right),$$

(8)

and the disagreement loss is defined as:

$${\mathcal{L}}_{\mathrm{disagree}}\left(x\right)=\sum _{i=1}^3{\left|q_i\left(x\right)-\overline{q}\left(x\right)\right|}^2.$$

(9)

This term penalizes scores that are too similar, encouraging the models to produce more varied assessments.

3.6.2. Penalty for Quality

To enforce linguistic quality, a penalty is applied to essays with repetitive or non-English content:

$${\mathcal{L}}_{\mathrm{penalty}}\left(x\right)={\lambda }_1·\mathrm{Rep}\left(x\right)+{\lambda }_2·\mathrm{NonEng}\left(x\right),$$

(10)

where $\mathrm{Rep}\left(x\right)$ and $\mathrm{NonEng}\left(x\right)$ denote repetitiveness and non-English content, respectively. This discourages low-quality inputs.

3.6.3. Final Loss Function

The total loss function is the weighted sum of disagreement and quality penalties:

$${\mathcal{L}}_{\mathrm{total}}={\lambda }_1·{\mathcal{L}}_{\mathrm{disagree}}\left(x\right)+{\lambda }_2·{\mathcal{L}}_{\mathrm{penalty}}\left(x\right).$$

(11)

The model is trained to minimize this loss, ensuring it maximizes model disagreement while preserving essay quality.

3.7. Data Preprocessing

Data preprocessing involves preparing the essay topics and input data to ensure maximum model disagreement while maintaining coherence.

3.7.1. Topic Normalization

Essay topics are normalized through lowercasing, tokenization, and lemmatization:

$$t_{\mathrm{norm}}=\mathrm{Lemmatize}\left(\mathrm{Tokenize}\left(\mathrm{Lowercase}\left(t\right)\right)\right).$$

(12)

This standardization ensures uniform input formatting. As shown in Figure 3, we visualized topic clusters using both simple grouping and centroid-based methods. While adding more topics raises training costs, it does not enhance model performance.

3.7.2. Text Input Preprocessing

The essays are preprocessed by removing excessive punctuation and tokenizing the text. Let x represent the original essay, and $x_{\mathrm{proc}}$ the processed essay:

$$x_{\mathrm{proc}}=\mathrm{Padding}\left(\mathrm{Tokenize}\left(\mathrm{RemovePunctuation}\left(x\right)\right)\right).$$

(13)

This prepares the essays for input into the model, ensuring consistent and efficient processing.

4. Evaluation Metrics

To evaluate the performance of our model in maximizing disagreement among LLaMA-based judges, we use several key metrics, each targeting different aspects of the evaluation process.

4.1. Horizontal Variance

Horizontal variance quantifies the disagreement among the three LLaMA models by computing the variance of their scores $q_1,q_2,q_3$:

$${\mathrm{avg}}_h={\displaystyle \frac{1}{3}}\sum _{i=1}^3{\left(q_i-\overline{q}\right)}^2,$$

(14)

where $\overline{q}$ is the mean score. A larger value reflects higher inter-model disagreement.

4.2. Vertical Variance

Vertical variance measures a model’s scoring consistency across essays. For model i, it is defined as:

$${\min }_v={\displaystyle \frac{1}{N}}\sum _{j=1}^N{(q_i\left(x_j\right)-\overline{q_i})}^2,$$

(15)

where $\overline{q_i}$ is the average score assigned by model i. Greater variance suggests a wider score distribution.

4.3. English Language Score

The English language score evaluates the linguistic quality of the essay. It is a measure of how fluent and grammatically correct the essay is. We calculate the average English confidence score across the three models:

$${\mathrm{avg}}_e={\displaystyle \frac{1}{3}}\sum _{i=1}^3\mathrm{Eng}\left(q_i\right),$$

(16)

where $\mathrm{Eng}\left(q_i\right)$ is the English confidence score from model i.

4.4. Sequence Similarity Score

This metric evaluates an essay’s structural redundancy by averaging its cosine similarity with all other essays:

$${\mathrm{avg}}_s={\displaystyle \frac{1}{N}}\sum _{j=1}^N\mathrm{cosine}\text{-}\mathrm{sim}(x_i,x_j),$$

(17)

where $\mathrm{cosine}\text{-}\mathrm{sim}(x_i,x_j)$ denotes the similarity between essays $x_i$ and $x_j$. Lower scores indicate higher uniqueness and reduced repetition.

5. Experiment Results

In this section, we evaluate the performance of different models using several metrics. The models tested include LLaMA-Base, LLaMA-Disagree, LLaMA-Adv, and LLaMA-Final.

The following metrics were used to evaluate the models: Horizontal Variance (avg_h): Measures the disagreement between the three judges for each essay. Vertical Variance (min_v): Quantifies the consistency of each judge’s scores across all essays. English Language Score (avg_e): Represents the fluency and correctness of the essay. Sequence Similarity Score (avg_s): Measures the uniqueness of the essay by comparing it to others in the dataset. The results for all models are summarized in Table 1. The changes in model training indicators are shown in Figure 4.

6. Conclusions

In this paper, we proposed a complex multi-level meta-learning approach using the LLaMA model to exploit biases in LLaMA-based judging systems. Our method, LLaMA-Final, incorporates both a disagreement loss function and adversarial perturbation preprocessing, which together maximize disagreement between the models. The experimental results show that this approach significantly increases model divergence while maintaining linguistic quality, outperforming other configurations in the ablation study. These findings highlight the potential of using such models for subjective evaluation tasks at scale, and suggest avenues for further research in improving model robustness and fairness in automated assessments.