Evaluating Open-Source LLMs for Automated Essay Scoring: The Critical Role of Prompt Design

1. Introduction

Automated Essay Scoring (AES) has long held promise as a scalable solution for formative and summative assessment in educational settings. By providing timely, consistent, and cost-effective feedback, AES systems can alleviate instructor workloads and expand access to high-quality writing evaluation.

Large Language Models (LLMs) have transformed artificial intelligence by enabling advanced language understanding, reasoning, and generation across domains such as healthcare, finance, and education [1].

Despite significant progress in areas such as contextual reasoning and ethical alignment, many current evaluation strategies remain narrow in scope. They often rely on limited benchmarks or assess individual models in isolation, failing to capture broader issues such as algorithmic bias, and performance scalability across diverse linguistic and cultural contexts [2].

AES represents a particularly compelling use case for LLMs in education, with the potential to address persistent challenges in manual essay grading namely inconsistency, subjectivity, and inefficiency. Traditional grading methods often suffer from grader fatigue, implicit biases, and delayed feedback, all of which can hinder both the reliability of assessment and student learning outcomes [3]. LLM-powered AES systems offer scalable, fast, and consistent alternatives that, when aligned with robust scoring rubrics, can approach or even exceed the performance of human raters [4].

Initially, this study intended to leverage both the Automated Student Assessment Prize (ASAP) 2.0 and PERSUADE 2.0 datasets. While ASAP offers a single holistic score that bundles various writing dimensions together [5], PERSUADE 2.0 separates the evaluation into six distinct criteria Position, Claim, Evidence, Counterclaim, Rebuttal, and a Holistic score enabling finer-grained assessment and interpretability for argumentative writing tasks. Given the focus of this study on prompt customization strategies as detailed in the Experiments chapter, PERSUADE 2.0 provides a more suitable framework for evaluating how well Large Language Models respond to targeted prompts, especially those aligned with argumentative structure. [6]

Crucially, emerging evidence suggests that how we prompt LLMs may matter as much as—or more than—the models themselves. Techniques such as Chain-of-Thought (CoT) reasoning or explicit rubric integration have shown promise in structured scoring tasks, but their impact across diverse open-source architectures has not been systematically evaluated in the context of essay assessment.

To address these gaps, we present a comprehensive evaluation of five representative open-source LLMs—LLaMA 3.2 3B, DeepSeek-R1 7B, Mistral 8×7B, Qwen2 7B, and Qwen2.5 7B—for AES on the PERSUADE 2.0 dataset.

We benchmark their performance under three prompting strategies: (1) basic instruction-based prompting, (2) Chain-of-Thought (CoT) prompting, and (3) rubric-aligned prompting that explicitly embeds scoring criteria. Using a suite of standard AES metrics—including F1 Score, Mean Absolute Error (MAE), Pearson and Spearman correlations, and Cohen’s k—we assess both scoring accuracy and alignment with human judgments. In addition, we conduct human comparison studies to evaluate which prompting approach yields outputs most consistent with expert graders.

The rest of this paper is organized as follows: Section 2 review related works, Section 3 describe our approach, Section 4 presents experimental results. Finally, Section 5 draw conclusions.

2. Related Works

Automated Essay Scoring (AES) has undergone a fundamental transformation over the past two decades—from rule-based systems relying on surface-level features to modern Large Language Models (LLMs) capable of nuanced semantic and rhetorical analysis. This section situates our work within this trajectory, reviewing (i) traditional and neural AES approaches, (ii) the rise of LLMs in educational assessment, and (iii) the emerging role of prompt design. We conclude by identifying key gaps that motivate our study.

2.1. Traditional AES Approaches

Early AES systems such as Project Essay Grade (PEG) [7] and e-rater [8] relied on handcrafted linguistic features—including grammar counts, lexical diversity, and syntactic complexity—combined with regression or decision tree models. While interpretable and effective for constrained tasks, these systems struggled to capture higher-order writing traits such as argument coherence, logical development, or rhetorical sophistication. Later machine learning methods, including SVMs, Random Forests, and Gradient Boosting [9,10], improved generalization but remained bottlenecked by manual feature engineering, limiting their scalability and cross-prompt adaptability. As a result, these systems often failed to align with holistic or analytic rubrics, particularly for argumentative writing [11,12].

2.2. Neural and Deep Learning Models

Neural architectures such as LSTMs [13] and CNNs reduced reliance on handcrafted features by learning representations directly from raw text. These models captured sequential and local discourse patterns more effectively but still struggled with long-range dependencies and holistic reasoning. Transformer-based architectures [14] addressed these limitations through self-attention mechanisms, enabling superior contextual modeling and paving the way for LLM-based AES.

2.3. LLMs in Automated Essay Scoring

Large Language Models—including GPT, BERT, LLaMA, Mistral, DeepSeek, and Qwen—leverage large-scale pretraining to capture rich linguistic and semantic patterns [15,16,17]. In AES, they support both holistic and analytic scoring, as well as feedback generation and rubric alignment [18,19]. Recent studies show that GPT-4 can achieve human-level agreement in specific contexts [20,21]. However, reliance on closed, proprietary models raises concerns about transparency, reproducibility, and equitable access—especially in public education. Consequently, open-source LLMs offer a compelling alternative, yet research on their AES capabilities remains sparse, particularly in systematic, multi-model comparisons under standardized conditions.

2.4. Prompting Strategies in AES

Prompt engineering has emerged as a critical factor in LLM-based AES performance [19]. Instruction-based prompting explicitly directs models to apply scoring criteria; rubric-aligned prompting embeds detailed definitions of each dimension (e.g., Lead, Counterclaim); and Chain-of-Thought (CoT) prompting elicits step-by-step reasoning before scoring. Recent work demonstrates that structured, rubric-integrated prompts significantly improve scoring accuracy and interpretability [22]. However, most evaluations lack systematic comparison across multiple open models and diverse prompt formulations.

Notably, [23] found significant misalignment between LLMs and human graders in analytic rubric-based scoring, attributing this gap to LLMs’ limited logical reasoning. They emphasize the need to better mirror human evaluators’ judgment processes. Similarly, [22] proposed a hybrid evaluation framework combining automatic metrics and human assessment, showing that LLMs can approach expert-level performance when carefully prompted—but also highlighting risks of overfitting and poor generalization across prompts.

2.5. Research Gap

Despite these advances, three key gaps remain: (i) limited benchmarking of modern open-source LLMs for AES across consistent datasets and metrics; (ii) insufficient understanding of how prompt design choices—from minimal instructions to full rubric integration—affect scoring reliability and human alignment; and (iii) a lack of reproducible, rubric-grounded evaluation frameworks that combine quantitative metrics with qualitative human comparison.

This study directly addresses these gaps through a comparative evaluation of five open-weight LLMs on the PERSUADE 2.0 dataset under three distinct prompting strategies—rubric-aligned, instruction-based, and minimal instruction—using standard AES metrics and human-graded validation.

3. Our Approach for Automated Essay Scoring

The primary aim of this work is to assess the effectiveness of LLMs in replicating human scoring, particularly within the context of argumentative writing. Rather than using fine-tuning techniques.

Five state-of-the-art open-source LLMs LLaMA 3.2 3B, DeepSeek-R1 7B, Mistral 8x7B, Qwen2 7B, and Qwen2.5 7B were selected for their advanced natural language Each model was asked to give an overall essay score from 1 to 6, using grading rules similar to those used by human graders.

3.1. Dataset

This study employs the PERSUADE 2.0 corpus, a large-scale dataset collaboratively developed by The Learning Agency and Vanderbilt University [6]. Released under the CC BY-NC-SA 4.0 license, PERSUADE 2.0 substantially extends the earlier PERSUADE 1.0 version by including both holistic essay scores and fine-grained discourse annotations for more than 25,000 argumentative essays written by students in grades 6–12 across the United States. The dataset spans 15 writing prompts covering both independent and source-based argumentative tasks.

Each essay is annotated with a holistic score on a 1–6 scale and discourse-level labels derived from a rubric grounded in Toulmin’s argumentative framework [24] and refined using prior work by Nussbaum et al. [25] and Stapleton and Wu [26]. Annotated elements include:

• Lead (introductory segment),
• Position (explicit stance),
• Claim (supporting argument),
• Counterclaim (opposing viewpoint),
• Rebuttal (response to counterclaim),
• Evidence (facts/examples supporting claims),
• Concluding Summary (restatement of key arguments)

Annotation followed a double-blind rating process with full adjudication, where two expert raters independently evaluated each essay, and disagreements were resolved by a third rater to ensure high reliability.

Additionally, the dataset provides comprehensive demographic metadata, including grade level, gender, race/ethnicity, English language learner (ELL) status, disability status, socioeconomic indicators, and prompt/task identifiers, enabling detailed analyses of fairness and bias in automated essay scoring (AES) systems.

For this study, the main file utilized was:

• persuade_2.0_human_scores_demo_id_github.csv, which contains full essay texts, holistic scores, demographic profiles, writing task types, source text information, and prompt metadata.

Overall, PERSUADE 2.0 offers a rich, multi-layered benchmark for evaluating Large Language Models (LLMs) in AES, supporting investigations into both scoring accuracy and equity across diverse demographic subgroups [6].

Table 1 presents a representative snippet of the dataset, illustrating the structure and content of the essays along with their corresponding scores. This excerpt is provided to give readers a clear understanding of the dataset’s format, features, and the type of textual input used in the experiments.

3.2. Model Selection

We evaluate five open-source LLMs spanning diverse architectures and design objectives; their key characteristics are summarized in Table 2. Models were selected to represent diverse architectures and parameter scales while remaining lightweight enough for resource-constrained educational deployment. Their open-source nature ensures transparency and reproducibility in AES research.

3.3. Prompt Engineering

To systematically investigate the impact of prompt design on automated essay scoring, we implement three distinct prompting strategies that vary in the level of guidance and structural scaffolding provided to the model. These range from minimal directives that merely name scoring dimensions to fully explicit rubric-embedded instructions. Table 3 presents representative examples of each prompt type used in our experiments, along with their corresponding classifications. Notably, all prompts enforce a constrained output format—requiring only a numerical score—to ensure consistency and comparability across conditions, while differing in how (or whether) they articulate the underlying assessment criteria or reasoning process. This design allows us to isolate the effect of prompt engineering on scoring accuracy and alignment with human judgments.

3.4. Evaluation Metrics

To rigorously evaluate the performance of each Large Language Model (LLM) in predicting essay scores, a comprehensive set of evaluation metrics was employed. These metrics were selected to assess not only the accuracy of the models but also their alignment with human grading and their reliability across diverse student responses. Together, they offer a robust and multidimensional framework for assessing LLM effectiveness in Automated Essay Scoring (AES).

3.4.1. Exact Match (EM)

Exact Match (EM) serves as a straightforward baseline metric that quantifies the percentage of model-generated scores that exactly match those assigned by human raters [32].

$$EM={\displaystyle \frac{\mathrm{Number}\mathrm{of}\mathrm{exact}\mathrm{matches}}{\mathrm{Total}\mathrm{number}\mathrm{of}\mathrm{essays}}}$$

3.4.2. F1 Score

The F1 Score, calculated as the harmonic mean of precision and recall, offers a balanced evaluation of the model’s performance [33].

$$F1=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

3.4.3. Mean Absolute Error (MAE)

MAE evaluates the average magnitude of deviation between predicted and actual human scores [34].

$$MAE={\displaystyle \frac{1}{N}}\sum _{i=1}^N|y_i-{\widehat{y}}_i|$$

3.4.4. Root Mean Squared Error (RMSE)

RMSE, like MAE, measures the deviation between predicted and actual scores but places a greater emphasis on larger errors [35].

$$RMSE=\sqrt{{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{(y_i-{\widehat{y}}_i)}^2}$$

3.4.5. Average Absolute Deviation (AAD)

AAD measures the average dispersion between the predicted and human-assigned scores [36].

$$AAD={\displaystyle \frac{1}{N}}\sum _{i=1}^N|(y_i-\overline{y})-({\widehat{y}}_i-\overline{\widehat{y}})|$$

3.4.6. Pearson and Spearman Correlation

The Pearson correlation assesses the linear relationship between model predictions and human scores, while the Spearman correlation evaluates rank-order consistency [37].

$$r_{pearson}={\displaystyle \frac{\sum (y_i-\overline{y})({\widehat{y}}_i-\overline{\widehat{y}})}{\sqrt{\sum {(y_i-\overline{y})}^2\sum {({\widehat{y}}_i-\overline{\widehat{y}})}^2}}}$$

$${\rho }_{spearman}=1-{\displaystyle \frac{6\sum d_i^2}{N(N^2-1)}}$$

3.4.7. Cohen’s Kappa

Cohen’s Kappa is a statistical measure of inter-rater agreement that adjusts for chance-level agreement [38].

$$\kappa ={\displaystyle \frac{p_o-p_e}{1-p_e}}$$

where $p_o$ is the observed agreement and $p_e$ is the expected agreement by chance.

In summary, this study employed a comprehensive suite of evaluation metrics to assess the performance of Large Language Models (LLMs) in Automated Essay Scoring. Metrics such as Exact Match, F1 Score, MAE, RMSE, and Average Absolute Deviation provided insights into the models’ scoring accuracy and error magnitude. Pearson and Spearman correlations assessed the alignment of predictions with human scoring trends, while Cohen’s Kappa offered a measure of inter-rater agreement beyond chance. Collectively, these metrics enabled a rigorous, multi-dimensional evaluation of model reliability, consistency, and alignment with human judgment - critical qualities for the effective integration of LLMs in educational assessment systems.

4. Experimental Results

This chapter presents the experimental setup and evaluation results for assessing the performance of various large language models (LLMs) in the task of Automated Essay Scoring (AES). The evaluation focuses on five models described in Section 3.2 using various prompting strategies described below to assess their effectiveness against human grader scores.

4.1. Prompt 1 Results

Table 4 presents the performance of all five open-source LLMs under Prompt 1 (rubric-aligned prompting), which explicitly embeds descriptive scoring criteria for each dimension of argumentative writing. The results reveal substantial variation across models, with DeepSeek-R1 7B and Mistral 8×7B emerging as the strongest performers. DeepSeek-R1 achieves the highest F1 Score (0.93), lowest MAE (0.593), and highest precision and recall (0.93 each), demonstrating balanced classification accuracy across score levels. Meanwhile, LLaMA 3.2 3B attains the strongest correlation with human graders, yielding the highest Pearson (0.863) and Spearman (0.831) coefficients—despite its smaller size—suggesting that rubric alignment enables even lightweight models to capture holistic scoring patterns. In contrast, Qwen2 7B exhibits notably weaker performance across all metrics (e.g., Pearson = 0.371, MAE = 1.000), indicating sensitivity to prompt formulation or domain alignment. Interestingly, Cohen’s k remains low overall (max = 0.16 for Mistral), reflecting the well-documented challenge of achieving strong inter-rater agreement in ordinal AES settings—even with explicit rubrics. Nonetheless, the strong correlations and F1 scores underscore that rubric-aligned prompting significantly enhances the reliability of open-source LLMs for automated scoring, particularly when paired with models like DeepSeek-R1 and Mistral.

Table 5 presents a qualitative comparison of model predictions against human-assigned scores for five randomly selected argumentative essays from the PERSUADE 2.0 dataset. All models were prompted using the rubric-aligned strategy (Prompt 1), which embeds explicit scoring criteria and requests a single numerical score. Each row shows the human holistic score (on a 1–6 scale) alongside the predictions generated by the five evaluated LLMs. This snapshot illustrates both alignment and divergence in model behavior: for instance, DeepSeek-R1 7B matches the human score in three of the five cases and never deviates by more than 2 points, while Mistral 8×7B shows greater variability (e.g., predicting a score of 3 for an essay rated 5 by humans). Such examples highlight how prompt design and model choice jointly influence scoring fidelity at the instance level.

4.2. Prompt 2 Results

Table 6 reports model performance under Prompt 2 (instruction-based prompting), which provides a clear grading directive and lists the six scoring dimensions but omits their detailed definitions. Compared to the rubric-aligned condition (Prompt 1), all models exhibit a noticeable drop in performance, confirming that mere mention of criteria—without explicit elaboration—is insufficient to guide accurate scoring. Nevertheless, consistent trends emerge: Qwen2.5 7B achieves the highest correlation with human judgments (Pearson = 0.526, Spearman = 0.542) and the best inter-rater agreement (Cohen’s k = 0.38), while DeepSeek-R1 7B leads in classification-oriented metrics such as F1 Score (0.55) and Exact Match (0.57). The generally low k values (0.31–0.38) indicate only fair agreement with human graders—a known limitation in AES when models lack fine-grained guidance. Notably, even the best-performing models under Prompt 2 fall short of the correlations observed with Prompt 1 (e.g., Pearson > 0.83), reinforcing the central finding that embedding explicit rubric language is far more effective than naming criteria alone. This gap underscores the sensitivity of open-source LLMs to prompt specificity, especially in complex, multi-dimensional assessment tasks.

Table 7 illustrates model behavior under Prompt 2 (instruction-based prompting) through a side-by-side comparison of human scores and predictions on five representative essays from PERSUADE 2.0. Unlike the rubric-aligned condition (Prompt 1), which embeds explicit scoring definitions, Prompt 2 only names the assessment criteria—resulting in less consistent and often less accurate predictions. For instance, on the high-scoring essay (“New software has been created...”, human score = 5), LLaMA 3.2 and Mistral 8×7B under-predict (3 and 2, respectively), while Qwen2.5 over-predicts (6). Similarly, for the lowest-scoring essay (“Cars—no driver...”, human score = 1), Qwen2.5 assigns a notably inflated score of 3, suggesting a failure to recognize poorly developed arguments without explicit rubric guidance. DeepSeek-R1 7B demonstrates relatively stable performance, matching the human score in two cases and deviating by at most 1 point in others—consistent with its strong quantitative results in Table 6. Overall, this qualitative snapshot reinforces a key trend: without detailed rubric descriptors, even capable open-source LLMs struggle to replicate human grading logic, particularly at score extremes. This contrasts sharply with Prompt 1, where the same models showed markedly better alignment—highlighting that prompt specificity, not just model capacity, governs AES reliability.

4.3. Prompt 3 Results

Table 8 presents model performance under Prompt 3 (minimal instruction-based prompting), which provides only a concise directive to grade the essay based on key argumentative dimensions—without assigning a role (e.g., “teacher”) or defining the criteria. Surprisingly, this stripped-down approach yields modest but consistent improvements over Prompt 2 (the role-assigned instruction variant) across several metrics, particularly in inter-rater agreement and correlation. DeepSeek-R1 7B again emerges as the strongest performer, achieving the highest F1 Score (0.63), Cohen’s k (0.49), and Precision (0.59), indicating robust classification capability even with minimal guidance. Qwen2.5 7B leads in correlation metrics (Pearson = 0.582, Spearman = 0.577) and achieves the lowest MAE (0.598), suggesting it is especially adept at capturing the ordinal structure of human scores when not constrained by potentially misleading role framing. Notably, all models show higher k values under Prompt 3 (0.43–0.49) than under Prompt 2 (0.31–0.38), implying that the addition of a “teacher” persona in Prompt 2 may have introduced unintended biases or inconsistencies—particularly for models less attuned to role-based reasoning. Although performance under Prompt 3 still lags significantly behind the rubric-aligned condition (Prompt 1), these results reveal a nuanced insight: for some open-source LLMs, simplicity in prompting can outperform overly prescriptive or role-heavy instructions, especially when the role is not grounded in explicit task definitions. This underscores that effective prompt engineering for AES is not merely about adding more information, but about providing the right kind of guidance.

Table 9 shows model predictions versus human scores for five representative essays under Prompt 3 (minimal instruction-based prompting)—a bare-bones directive that names scoring dimensions but provides no role assignment or rubric definitions. Despite its simplicity, this prompt yields more calibrated predictions than Prompt 2 (the role-enhanced variant), particularly for mid- to high-scoring essays. For instance, on the top-rated essay (“New software has been created...”, human = 5), three models (DeepSeek-R1, Mistral, Qwen2) predict 5 or 4, showing reasonable fidelity—unlike Prompt 2, where the same essay triggered more extreme errors (e.g., Mistral scoring it 2). Similarly, DeepSeek-R1 7B matches the human score in two cases and stays within ±1 point in all others, aligning with its strong quantitative performance in Table 8. However, challenges persist at the score extremes: Qwen2 7B over-predicts the lowest-quality essay (“Cars—no driver...”, human = 1) as a 3, while LLaMA 3.2 under-predicts the solid mid-range essay (“When you need advice?...”, human = 4) as a 2. Notably, no model consistently outperforms others across all items, suggesting that minimal prompting reduces—but does not eliminate—model idiosyncrasies. Crucially, even in this simplified setting, performance remains substantially below Prompt 1, reinforcing that explicit rubric integration provides irreplaceable scaffolding for reliable AES. These observations support a nuanced conclusion: while stripping away unnecessary role framing (as in Prompt 3) can improve stability over more verbose but under-specified instructions (Prompt 2), detailed criteria remain essential for human-aligned scoring.

4.4. The Primacy of Prompt Design in AES

Across all evaluated models, prompt design exerts a decisive influence on Automated Essay Scoring (AES) performance—often outweighing differences in model architecture or size. Rubric-aligned prompting (Prompt 1), which embeds explicit, human-readable definitions of each scoring dimension, consistently yields the strongest alignment with human judgments, achieving Pearson correlations above 0.83 and F1 scores near 0.93 for top models. In contrast, both instruction-based variants—whether enhanced with a grading role (Prompt 2) or stripped to a minimal directive (Prompt 3)—produce markedly lower agreement, with correlations generally below 0.60 and Cohen’s k in the “fair” range (0.31–0.49). Interestingly, the minimal prompt (Prompt 3) often outperforms the role-assigned version (Prompt 2), suggesting that role priming without substantive criteria may introduce noise or misalignment, particularly for models not fine-tuned for pedagogical tasks. These findings collectively demonstrate that the most effective path to reliable, transparent, and equitable AES with open-source LLMs is not model scaling, but prompt precision: providing models with the same descriptive rubric used by human graders enables even modest-sized LLMs to emulate expert scoring behavior with surprising fidelity.

5. Conclusion and Future Work

This work demonstrates that prompt engineering—not model scale or proprietary access—is the critical lever for achieving reliable, human-aligned Automated Essay Scoring (AES) with open-source Large Language Models. Through a systematic evaluation of five contemporary open-weight LLMs—LLaMA 3.2 3B, DeepSeek-R1 7B, Mistral 8×7B, Qwen2 7B, and Qwen2.5 7B—on the PERSUADE 2.0 dataset, we show that embedding explicit rubric descriptors directly into the prompt (rubric-aligned prompting) dramatically improves scoring accuracy, correlation with human judgments, and inter-rater agreement, consistently outperforming both role-based and minimal instruction strategies. Notably, lightweight models like DeepSeek-R1 7B and Mistral 8×7B achieve Pearson correlations above 0.83 and F1 scores near 0.93 under this approach—performance levels that rival those reported for much larger, closed commercial systems.

Our findings carry significant implications. First, they challenge the assumption that AES requires massive, black-box models, offering a viable pathway toward transparent, auditable, and locally deployable assessment tools—a crucial step toward equitable educational technology. Second, they reveal that not all prompting is equally effective: simply naming scoring criteria or assigning a grading persona is insufficient; explicit, human-readable rubric integration is essential. Surprisingly, we also find that overly verbose or role-heavy instructions can underperform minimalist directives, underscoring that prompt design must prioritize semantic precision over surface-level scaffolding.

Several promising avenues emerge from this work. First, adaptive or dynamic prompting—where the prompt is tailored based on essay characteristics (e.g., length, topic, or predicted difficulty)—could further refine scoring fidelity. Second, enabling models to generate justifications alongside scores (e.g., via constrained Chain-of-Thought) would enhance interpretability and support formative feedback, moving beyond mere assessment to pedagogical utility. Third, evaluating prompt robustness across diverse student populations, grade levels, and writing genres—including non-argumentative tasks—is essential for real-world deployment. Finally, extending this framework to multilingual and low-resource educational contexts could democratize access to high-quality automated feedback, especially where human grading capacity is limited.

These results position prompt engineering as a low-cost, high-impact intervention for educators, developers, and researchers seeking to adopt open-source LLMs in real-world assessment settings. Nevertheless, this study establishes a clear principle: to make LLMs grade like humans, we must first teach them the rules humans use.