BERT-Enhanced HyperGAT with Siamese Networks and Reference Answer Set for Automated Short-Answer Scoring

1. Introduction

In everyday classroom practice, teachers spend countless hours grading short-answer questions, a task that is not only time-consuming but also prone to inconsistency due to fatigue or subjective judgment. This burden limits teachers’ ability to provide timely, personalized feedback, which is critical for student learning. While automated short-answer grading (ASAG) offers a promising solution, existing methods struggle with three practical challenges that hinder their adoption in real classrooms. First, the diversity of student answers. The same correct idea can be expressed in countless ways, yet most automated scoring models rely on limited reference answers, leading to frequent misclassifications (Tan et al., 2022). For example, in a biology short-answer question, some students may express “mRNA leaves the nucleus” as “mRNA goes out of the nucleus”—semantically correct but easily flagged as incorrect by keyword-matching systems. Second, the ambiguity of score boundaries. Teachers often encounter partially correct or creatively worded answers that fall into the “gray area” between score levels, making precise classification difficult. Third, imbalanced data distribution. In real classrooms, high-scoring answers are relatively rare, resulting in training data that biases models toward lower scores and reduces their ability to recognize excellent responses (Kaldaras et al., 2022). These challenges not only reduce scoring accuracy but also undermine teachers’ trust in automated grading systems.

To address these challenges, researchers have explored various approaches to improving ASAG. The following sections review prior work on text classification methods, reference answer set construction, and data augmentation, leading to the formulation of our research hypotheses.

1.1. Text Classification Methods and Their Educational Limitations

In automated scoring research, text classification serves as the core technical framework. Early studies employed machine learning methods that relied on manually engineered features to build scoring models (Kumar et al., 2019; Saha et al., 2018; Sultan et al., 2016). These approaches are not only labor-intensive but also struggle to capture complex semantic relationships in student answers. Deep learning techniques, such as convolutional neural networks (CNNs) and long short-term memory networks (LSTMs), reduce the need for manual feature extraction but often fail to account for global semantic interactions (Alikaniotis et al., 2016; Huang et al., 2018; Riordan et al., 2017).

Recent studies have introduced graph convolutional networks (GCNs) to model global semantic structures in student answers (Tan et al., 2023). However, GCNs assume uniform importance among adjacent nodes, neglecting variations in how different words contribute to meaning (Gilmer et al., 2017). More importantly, traditional graphs are limited to pairwise connections, making it difficult to capture complex multi-word interactions in student answers. Hypergraphs address this limitation by allowing a single edge (hyperedge) to connect any number of nodes, enabling more effective modeling of high-order semantic relationships (Feng et al., 2019; Kim et al., 2020). Ding et al. (2020) proposed the HyperGraph Attention Network (HyperGAT), which achieved superior performance on text classification tasks compared to traditional methods, though its accuracy on the Ohsumed medical dataset remained modest at 0.69, indicating room for improvement.

Meanwhile, the emergence of pre-trained language models such as BERT has brought significant advances in text understanding. Unlike static word embeddings, BERT generates dynamic, context-dependent representations, effectively resolving polysemy (Devlin et al., 2019). However, BERT alone struggles to capture the structured semantic relationships between student answers and reference answers. Therefore, Study 1 proposes integrating BERT with HyperGAT to develop a HyperGAT-BERT framework, with the goal of improving text classification accuracy. We hypothesize that combining BERT’s contextual awareness with HyperGAT’s ability to model high-order relationships will better represent student answer semantics and thereby enhance automated scoring performance.

1.2. Reference Answer Sets

The coverage of reference answers is a critical factor affecting ASAG accuracy (Burrows et al., 2015; Valenti et al., 2003). In practice, the diversity of student answers often exceeds the coverage of predefined reference sets, leading to many correct answers being incorrectly flagged as wrong. This problem is particularly acute for open-ended questions (Tan et al., 2022).

To address this issue, researchers have explored ways to construct more comprehensive reference answer sets. Lan et al. (2015) proposed clustering student answers and having experts select representative examples from each cluster, streamlining the scoring process. Marvaniya et al. (2018) further clustered answers by score level, selecting representative answers from each level. Specifically, reference answer set construction involves two key steps (Tan et al., 2019): (1) clustering analysis to identify distinct answer patterns, with each cluster representing a unique response type; and (2) selecting one or more prototypical answers from each cluster to compile the reference answer set, as illustrated in Figure 1.

Furthermore, Siamese Neural Networks (SNNs), initially proposed for tasks such as face recognition, signature verification, and similarity learning (Bromley et al., 1993), have recently been adapted to educational applications, particularly in automated scoring of constructed-response items. The architecture comprises twin subnetworks that share identical parameters (e.g., weights and biases) but process two distinct input samples. As shown in Figure 2, this design enables SNNs to effectively quantify similarity or dissimilarity between paired inputs through comparative analysis.

Based on this analysis, Study 2 proposes constructing a Reference Answer Set (RAS) via clustering and integrating it with SNNs to form the HyperGAT-BERT-RAS framework. We hypothesize that the RAS will cover more diverse answer patterns, while the SNN will more precisely compute semantic similarity between student answers and reference answers. Ablation experiments will validate the contribution of RAS to scoring accuracy.

1.3. Data Augmentation

In real classrooms, the distribution of student scores is often skewed, most students cluster around middle or low scores, while high-scoring answers are relatively rare. This imbalance biases automated scoring models toward majority classes and reduces their ability to recognize minority classes (Kaldaras et al., 2022). While balanced data distribution is critical for model performance (Shorten & Khoshgoftaar, 2019), collecting more high-scoring answers in classroom settings is often impractical. Data augmentation thus emerges as a viable alternative (Cochran et al., 2022).

Early NLP data augmentation methods included synonym replacement, back-translation, and random deletion (Wei & Zou, 2019; Yu et al., 2018). However, these approaches risk semantic distortion or over-reliance on rule-based heuristics. The advent of generative language models has opened new avenues for data augmentation (Bayer et al., 2023). Notably, GPT-4, with its powerful language understanding and generation capabilities, has demonstrated exceptional performance in text augmentation tasks (OpenAI, 2022). Studies have shown that GPT-4-generated synthetic data can effectively improve model performance in low-resource classification tasks (Dai et al., 2023; Ubani et al., 2023; Møller et al., 2023). Cochran et al. (2023) further demonstrated that ChatGPT-augmented data improves automated essay scoring accuracy in low-data regimes.

Therefore, the latter phase of Study 2 employs GPT-4 for data augmentation, generating student answers for score levels 2 and 3 in the ASAP-5 dataset, which are underrepresented. By balancing data distribution, we aim to further improve the accuracy of HyperGAT-BERT-RAS for automated short-answer scoring.

1.4. Novel Contributions of This Work

Based on the above analysis, this study makes the following four contributions: (1) Technical integration. We are the first to jointly optimize BERT’s contextualized representations with HyperGAT’s hypergraph structure for ASAG, addressing the challenge of precise semantic matching. (2) RAS-guided Siamese scoring. Unlike prior work that uses RAS only for answer selection, we embed RAS directly into a Siamese architecture to compute fine-grained similarity scores. (3) Controlled LLM augmentation. We provide the systematic evaluation of GPT-4 augmentation for ASAG, including prompt design, quality assessment, and leakage prevention. (4) Educational significance. By improving scoring accuracy and consistency, this study aims to reduce teacher grading burden, support formative assessment implementation, and provide a reliable tool for AI-assisted assessment in real classrooms.

2. HyperGAT-BERT Framework and Performance Validation

2.1. HyperGAT-BERT Architecture

The HyperGAT framework incorporates a text hypergraph encoding syntactic and semantic information, along with dual attention mechanisms at both node and hyperedge levels (Ding et al., 2020). In this study, the proposed HyperGAT-BERT similarly treats each word as a hypernode and models contextual syntactic and semantic relationships through hyperedges. Multi-relational hyperedges, specifically syntactic hyperedges (encoding word order) and semantic hyperedges (encoding meaning) are defined following the original methodology. After constructing these hyperedges, dual attention mechanisms are employed to capture high-order word interactions while emphasizing critical information at varying granularities during node representation learning. For each document, node representations across the hypergraph are computed post-attention, followed by a mean pooling operation to derive the text representation. The BERT model, pretrained on large-scale unlabeled corpora, generates text embeddings rich in semantic information, which can be directly transferred to downstream tasks (Devlin et al., 2019) . In this framework, text inputs are processed through BERT to obtain sequence representations, which are concatenated with HyperGAT-derived text representations (denoted as Concat). This fused document representation (Z) is then fed into a softmax layer for classification. The architecture is illustrated in Figure 3, where the green dashed box represents the original HyperGAT method (Ding et al., 2020), and the combined red-green components depict the BERT-enhanced HyperGAT-BERT framework.

2.2. Text Classification Performance Validation

2.2.1. Dataset

The Ohsumed corpus comprises medical literature abstracts sourced from the MEDLINE database, a critical repository curated by the U.S. National Library of Medicine (http://disi.unitn.it/moschitti/corpora.htm). This study utilizes 13,929 cardiovascular disease abstracts selected from the first 20,000 articles published in 1991. Each document in the dataset is associated with one or more category labels spanning 23 disease classes. To ensure label consistency, documents with multiple category assignments were excluded, resulting in a final subset of 7,400 single-label documents. In each experimental run, 80% of the training samples are randomly selected for model training, while the remaining 20% are reserved for validation. Table 1 summarizes the basic characteristics of the Ohsumed dataset.

2.2.2. Experimental Design

To investigate whether HyperGAT (Ding et al., 2020) and the proposed BERT-enhanced HyperGAT-BERT can better represent texts and improve text classification performance, this study conducted comprehensive experiments to evaluate model performance on text classification tasks. The HyperGAT-BERT model was compared against four categories of baseline models, including word embedding-based methods (SWEM), sequence-based methods (CNN-non-static), and graph-based methods (Text-level GNN and HyperGAT). Data analysis is conducted using Python.

The optimal hyperparameters were determined via grid search over the following ranges: embedding dimensions {128, 200, 300, 400}; learning rates {1e-4, 5e-4, 1e-3}; dropout rates {0.2, 0.3, 0.4}; batch sizes {8, 16, 32}. The validation set (10% of the training data) was used for model selection. The best configuration was identified as: embedding dimension = 300, learning rate = 0.001, dropout = 0.3, batch size = 16. Sensitivity analysis for the embedding dimension is presented in Figure 5. For all models, we employed early stopping with a patience of 5 epochs, meaning that training would terminate if validation performance did not improve for five consecutive epochs. By default, the maximum number of epochs is set to 100. The model will run for 100 epochs or stop early if the performance on validation samples does not improve for 5 consecutive epochs. The Adam optimizer was used with weight decay set to 1e-6, and L2 regularization was applied with a coefficient of 1e-6. The dropout rate is set to 0.3 to prevent overfitting.

The evaluation metrics are Precision (P), Accuracy (ACC), F1-score (F1). Precision (P) evaluates the proportion of correctly predicted positive samples among those identified as positive. Accuracy (ACC) represents the proportion of correctly classified samples among all samples. The F1-score balances precision and recall. Although recall (R) is a conventional metric for classification tasks, we do not report it separately in our results. The F1-score, as the harmonic mean of precision and recall, already captures the balance between the two. Reporting recall alone would provide redundant information without additional insight into model performance for ASAG. Therefore, only P, ACC, F1 are presented in the following result tables. The formulas are as follows (1), (2), and (3):

$$\mathrm{P}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}}$$

(1)

$$\mathrm{F}1={\displaystyle \frac{2\times \mathrm{P}\times \mathrm{R}}{\mathrm{P}+\mathrm{R}}}$$

(2)

$$ACC={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(3)

where TP (True Positive), TN (True Negative), FP (False Positive), and FN (False Negative) are defined as follows: TP, the number of samples correctly predicted as the positive class. TN, the number of samples correctly predicted as the negative class. FP, the number of samples incorrectly predicted as the positive class. FN, the number of samples incorrectly predicted as the negative class.

2.2.3. Experimental Results

As shown in Table 2, graph-based methods (Text-level GNN and HyperGAT) achieved superior performance compared to word embedding-based and sequence-based methods. This result indicates that graph structures can better capture long-distance word interactions, thereby improving text classification performance. Compared to the baseline models, HyperGAT-BERT achieved the highest accuracy (0.7317) among all compared models, surpassing SWEM by 10.03%, CNN-non-static by 14.74%, Text-level GNN by 3.72%, and HyperGAT by 3.27%. This improvement demonstrates that integrating BERT’s rich semantic representations enhances HyperGAT’s text representation capability, which is significant for text classification tasks. To further validate the effectiveness of HyperGAT-BERT, we compared it with results reported in existing literature on the same dataset. Lv et al. (2024) proposed the RB-GAT model (combining RoBERTa-BiGRU with graph attention networks), which achieved 71.48% accuracy on the Ohsumed dataset. In comparison, our proposed HyperGAT-BERT model achieved 73.17% accuracy, surpassing this existing method. This comparison further confirms the effectiveness of jointly optimizing HyperGAT with BERT for medical text classification tasks.

Figure 4 presents the confusion matrices and text embedding visualizations for both HyperGAT and HyperGAT-BERT. Analysis of the confusion matrices reveals that HyperGAT-BERT achieves higher classification accuracy for most categories, with comparable performance on the remaining classes. In the embedding visualizations (node colors correspond to 23 class labels), HyperGAT-BERT exhibits tighter intra-class clustering and improved inter-class separability, further validating its superior representation capability.

Figure 5 illustrates the performance of the HyperGAT-BERT model on the Ohsumed dataset with varying embedding dimensions. Notably, HyperGAT-BERT achieves optimal performance when the embedding dimension is set to 300. This suggests that smaller dimensions may lead to insufficient expressive capacity, while excessively large dimensions risk overfitting.

3. HyperGAT-BERT Integrating Siamese Neural Networks and Reference Answer Sets and Its Application in Automatic Short-Answer Scoring

The HyperGAT-BERT model, which integrates the rich semantic information from BERT, outperforms other models in its ability to better represent texts, thereby improving text classification results. Therefore, the HyperGAT-BERT model is selected for continued research. This section aims to explore whether the automatic short-answer scoring method based on Siamese Neural Networks and reference answer sets, utilizing a hypergraph attention network, can achieve the expected good results. Additionally, ablation experiments are conducted to verify whether different modules within the HyperGAT-BERT model can effectively enhance the scoring performance for short answers.

3.1. Construction of HyperGAT-BERT Integrating Siamese Neural Networks and Reference Answer Sets

3.1.1. Dataset

We utilize Data Set #5(ASAP-5) from the Hewlett Foundation-supported 10th-grade student responses in biology (https://www.kaggle.com/c/asap-sas/data). The specific details are presented in Table 3.

The question prompt is: “Starting with mRNA leaving the nucleus, list and describe four major steps involved in protein synthesis.”

The scoring rubric is as follows: 3 points for satisfying all four key elements, 2 points for satisfying three key elements, 1 point for satisfying one or two key elements, and 0 points for not meeting the above requirements. There are a total of eight key elements in the response: (1) mRNA exits nucleus via nuclear pore. (2) mRNA travels through the cytoplasm to the ribosome or enters the rough endoplasmic reticulum. (3) mRNA bases are read in triplets called codons (by rRNA). (4) tRNA carrying the complementary (U=A, C+G) anticodon recognizes the complementary codon of the mRNA. (5) The corresponding amino acids on the other end of the tRNA are bonded to adjacent tRNA’s amino acids. (6) A new corresponding amino acid is added to the tRNA. (7) Amino acids are linked together to make a protein beginning with a START codon in the P site (initiation). (8) Amino acids continue to be linked until a STOP codon is read on the mRNA in the A site (elongation and termination).

3.1.2. Construction of Reference Answer Set

The construction of the Reference Answer Set (RAS) involves aggregating different student answer situations into a unified set. The construction process of the reference answer set can be divided into two steps:

Step 1: Obtain possible student answer situations based on clustering, where each cluster obtained after clustering represents a type of student answer situation.

1. Data Preprocessing: Load text data from files, split each document into sentences, and perform a series of text cleaning and preprocessing operations, including tokenization, lemmatization, and removal of stop words, to generate a doc content list.

2. Label Processing: Load label information associated with the text data and split the data into training and testing sets. Simultaneously, establish a label dictionary named labels dic to map labels to numerical values and calculate the count of each label.

3. Answer Set Selection: Based on the labels, select texts with a specific label value equal to 3 and store them in doc answer list original as the answer set.

4. Text Embedding: Utilize the BERT model to embed the texts in doc answer list original, converting each text into a BERT vector representation.

5. Clustering Analysis: Conduct clustering analysis on the BERT-embedded texts using the K-means clustering algorithm. Perform clustering within different ranges of cluster numbers (4, 6, 8, 10, 12, 14, 16, 18, 20) and calculate the silhouette coefficient for each scenario.

6. Selection of Optimal Number of Clusters: Based on the silhouette coefficients, select the number of clusters with the highest silhouette coefficient, indicating that the texts have better clustering performance under this number of clusters.

As shown in Figure 6, the silhouette coefficient peaks at k=4 (silhouette score = 0.67), compared to k=6 (0.61), k=8 (0.58), and k=10 (0.54). This indicates that four clusters best balance intra-cluster cohesion and inter-cluster separation. We therefore selected four representative answers to construct the RAS (Table 4). The choice of k=4 is also pedagogically interpretable: it corresponds to the four major steps of protein synthesis (initiation, elongation, termination, and translocation), aligning with the scoring rubric.

Step 2: Select the most representative answer from each cluster as the representative of that cluster to construct the RAS. Ultimately, four representative answers are chosen, each with a different focus. The specific content of these four categories of representative answers is detailed in Table 4.

3.1.3. Constructing HyperGAT-BERT-RAS

With the dataset prepared and the reference answer set constructed, we began building an automatic short-answer scoring model based on Siamese neural networks and the reference answer set, using a hypergraph attention network. The specific steps are as follows:

1. Read the corpus files from the ASAP-5 dataset and perform a series of text cleaning and preprocessing operations, including tokenization, lemmatization, stop-word removal, etc. A validation set is then split from the training set at a ratio of 10%.

2. Further process the training data, validation set, and test set by storing different parts of the input data and document answer lists as NumPy arrays, including inputs, targets, and text, which represent vector representations, labels, and initial text, respectively. Data batches are generated for training or evaluation, with each batch containing 8 texts (batch=8). To ensure consistent training for both student texts and answer texts, each batch concatenates student texts with answer texts, meaning that 12 texts are trained together each time. Additionally, node information (items) is generated to represent all nodes in the document. Edge information is generated by iteratively processing each document’s sentences and semantic information, including lists of node and edge transpose matrices (HT: each element corresponds to the node and edge relationships of a document) and adjacency matrices (adj), both commonly used to represent the graph structure of a document. Finally, node masks are generated for each document, related to node information, to indicate which nodes are valid, which are padding values, or unused, maintaining dimensional consistency of the data, especially when different documents have varying numbers of nodes.

3. Use the train data obtained in the previous step as input to train the model, iterating over each batch. To calculate the model’s score or predicted output, the input is first passed to an embedding layer to obtain embedded representations (hidden). Next, the embedded representations (hidden) and node information (HT) are passed to the hypergraph attention layer, the hypergraph attention mechanism, allowing the model to capture complex relationships in graph data. These operations help the model extract useful information from hypergraph data, yielding intermediate representation outputs (seq hidden) for subsequent tasks.

The intermediate representation (seq hidden) is then multiplied by the node masks (node masks), setting unnecessary nodes to zero and filtering out invalid nodes. Subsequently, graph embeddings for the nodes are calculated and normalized. At this point, the graph embeddings include the graph embeddings for 8 student answer texts and 4 reference answer set texts per batch (b). The entire embedding is sliced to obtain embeddings for student answer texts (b0) and reference answer set texts (b1) separately.

Meanwhile, the BERT model is used to process text inputs (text inputs) and answer texts, embedding them into text emb and answer emb. The hypergraph embeddings and BERT embeddings of both texts are concatenated, and cosine similarity is used to calculate the similarity between the student text and the 4 reference answers, with the highest similarity (Smax) selected as the similarity between the student’s answer and the reference answer. Additionally, the similarity (Smax) between the student’s answer and the reference answer is concatenated with the hypergraph-embedded student answer text (b0) as the final text representation (Z).

4. The obtained document representation (Z) is fed into a Softmax layer for text classification, ultimately producing the model’s predicted output. The structure of the HyperGAT-BERT-RAS model is shown in Figure 7.

3.2. Automated Short-Answer Scoring Validation

This section aims to validate the effectiveness of the HyperGAT-BERT-RAS method based on Siamese Neural Networks and Reference Answer Sets for automated short-answer grading tasks. The contributions of each module are evaluated through ablation experiments, and class imbalance is addressed via GPT-4 data augmentation.

3.2.1. Ablation Experiment Design

To systematically evaluate the contributions of individual components in the HyperGAT-BERT-RAS framework, three model variants were designed and tested on the ASAP-5 dataset:

w/o LDA: HyperGAT-BERT-RAS without semantic hyperedges.

w/o BERT: HyperGAT-BERT-RAS without BERT embeddings.

w/o RAS: HyperGAT-BERT-RAS without the Reference Answer Set (RAS).

Three primary metrics were used for evaluation: Accuracy (ACC), Precision (P), and F1-score (F1). For the data augmentation experiment, Quadratic Weighted Kappa (QWK) was additionally employed to assess the agreement between predicted scores and actual scores.

3.2.2. Ablation Experiment Results

Table 5 presents the ablation results. The full HyperGAT-BERT-RAS model achieved optimal performance on the test set, with the highest accuracy (0.7991), precision (0.8227), and F1-score (0.7956), demonstrating its superiority in automated short-answer grading.

The ablation results validate the contributions of the different modules constructed in this study within HyperGAT-BERT-RAS. Among them, the construction of RAS had the most significant impact on model performance. The model without RAS had the lowest performance among the four variants (F1 decreased by 4.13%), indicating that RAS is the most crucial module. Models without LDA semantic hyperedges or BERT embeddings also showed performance degradation. Specifically, removing LDA semantic hyperedges had a relatively smaller impact on model precision, and its performance was better than that of the w/o BERT and w/o RAS models, suggesting that LDA semantic hyperedges contribute to model performance on this dataset, albeit to a lesser extent than the other two components.

3.2.3. GPT-4 Data Augmentation

In the ASAP-5 dataset, the numbers of students scoring 0, 1, 2, and 3 points were 1,853, 429, 64, and 47, respectively, exhibiting a clear skewed distribution. This imbalanced distribution increases the difficulty for automated grading models to learn, thereby affecting their performance. Although the ablation experiment results show that the HyperGAT-BERT-RAS method achieved good performance (ACC = 0.7991, F1 = 0.7956), this inevitably came at the cost of accuracy for minority categories.

Given the small number of students scoring 2 and 3 points, GPT-4 was employed to generate student answers corresponding to these scores based on relevant information such as question stems, key scoring points, and grading requirements (Ubani et al., 2023). The augmented dataset was named E-ASAP-5, and its score distribution is shown in Table 6. Compared to ASAP-5, E-ASAP-5 has significantly more data for scores 2 and 3.

Prompts must be specifically designed for different questions and target scores. This study adopted two data augmentation strategies. The first was semantic augmentation based on existing student answers: for existing student answers in the original dataset, prompts were designed to instruct GPT-4 to generate synonymous variants that are semantically similar but diverse in expression. The second strategy involved training GPT-4 to generate student answers for specific scores: to address the scarcity of samples for scores 2 and 3 in the original dataset, prompts were designed based on question stems, key scoring points, and scoring criteria to instruct GPT-4 to generate student answers that meet the corresponding score standards. Manual sampling analysis confirmed that the text generated by both strategies achieved the expected results. The variants generated by Strategy 1 fully preserved the key steps of the original answers while exhibiting diversity in vocabulary choice and sentence structure. The answers generated by Strategy 2 were validated through GPT-4’s self-explanation, clearly containing the required number of key elements for the corresponding scores and strictly aligning with the scoring criteria. Furthermore, the generated text adopted a language style close to real student answers, avoiding overly perfect or textbook-like expressions, demonstrating the effectiveness of the data augmentation.

3.2.4. Data Augmentation Effectiveness

The experimental results after data augmentation (on the test set) are shown in Table 7. Using the augmented dataset E-ASAP-5, all metrics improved significantly compared to using the original data. Accuracy increased from 0.7991 to 0.8257, F1-score increased from 0.7956 to 0.8344, and QWK increased from 0.5841 to 0.8797, representing a substantial improvement from moderate agreement to near-perfect agreement. This indicates that data augmentation effectively enhances the model’s ability to recognize minority classes.

Figure 8 presents the confusion matrix and text embedding visualization for HyperGAT-BERT-RAS on E-ASAP-5. From the confusion matrix, it can be observed that classification accuracy for all categories improved after data augmentation. The embedding visualization shows clearer inter-class boundaries and tighter intra-class clustering for the augmented dataset, further validating the effectiveness of data augmentation.

3.2.5. Error Analysis

Based on the confusion matrix for the E-ASAP-5 test set shown in Figure 8, an error analysis of the HyperGAT-BERT-RAS model was conducted. The test set contained a total of 631 samples. From the confusion matrix, it can be observed that misclassifications primarily occurred between adjacent score categories: 47 instances of score 0 were misclassified as score 1 (13.70% of score 0 samples), and 14 instances of score 2 were misclassified as score 3 (11.97% of score 2 samples). In contrast, cross-level misclassifications (e.g., score 0 misclassified as score 2 or 3) accounted for only 8 instances (2.33% of score 0 samples). This pattern indicates that the model has good ordinal consistency, with scoring errors mainly limited to adjacent score levels.

The classification accuracy for score 1 was relatively lower (74.42%), with misclassifications toward score 0 (12 instances) and toward scores 2 or 3 (10 instances), indicating that boundary categories present challenges in semantic discrimination. Score 2 had similar numbers of misclassifications toward score 1 (8 instances) and score 3 (14 instances), suggesting that it occupies a semantic transition position between scores 1 and 3.

Based on the above observations, misclassifications can be categorized into three types: (1) boundary ambiguity (approximately 60%), occurring between adjacent scores; (2) semantic similarity (approximately 25%), arising from expression similarities between student answers and incorrect answers; and (3) unconventional expression (approximately 15%), stemming from novel but correct answers that do not adequately match the RAS prototypes. Future work should focus on three aspects: strengthening ordinal relationship modeling, expanding the reference answer set, and enhancing boundary samples.

4. Discussion

This study sought to address three major challenges in ASAG: semantic matching ambiguity, limited coverage of reference answers, and imbalanced data distribution. This study consisted of two research phases: text classification optimization and RAS-enhanced scoring with GPT-4 augmentation, from which meaningful results were obtained to advance ASAG performance.

4.1. Integrating BERT Enhances Text Classification Performance of the HyperGAT Method

To alleviate the semantic matching issue in automatic scoring of subjective questions, the study optimized a text classification model by integrating BERT into the HyperGAT framework, resulting in the HyperGAT-BERT method. The results indicate that HyperGAT-BERT outperforms HyperGAT across various metrics, demonstrating that combining BERT with graph neural networks yields a more robust model. When the pre-trained representation model BERT is combined with the graph neural network-based HyperGAT model (Ding et al., 2020), this integration leverages both BERT’s contextual awareness and HyperGAT’s strengths in modeling complex relationships, thereby improving text classification accuracy and F1 scores. This finding indicates that integrating language representation models with graph neural networks can endow the new model with richer contextual semantic information, significantly enhancing its text comprehension capabilities and demonstrating superior performance in relation extraction tasks (Vashishth et al., 2020; Zhang et al., 2018). From an educational perspective, improved text classification accuracy means fewer mis-scored student answers, which directly enhances teachers’ trust in automated scoring systems and encourages their adoption in daily classroom practice.

4.2. RAS is a Critical Factor Influencing the Accuracy of Automatic Scoring in HyperGAT-BERT

In practice, the coverage of reference answers is a pivotal factor in the scoring of subjective questions. Study Two constructed a HyperGAT-BERT model with a reference answer set (HyperGAT-BERT-RAS) based on SNNs and answer clustering, and conducted ablation experiments using the ASAP-5 dataset. The results demonstrated that each module in HyperGAT-BERT-RAS collectively enhanced the performance of automatic scoring, with the most significant decline observed in model performance upon the removal of RAS (F1 decreased by 4.13%), indicating its crucial role in automatic scoring of subjective questions (Tan et al., 2022). This finding aligns with existing research (Lan et al., 2015; Marvaniya et al., 2018) and resonates with the matching mechanism of SNNs (Bromley et al., 1993), which also emphasizes the importance of learning similarities and differences for pattern recognition. Additionally, the absence of BERT embeddings led to a decline in various metrics, further validating the significance of rich text representation information for automatic scoring of subjective questions and suggesting that rich textual information is an effective approach to addressing the challenge of semantic matching between student answers and reference answers (Tan et al., 2022). For classroom teachers, a well-constructed RAS reduces the need to manually review a wide range of student answers. By covering diverse answer patterns, the RAS enables the system to correctly score creative but correct responses, which are often penalized by traditional keyword-matching methods.

4.3. Data Augmentation is an Effective Solution to Address Imbalanced Student Answer Distribution

Balanced class distribution is crucial for model training, as imbalanced data can lead the model to favor majority classes while neglecting minority ones (He & Garcia, 2009). In the field of NLP, data augmentation is recognized as an effective method to enhance model generalization, particularly when training data is limited (Shorten & Khoshgoftaar, 2019). In the ASAP-5 dataset, student answer distribution exhibits a skewed pattern. To mitigate the impact of this imbalanced distribution on automatic scoring, Study Two included an additional data augmentation experiment to explore the feasibility of enhancing student answers using GPT-4’s powerful language generation capabilities to further improve automatic scoring effectiveness. The results indicate that data augmentation significantly enhances the model’s learning ability for minority classes, thereby improving overall scoring accuracy. The strategy of using GPT-4 to generate data in the experiment benefits from GPT-4’s advanced language understanding and generation capabilities (Brown et al., 2020). This approach allows for effective simulation of real student answers, providing more diverse training data, which results in more accurate model scoring and clearer classification boundaries, as visually evident in Figure 8’s text embedding visualization and confusion matrix. Data augmentation using GPT-4 addresses a practical bottleneck in classroom assessment: the scarcity of high-scoring student answers. By generating realistic high-scoring samples, the model learns to recognize excellent responses that occur infrequently in real classrooms, leading to fairer assessment outcomes for high-achieving students.

4.4. Implications from Error Analysis

The error analysis reveals two core misclassification patterns: adjacent-category confusion and boundary-category difficulty. The rates of score 0→1 and score 2→3 misclassifications were 13.70% and 11.97%, respectively, while cross-level errors accounted for only 2.33%. This indicates that the model has good ordinal consistency, with scoring errors mainly limited to adjacent levels.

Adjacent-category confusion reflects the inherent ambiguity of category boundaries in scoring tasks. Responses that fall between two score levels often contain both correct and incorrect elements, making precise classification difficult (Liu et al., 2018). This issue is particularly prominent for score 1 (recall: 74.42%), which semantically lies between scores 0 and 2. To address this, ordinal loss functions can be introduced to impose smaller penalties on adjacent errors and larger penalties on cross-level errors.

The near-equal misclassification of score 2 toward scores 1 and 3 (6.84% vs. 11.97%) suggests that this category occupies a semantic transition position. This relates directly to the coverage of the Reference Answer Set (RAS). The current RAS constructed solely from full-score answers insufficiently covers score 2 variants. Expanding RAS to include representative answers from all score levels is expected to reduce misclassifications (Marvaniya et al., 2018; Tan et al., 2022).

The error patterns identified have practical implications for classroom use. Adjacent-category errors (e.g., 0 vs. 1, 2 vs. 3) are pedagogically acceptable in formative assessment, where the goal is to identify students who need additional support rather than to assign precise final grades. Teachers can use the model’s scores as a screening tool, manually reviewing only boundary cases. This hybrid approach, which uses automated scoring for clear cases and human review for ambiguous ones, balances efficiency with accuracy, making AI-assisted assessment more feasible in real classrooms. In summary, the error analysis validates the model’s ordinal consistency and identifies three improvement directions: (1) introducing ordinal loss functions, (2) expanding RAS coverage, and (3) performing targeted data augmentation for boundary categories.

4.5. Research Limitations

Although this paper conducts a relatively systematic and in-depth exploration of the three key issues currently facing automatic scoring of subjective questions, there are still some limitations. Firstly, in terms of model optimization, this paper only explores the integration of BERT and HyperGAT and does not systematically compare it with more pre-trained models (e.g., RoBERTa, SciBERT). Future research can further explore the fusion of graph neural networks with other techniques (such as attention mechanisms, transformer variants, etc.) and evaluate the generalization ability of the model through multi-perspective comparisons. Secondly, the RAS in this paper is selected from full-score answers, without fully considering student responses under different score levels. Future research can select representative answers from various score levels to further enhance the coverage of RAS for all possible response outcomes. Thirdly, although data augmentation improves the overall performance of the model, there is still room for improvement in the recognition of minority categories. Future research can explore more data augmentation techniques to further enhance the fairness and accuracy of automatic scoring. Finally, this study was only validated on the ASAP-5 biology dataset. Future research should test the model’s generalization ability across more subjects (e.g., history, literature) and more languages.

5. Conclusions

This study addresses the challenges in automatic scoring of subjective questions by constructing the HyperGAT-BERT and HyperGAT-BERT-RAS methods and performing data augmentation through GPT-4, aiming to enhance the effectiveness of automatic scoring of subjective questions. The following conclusions are drawn:

(1) The incorporation of BERT to enrich semantic information enables HyperGAT-BERT to better represent texts and improve text classification performance. On the Ohsumed dataset, HyperGAT-BERT achieved 73.17% accuracy, surpassing the baseline HyperGAT model by 3.27 percentage points, demonstrating the effectiveness of BERT’s semantic enhancement for text classification tasks.

(2) When applied to automatic scoring of subjective questions, HyperGAT-BERT-RAS outperforms the other three model variants in all metrics. Different modules within the model are effective, with RAS contributing the most (F1 decreased by 4.13% when removed), followed by BERT. On the ASAP-5 dataset, the full model achieved 79.91% accuracy and 79.56% F1-score.

(3) Data augmentation using GPT-4 for categories with fewer samples effectively addresses the problem of uneven answer distribution. After augmentation, accuracy increased to 82.57%, F1-score increased to 83.44%, and QWK significantly improved from 0.5841 to 0.8797. Practically, this means that high-achieving students, whose answers are often underrepresented in training data, are no longer systematically disadvantaged by the scoring model. The confusion matrix and error analysis further validate the effectiveness of data augmentation and reveal that misclassifications primarily occur between adjacent score categories, which is acceptable in formative assessment contexts where the goal is to identify students needing support rather than to assign precise final grades.

Future research should test the model’s generalization ability across more subjects and languages, incorporate representative answers from all score levels into the RAS, and explore lightweight techniques such as model distillation to reduce computational costs.