Automatic Assessment in Higher Education Using General-Purpose AI Tools: A Design-Based Research Approach

1. Introduction

Higher education studies offered as fully online programs are by their nature highly accessible by a wide spectrum of students, reduce costs and resources spent on mobility, strongly contributing to sustainability goals. However, significant challenges arise, especially in distant postgraduate programs, such as safeguarding academic integrity, ensuring result transparency, designing authentic assessments [1], providing personalized feedback [2], addressing the lack of immediacy and direct communication from the instructor [3], and managing the difficulty of evaluating massive volumes of data [4]. Student satisfaction, learning outcomes [5], and completion rates are influenced by the instructor's role and presence, as well as the establishment of clear assessment criteria [6]. In parallel, the rapid progress of AI technologies has greatly improved the efficiency of automatic assessment and feedback provision in digital environments. Distance learning postgraduate programs may thus rely on automated assessment to capture essential information regarding learning processes, knowledge levels, and learning difficulties, thereby enhancing quality, while saving human effort and financial resources. As Cope et al. [7] note, assessment is perhaps the most important area of opportunity that AI offers for transformative change in education. AI tools are promising for reducing the resources spent on assessment, teachers’ workload and stress [8] and at the same time offer powerful features such as the prediction of future student progress [9,10]. Moreover, AI tools offer support to students and improve their learning process, strengthening the students’ motivation, increasing their participation, interaction and learning interest, especially in online courses [11,12].

However, limitations are identified in the use of AI in educational contexts in terms of the accuracy, validity, and reliability of the results [13]. Their ease of use and the sense they create that they fully replicate human thinking, can lead to uncritical acceptance of the suggestions they provide [14]. At the same time, issues of plagiarism or inability to distinguish between real and artificial text are raised [15] and concerns are reported regarding privacy and protection of personal data [16,17,18]. Luckin and Cukurova [19] point out the absence of pedagogical approaches by the developers who shape the specifications of Automatic Assessment Systems (AAS), as well as the lack of transparency regarding how each algorithm responds. Previous research has shown that the evaluation of assignments with simple criteria of either linguistic characteristics or mathematical operations can be successfully managed by AI. However, the validity of the results in more complex tasks, such as university projects, needs further investigation [20]. Many existing automated scoring systems focus on specific topics or question types, thereby limiting their scope and flexibility [21], while the focus of most tools on specific domains and cognitive abilities limits their applicability to higher-level assignments [13].

Considering the impressive increase in the power of AI tools and their widespread use in complex cognitive problems, many solutions are offered nowadays to educators for performing automatic assessments, which can be divided in three main categories: (a) specialized Automatic Assessment Systems (AAS) or Automatic Essay Scoring systems (AES), available either as commercial products or results of research projects, (b) customised applications based on Large Language Models (LLMs), where a pretrained LLM is fine-tuned to the specific problem, using large volumes of training data, and (c) general-purpose AI platforms through a chat environment, using suitable prompts [22].The first two categories are problem-specific, have significant cost and rely on the availability of suitable training data. On the contrary, the use of a general-purpose everyday tools such as OpenAI’s ChatGPT to automatically assess student assignments is a highly appealing minimal cost solution. ChatGPT, utilizing advanced algorithms and machine learning, can offer a new perspective on assessment or act as a consultant to the educator, who will make the final decision [23]. Examples can be found in science, technology, writing, experiments, and social sciences [24]. The literature has confirmed that ChatGPT’s responses are often human-like, accurate, and productive but at the same time pinpoints limitations and pitfalls [25]. While ChatGPT demonstrates excellent broad conversational skills, there are inherent limitations when using it without adaptations in specialized domains such as automatic assessment of higher education assignments [26].

The effect of different prompting techniques on the ability of GPT-4 to automatically grade and provide feedback has been examined, exhibiting the importance of skillful prompting [27] and the requirement that the criteria have been predefined clearly and in detail [28]. Furthermore, the capabilities of AI tools are constantly improving and allow them to handle complex reasoning and solve demanding problems [29,30]. In their recent systematic literature review, Fianu et al [31] outline the terminologies related to AES and capture a shift from traditional machine learning to LLMs and their advances. They also compare the reliability of key AI modeling technologies, showing a progressive evolution in the field of automated evaluation with more sophisticated models and support for larger data. Elmassry et al [32] report a significant development in the operating techniques of the corresponding models is the introduction of the Transformers architecture consisting of encoders and decoders at 2017. These allow the model to examine the words that have weight and meaning in a sentence and to capture long-term dependencies in the data in a more efficient way than Convolutional Neural Networks (CNNs) and Recursive Neural Networks (RNNs). Well-known examples of Transformer-based models include pretrained Bidirectional Encoder Representations from Transformers (BERT), Generative Pretrained Transformer (GPT) and eXtreme Language Model Pretraining Network (XLNET). The spread of these models is wide due to their ability to adapt and model all data in all disciplines and cognitive domains. In the field of automatic evaluation, Longformers have also been added, which are a specialized form of Transformer, capable of processing large text sequences, addressing the previous length limitations in a feasible way.

A key parameter for evaluating automatic scoring systems is the datasets that researchers use to test available or new models in studies and pilot procedures. Well known datasets include the Automated Essay Grading Dataset (ASAP), ASAP++, which is an advanced form with the integration of additional features, TOFEL11, which concerns the specific area of Cambridge English, and others that are created for individual studies [32] In several experiments, AASs were compared with human evaluators to determine their reliability and their impact on the learning process, with controversial results. Reported problems include inconsistency between systems and instructor ratings, inconsistency between exercises for assessing two tasks of similar quality, differences in rater rigor and random variation in ratings of the same task under changing conditions [33]. An important limitation of the relevant efforts was that they were based on specific approaches and algorithms that were not capable of handling the level of complexity required for educational environments today. Messer et al [34] reported that disagreement rates were approaching 44% even on short-form questions, and this percentage increased as the questions became more complex and intricate. A sensitive area of automatic assessment that has been less studied is fairness leaving a research gap related to the investigation of fairness or potential bias of automatic assessments across subgroups of populations with different language [26].

An important dimension of AAS, particularly relevant to sustainability, are the positive implications for students. AI-assisted assessment and feedback encourages students to think critically and reflectively, cultivates creativity, develops problem-solving skills and effectively contributes to enhanced understanding of concepts. Tu et al [21] supported that on average, the improvement in the performance of students who used automatic assessment systems with the corresponding feedback improved by 24.9%, while problem-solving skills and critical thinking showed improvements of 25.2% and 25.8% respectively. A positive and promising finding was a strong positive correlation between student performance and the increased participation caused by the fact that students feel more active in the educational process, improving their skills. This is a strong argument that supports the positive impact which AAS using general-purpose AI tools may have on online higher education programs, not only on their sustainability related to cost and resources but also on their quality, attractiveness, and long-term success.

As regards the use of Generative AI as an AAS tool, it is established that nowadays it is more powerful, able to align with human way of thinking based on instructions, adapt to the context and function as an autonomous evaluation tool [35]. AI tools such as ChatGPT have the potential to perform automatic assessment in education and achieve accuracy and adaptability in their responses by leveraging their extensive knowledge base [36]. Several researchers confirmed that a general-purpose conversational model such as ChatGPT, after careful guidance with grading rubrics, task descriptions and instructions, aligned with human assessment criteria managed to match its grades with those of a teacher and could function as a stand-alone automatic grading tool, providing qualitative feedback [26,37,38,39,40,41,42]. Mizumoto and Eguchi [38] evaluated 12,100 essays using the GPT-3.5 model and found that the tool demonstrated a satisfactory degree of accuracy and reliability in grading essays. In [43], a comparative study between GPT-4, GPT-3.5, Claude 2 and PaLM 2 was conducted, concluding to the superiority of GPT-4 in the field of grading English writing. In another recent study [37], the application of LLMs, including GPT and Gemini to two large datasets, interestingly showed that both systems achieved improved objectivity and reduced bias compared to human raters, while Gemini outperformed ChatGPT, achieving greater efficiency.

On the other hand, several of the related studies identified the limitations of ChatGPT, reporting that its scores did not strongly correlate with those given by human raters [28] and that discrepancies exist in the automatic scores, e.g. giving lower or higher scores on various criteria of the rubric [44,45]. Limitations have also been reported in providing comprehensive feedback [38,44,46], as well as failures such as producing feedback which was off-topic [38].

The conflicting findings of these studies raise questions about the validity and reliability of automatic scoring using AI [35]. Therefore, tools like ChatGPT need to be optimized through domain-specific data and expert feedback, to create more accurate frameworks. Solutions proposed for these tools to produce consistent results that will not change in successive assessment cycles were to clearly define the rubric evaluation criteria in advance and to continuously train the model by the user each time in the specific context and task, to avoid model drift [28,47]. Common arguments are that the complexity and quality of rubrics have a great impact on the performance of automatic evaluation systems, affecting the reliability of criteria definitions, reducing subjective judgments and measurement errors [37]. The use of widely available AI tools in everyday educational practice is according to the above dependent on the ability of the educators to formulate the necessary prompts [48]. It therefore remains unanswered whether an educator with limited knowledge and effort can get accurate and reliable results to perform specialized tasks, such as evaluating higher education assignments.

The aim of this paper is to investigate the abilities of general-purpose AI tools in automatically assessing student’s assignments in higher education. The tool chosen for the research was ChatGPT, with which we experimented to evaluate whether educators, through elementary prompt engineering, can effectively use a costless general-purpose AI tool for their daily educational work. More specifically, the research questions addressed are:

RQ1. Can educators effectively and reliably use a general-purpose AI tool such as ChatGPT in educational practice to automatically assess complex student assignments, in alignment with a course lecturer, at low or zero cost for the educator?

RQ2. Which are the minimum skills in AI prompting required by an educator to successfully perform automatic assessment of student assignments in online higher education courses?

RQ3. Can AI-assisted automatic assessment contribute to sustainability goals in education by strengthening the quality and viability of online higher education courses?

The current study builds on the extensive existing research in AES but proceeds beyond the assessment of writing skills and simple problem solving to the assessment of specialized student work that consists of multiple heterogeneous parts (e.g. understanding and correct application of design process, multimedia content creation, publication of digital learning objects, ethics and licensing, knowledge management). To this end, we experimented in grading assignments of postgraduate students, engaging real-world users and analyzing empirical educational data. An iterative process of various levels of complexity was applied to instruct and refocus the AI tool to imitate the professor’s assessment criteria. The trials were applied on the course “Open Educational Resources” of the postgraduate program in Management of Educational Units, which was selected as a representative course that emphasizes on comprehension and knowledge application. The performance of the AI tool was evaluated not only in terms of accuracy and reliability, but also qualitatively, gaining insights in the tools’ behavior and highlighting its strengths and potential weaknesses. Further goal was to contribute to educational practice by investigating which are the skills required by lecturers and educators to effectively use this tool for their daily student assessments, with acceptable accuracy and reliability.

2. Materials and Methods

2.1. Methodology

To evaluate the applicability of AI tools as automated grading assistants, a Design-Based Research (DBR) methodological framework was adopted. DBR was chosen for its suitability to real-world educational problem solving through the iterative design and evaluation of an experimental intervention. In this study, the intervention was the AI-assisted automatic assignment grading. The aim of the methods was to acquire knowledge about the effectiveness of general-purpose AI tools to imitate a professor's assessment of complex student tasks in an online postgraduate environment and at the same time identify the competencies that professors need to possess to achieve high-quality outputs. As depicted in Figure 1, our approach included a multi-cycle process to calibrate the AI tool’s performance in a representative case in the context of higher education. More specifically, we designed and constructed a prompt-engineering and calibration framework, so that in iterative cycles, ChatGPT, was instructed to automatically assess student assignments, aligning its output with human educators’ grading standards. ChatGPT, in particular the free version, was chosen as a representative general-purpose free AI tool. The results produced by ChatGPT at various levels of prompting competence were quantitatively compared with human judgment and evaluated qualitatively by expert educators.

2.2. Experimental Setup and Dataset

The case selected for this study was the AI-assisted automatic assessment of the project assigned to the students of the postgraduate program on Management and Organisation of Educational Units, within the course titled “Open Educational Resources”. The assignment is mandatory and contributes to the 50% of the final grade. To complete the assignment, the students were required to create a new Open Educational Resource (OER) (e.g. a lesson or an interactive presentation) by mixing existing learning objects, using a common e-learning platform and content management tools. In addition to the delivery of the OER itself as a published resource, they needed to submit a report where they presented the educational aims, the process they followed for searching, adapting and remixing the content of their OER, and their decisions regarding licensing issues. Additional requirements were to reuse at least one existing learning object that they found in an OER repository, to modify an object by applying a multimedia processing tool, and to create at least one new learning object. The students were instructed to search for open content making use of metadata and to describe their own content with metadata. Emphasis was given to the understanding of the concepts related to OERs, the ability to apply them in educational practice by following correct procedures, while the originality and the interestingness of the content should not be considered important. For the purposes of grading, the criteria and their weights, along with grading instructions were explicitly defined by the course professor, as outlined in Table 1.

The particular assignment has been selected for this study as a representative complex task which poses challenges to its automatic assessment. The criterion of Completeness is fairly straightforward to assess and can be evaluated objectively. The criterion of Understanding is, however, is more abstract and complex. The rater is challenged to evaluate not only the correctness of the student’s statements or the alignment of its descriptions with an established template, but also to conclude from the student’s descriptions the level of comprehension. The AI-tool is expected to detect elements such as appropriate use of terminology, logically applied steps, and rationality of explanations. Furthermore, sub-criterion “2.4 Licensing” can be characterized as problem-solving (i.e. the student must choose the correct license, according to his/her goals and so that the restrictions posed by the reused objects are not violated). In order to assess correctly, before checking the student’s decisions, the AI rater is therefore required to correctly find itself the solutions. The 3^rd criterion “Quality & Richness” includes a certain degree of subjectivity and by its nature is difficult to quantify, at least without some kind of reference or comparison. It is noted that the specific assignment in OER has not been created for the current experiment but existed as an assessment method in the real online postgraduate course for several years.

The dataset used as input for this research was a sample of 39 reports, randomly selected among the 58 assignments from the course archives for which the students gave consent. It is noted that the assignments were anonymized, the current research was performed after the completion of the courses, and its results did not affect in any way neither the teaching process nor the students’ grades. It is noted that the particular postgraduate program was taught in Greek, therefore, the assignments were written in Greek. They were used in the current research in their original language.

All the assignments were evaluated manually, in accordance with the professor’s judgement, by following a detailed set of criteria and keeping notes of the justifications for each grading. Two human evaluators participated in the experiment: (1) the course leader, who is the professor that set up and has been teaching the course for several years and (2) a researcher in educational technologies acting as lecturer. The researcher is a philologist with moderate previous experience in prompt engineering, who has teaching experience and is at the same time an older graduate of the specific postgraduate program.

2.3. The Iterative Calibration Process

To calibrate and evaluate the potential of the AI tool, we applied a multi-cycle process of increasing prompting complexity and effort, replicating the actions of educators with increasing levels of prompting competence.

The process comprised the following steps:

• Zero-shot prompt. It was performed by the researcher, acting as course lecturer with limited experience in prompt engineering and without spending effort in the automatic assessment procedure. The prompt included (a) the assignment description, instructions and structure template, as it was given to the students, and (b) instructions about where to give emphasis (e.g. to evaluate the understanding of the concepts and the correctness of the design and OER mixing process, giving less emphasis to the linguistic characteristics of the description texts and the actual educational value of the learning resource). The AI tool was asked to generate the assessment rubric by itself and use it for grading, without any examples.
• One-shot prompt. In this phase, a fresh session, independent from the previous phase was initiated. In addition to the problem description and the structure of the expected assignment, a detailed rubric was given to the AI tool by the professor, specifying the criteria and their weighting, the overall scale and specific instructions about what to check and how to grade each criterion (see Table 1). Additionally, 3 labeled assignments were given as examples, providing the correct scores per criterion and justifications for these scores. The performance of the tool was tested on 6 unknown assignments.
• Few-shot prompt. In this phase, the results of the previous cycle were used as a starting point to further calibrate the AI tool. The output of ChatGPT for the previous 6 testing assignments was corrected in the same session, indicating the points were the tool deviated from the desirable output and providing further clarifications. In this way, a cycle of refinement was performed, where, following the initial 3 examples, 6 more cases were used for calibration. In order to produce unambiguous and well-structured prompts, the researcher spent considerable effort in familiarizing herself with the principles of prompting. She then compiled an elaborate structured prompt, which included clear definitions of role, aims and context, as well as tagging and strict structure. A strict structure was used for both problem/rubric definition supported with examples and also for submitting assignments for grading. The performance of ChatGPT was then tested on 30 new unseen assignments.

The entire process was performed in the free version of ChatGPT running the GPT5 model.

2.4. Evaluation Metrics

To evaluate the performance of the calibrated ChatGPT model, we employed an inter-rater reliability framework. Student assignments were independently graded by the professor (course leader), a researcher acting as lecturer, and the AI system—using a common grading rubric, a 0–20 scale for the overall grade, broken down to 3 criteria, as well as free-text justifications for the grade of each criterion. The adopted metrics for evaluating the inter-rater agreement and consistency were the Two-Way Intraclass Correlation Coefficients (ICC) [49]. The ICC2 type, focusing on agreement, was utilized to evaluate absolute agreement (i.e. whether the AI tool gives exactly the same grades compared to the human baseline) and the ICC3 type, focusing on consistency, to evaluate whether the AI tool ranks the assignments in the same relative order, even if it is systematically stricter or more lenient. Both coefficients are interpreted as excellent agreement if below 0.90, good agreement if 0.75 to 0.90, moderate agreement if 0.50 to 0.75, and poor agreement if below 0.50. The ICC framework was preferred against the Quadratic Weighted Kappa (QWK), although the latter is the standard in AES, because our 20-point grading scale behaves continuously rather than ordinal.

Pairwise ICCs were computed between the two human raters and between each human rater and the AI, enabling direct comparison of AI–human agreement against baseline human–human agreement. As a supplementary analysis, grading discrepancies between the course lecturers and the AI were quantified using error-based metrics. Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) were calculated on the original 0–20 grading scale to maintain interpretability and were used descriptively to characterize the magnitude of grading differences.

In this paper, Generative AI has only been used as an AAS, i.e. in the core of the experiment, as described in the research methodology.

3. Results

3.1. Results of 1^st Cycle (Zero-Shot Prompt)

Based on the assignment brief and general information about the assessment goals, and with minimal effort from the side of the lecturer, ChatGPT was prompted to generate its own rubric and then apply it. AI concluded to a set of 7 criteria, as shown in Figure 2, including the understanding of OER, search and metadata quality, and license analysis and justification.

To observe how the rubrics worked, a small sample of 3 assignments were given for assessment, asking ChatGPT to provide both numerical grades and justifications per criterion. The produced assessments were compared with the opinion of the researchers and were checked qualitatively in detail. The results of this 1st effort to apply automatically produced criteria were encouraging, however many errors were found. Although the rubrics were well formulated and the justifications given for each grade seemed reasonable, after a closer look it was found that several assessments were wrong. For example, the tool gave to one of the assignments the maximum grade to the criterion of “Platform and Format”, although important related information was totally missing from the report. It was interesting that the tool identified particular sections in the report, e.g. Metadata, and gave high scores and positive comments, but failed to detect that the content itself in these sections was erroneous or insufficient (e.g. it was missed that from the content of the section Metadata, it was clear that the student did not really understand what metadata is). Another negative effect was that when the same assignments were resubmitted, the tool responded with considerably different scores and totally different justifications.

The AI-generated framework missed several points which were considered important by the professor, e.g. it was not checked whether all the required sections according to the assignment instructions were covered, errors in the student’s design choices regarding licensing were not spotted, and the educational value of the produced learning objects was ambiguously evaluated with excessive weight. As a result, although the “fully automatic” generation of assessments looked appealing at first glance, it was proved to be far from applicable to educational practice, at least in complex higher education tasks.

Figure 4. The response of ChatGPT for grading assignment #2.

Figure 4. The response of ChatGPT for grading assignment #2.

3.2. Results of 2^nd Cycle (One-Shot Prompt)

In this cycle, ChatGPT was instructed by a manually created set of criteria that was fixed and given to ChatGPT explicitly, i.e. a prompt was compiled which defined three main criteria (Completeness, Understanding of concepts related to OERs, and Quality and richness of content), their sub-criteria, the scale and scores of each sub-criterion, and detailed instructions about what to check. The final set of criteria, as formulated by ChatGPT is shown in Figure 3.

The prompt with the instructions included information about the role and task assigned to ChatGPT, the objectives and some general rules. The instruction prompt included 3 pre-graded examples, chosen to represent one excellent, one fair and one poor assignment. Testing was performed on 6 assignments, comparing the grades suggested by ChatGPT with the professor’s grades per criterion and by evaluating qualitatively the provided justifications. The grades are summarized for comparison in Table 2 and an indicative output of ChatGPT is shown in Figure 3. Ιn just one from the 6 cases, both the AI and the human rater gave the same grade, while in one case there was a serious failure. In the 0-20 scale, the mean absolute error was MAE=1.93 and the Root Mean Squared Error was RMSE=3.07.

In more detail, assignment #1 was evaluated by both the professor and the AI tool with 20, in complete agreement. Assignment #2 was given a total grade of 18.5 by the AI and 19 by the professor. In this case, small differences were noticed in all 3 criteria, since the tool, in comparison with the human rater, evaluated Completeness slightly higher (9.5/10), while Understanding and Quality & Richness slightly lower (4.5/5). The worst case was assignment #4 (13 vs 20) and large deviations were also noticed at #6 (17.5 vs 20), where, although the professor found serious problems, the AI tool judged them both as excellent.

In order to get insights about the causes of the errors, we had a closer look at the most striking failures (assignments #4 and #6), where problems were found in all criteria. For this purpose, a follow-up conversation with ChatGPT was carried out. Feedback was given to the AI tool about the correct grades and their justifications (according to the professor) and it was asked to re-evaluate the assignments. This was performed in multiple steps, until ChatGPT confirmed that it is in agreement with the professor. The responses throughout these steps were informative of the behavior of the AI and the limitations of the initial one-shot prompt.

As an example, in assignment #4, regarding Completeness, the professor found that one of the sections was totally missing, and sub-criteria 1.2 and 1.3 were not fulfilled (the learning objects were not of at least 2 different levels and there was no object processed by the student). Although the criterion can be considered objective, ChatGPT failed to apply the instructions correctly. Corrective feedback was provided to the tool that “if a resource is only reused as-is, the criterion is not fulfilled – must involve active modification”. The modification of the grade from 2/2 to 0/2 was then justified by the tool as “The report mentions reuse but does not clearly describe any actual modification”. Regarding the criterion “Understanding”, when ChatGPT was asked to re-evaluate assignment #4, it responded that “the student described the process in a narrative, itemized way but lacked clear justification of choices made in reuse vs creation”. The justification for modifying the grade from 5/5 to 3/5 to match the professor’s judgement was that “there was factual correctness but weaker conceptual depth or reflection”.

The above conversation with the AI revealed that the initial instructions given by the educator, although contained a detailed rubric and examples, left plenty of room for improvement. Many aspects expected to be learned by the AI were not effectively described, even if they seemed fairly straightforward. The response of the AI tool to subjective elements such as understanding and quality, which are by their nature more challenging, was even more vague. The results of this 2^nd cycle of trials were thus encouraging but also showed the need for improvements through more elaborate instructions.

3.3. Results of 3^nd Cycle (Calibrated ChatGPT)

In this cycle, the 6 assignments from the previous cycle were used as training examples to calibrate ChatGPT through an extensive prompt in which there was feedback pointing at the previously detected failures, as well as more detailed justifications for the correct grades per criterion and further instructions. The prompt comprised explicitly tagged sections, which stated the Role assigned to ChatGPT (i.e. “you are a professor in the postgraduate program …”), the Objective (e.g. “you are asked to grade student assignments …”), Rules (general rules e.g. “do not diverge from the rubric”, “do not add new criteria”, …), Graded examples, and Rubric. Moreover, all student assignments fed to ChatGPT, either as graded examples or student submissions, were not inserted as unprocessed files but were cleaned and formatted as prompts i.e. all student identification data, cover page and formatting were removed and the main sections were tagged. ChatGPT responded with an acknowledgement of the new instructions and updated the assessments, considering the requested corrections.

The testing of the calibrated AI tool on 30 unseen assignments showed considerable improvements. The agreement between AI and the two human raters was evaluated by measuring the Agreement ICC2 metric, while the consistency of the AI was evaluated by the Consistency ICC3 metric. Results are shown in Table 3. The strong agreement and reliability coefficients between the two human raters (0.85 and 0.89, respectively) indicate that they share common grading approach and can be used as a reliable reference for evaluating the AI assessments. However, the considerable RMSE (1.48) and non-negligible average difference between the human raters show that their agreement is not absolute. While there is agreement in Completeness, it seems that the professor is systematically more lenient in Understanding and in Quality & Richness.

In terms of agreement in total grades, the correlation coefficient of ICC2=0.82 indicated that ChatGPT achieved good agreement with the professor. Considering that the agreement between lecturer and professor was only slightly better (ICC2=0.85) the performance of the calibrated ChatGPT can be considered more than satisfactory. As regards reliability of total grades, the result was similar: the reliability of the AI tool was very close to the level of another human rater. The absolute error between AI and professor was on average MAE=1.12 and the root mean squared error RMSE=1.5. In Figure 5, the differences in total grade between AI and professor are depicted graphically. The mean difference measured in the testing sample of 30 assignments was -0.48 (p=0.078), which means that the difference can marginally be characterized as statistically insignificant.

Observing performance at individual criteria, considerable variations were found. In criterion 1 “Completeness”, the alignment with the professor was more successful, since this criterion is more objective and is broken down to very specific sub-criteria. After careful calibration, ChatGPT managed to search for the correct elements e.g. to find whether all the sections were present or whether the student reused an OER after modification. On the other hand, it is worth noting a surprising error: in 2 cases, sub-criterion 1.1 (“Are all sections covered?”) was graded with 7 points, although the max score was explicitly defined as 6. In criterion 2 “Understanding” and criterion 3 “Quality & richness”, which by their nature are more abstract, it was noticed that ChatGPT tended to give “smooth” scores, i.e. the same moderate score (most often 4 from max 5) to a large number of cases, while the human evaluation seemed more sensitive in capturing fine differences and was often more extreme.

4. Discussion

The aim of the present study was to explore the potential of a widely accessible, general-purpose AI tool to support automated assessment, focusing on specialized higher education topics, where assessment is more demanding. The performance of the AI-tool was evaluated in relation to the skills and effort required from the side of the educators, following a process that started from zero-shot prompting with minimal effort and skills, up to multi-cycle calibration with structured and skillful prompts. In this work, we did not consider more sophisticated prompt engineering techniques, neither specialized commercial or research-level AAS, since our target was to contribute to the everyday educational practice of non-specialists to AI. The end goal of this paper was to provide insights of how AI-assisted automatic assessment can reduce human effort and costs, and at the same time enhance the quality and viability of online higher education courses, contributing to wider sustainability goals in education.

As a result of the experiments performed in real-world environment, the quality of the prompts and the use of detailed rubrics with scoring criteria, examples of evaluation at all levels, and multiple cycles of calibration did not just affect the quality of the assessment results. These prerequisites were proved to be critical to achieve an acceptable level of performance. Although tempting for an inexperienced educator, without effort to provide well-designed instructions, a general-purpose AI tool such as ChatGPT could not adapt its behavior to the way that a higher education professor assesses students. If the AI is allowed to assume and improvise, unexpected results may occur, which are not acceptable in practice.

Comparing the results of one-shot prompt with the calibrated few-shot prompt, it can be seen that there was significant distance. Quantitatively, the error in the total grades given by ChatGPT in the one-shot cycle was RMSE=3.07, while after calibration it was reduced to RMSE=1.5, which is equivalent to the difference between two human raters. Qualitatively, at first glance, it could be claimed that the grades and justifications provided by the AI tool seemed realistic and convincing, even without calibration. The 6 assessments performed in the 2^nd cycle, although reflecting complex tasks, gave the impression that they followed the rubric and instructions provided to the AI, and the justifications inspired confidence that all criteria were considered and measured. However, the comparison with the professor’s opinion revealed serious flaws. I was clear that grading was performed superficially and, in several cases, erroneously. The instructions given to the AI were not sufficient for the tool to establish by itself a correct and reliable grading process.

On the contrary, the performance achieved in the 3^rd cycle was found to be satisfactory. ChatGPT reached a level of agreement with the professor which was comparable with the agreement between two human course lecturers. Considering that, according to the literature, an AAS with an agreement coefficient higher than 0.7 is acceptable in real practice, the achieved ICC=0.82 showed that it is indeed feasible to use a general-purpose AI tool for complex assignment grading. Therefore, it was confirmed that automatic assessment of written assignments on a relatively complex subject can be achieved, provided that a solid process of aligning the tool with the course lecturer’s thinking is firstly carried out. Since a level of subjectivity is anyway inherent in such evaluations, the accuracy and consistency reached in our experiment was satisfactory. On the other hand, if an assessment was carried out without refining the criteria through several cycles of instructions, examples and corrective actions, the tool itself would output incorrect conclusions. An important disadvantage which is worth stressing, was that the tool always provided a result with confidence, creating the illusion that it was really aligned with the human lecturer.

As a result of this study, the minimum skills that the educator must at least have are: (a) to be familiarized with the basic principles of prompt engineering i.e. to be able to structure an extensive prompt with the appropriate tags and to carry out a multi-step calibration process (b) to be knowledgeable of the topic to be assessed and able to create a well-defined rubric, along with detailed instructions, (c) to be aware of the limitations and characteristics of AI technologies, so that he/she can critically and responsibly use, evaluate and develop tools, sensing limitations and potential problems. At the expense of certain effort spent to carefully calibrate AI-assisted assessment tools, online higher education study programs can have serious benefits towards sustainability and quality of education.