Participatory Co-Design and Evaluation of a Novel Approach to Generative AI-Integrated Coursework Assessment in Higher Education

1. Introduction

Recent advances in generative artificial intelligence (AI), powered by large language models, present opportunities and challenges for assessment in higher education. AI is now widely used across sectors including health, industry, and research (McKinsey, 2024; Sun et al., 2023), and is permanently reshaping the nature of academic tasks. In educational settings, AI has already shown potential to support learning by providing personalised feedback, scaffolding writing processes, and automating routine tasks (Kasneci et al., 2023). However, evidence on how best to use it in teaching, especially assessment design, remains limited.

Initial studies suggest that while students may benefit from AI-enhanced feedback, overreliance on these tools may undermine opportunities for deep learning and critical engagement (Kasneci et al., 2023; Zawacki-Richter et al., 2019). The integration of generative AI in education also presents challenges. Equity concerns persist, including unequal access to reliable AI tools and the digital skills needed to use them meaningfully (UNESCO, 2024). Academic integrity is also at risk, as AI can be used to cheat in ways that evade detection (Yusuf et al., 2024). Moreover, the use of AI complicates traditional concepts of authorship and originality, raising questions about what constitutes independent academic work. There are also concerns that critical thinking, a key goal of higher education, could be weakened if students accept AI outputs without careful evaluation (Bittle & El-Gayar, 2025).

In response, there is growing recognition of the need to build critical AI literacy among students and staff. This means not just knowing how to use AI tools, but understanding how they work, the wider impacts they have, and how to assess AI-generated content carefully and ethically (Abdelghani et al., 2023). Developing critical AI literacy is needed to prepare students to be thoughtful, responsible users of AI, and should be built into teaching and assessment strategies.

We aimed to improve the critical AI literacy of postgraduate students and teaching staff through the co-design and evaluation of an AI-integrated written coursework assessment. In this assessment, students used generative AI tools to draft a blog critically summarising an empirical research article and produced a reflective, critical commentary on the AI-generated content.

Specifically, we sought to:

• Use participatory methods to design and evaluate a novel assessment approach.
• Assess its acceptability and feasibility for students and teaching staff.
• Test whether teaching staff could distinguish between assessments completed in accordance with the brief and those generated entirely by AI.

Findings were used to develop practical guidance to support the implementation of the assessment within the postgraduate programme and contribute to the wider pedagogical literature on assessment in higher education.

2. Materials and Methods

This study used a participatory evaluation approach. Participatory evaluation involves contributors not just as participants, but as co-designers and co-evaluators (Fetterman et al., 2017), and has been used previously to explore AI-related resources and curriculum development (Cousin, 2006; Teodorowski et al., 2023). A strength of this approach is its emphasis on different forms of expertise, including lived experience, disciplinary knowledge, and teaching practice, which contribute to the development of assessments that are both grounded and relevant.

The protocol is available at OSF Registries: doi.org/10.17605/OSF.IO/JQPCE. We used the Guidance for Reporting Involvement of Patients and the Public short form (GRIPP2-SF) checklist to report involvement in the study (reported in Table A1 in the Appendix#;(Staniszewska et al., 2017). The study was approved by the Research Ethics Panel of King’s College London (LRS/DP-23/24-42387; 27 June 2024).

The study took place within a psychology-based postgraduate programme in the United Kingdom. Participants brought a range of expertise, from digital literacy and academic writing to assessment design. By drawing on the knowledge and experience of those directly involved in teaching and in learning, the participatory methods promoted shared ownership and practical relevance, while still allowing space for innovation.

Twelve participants took part in the study, which included eight students and four members of the teaching team from the same academic cohort (2023–24) of a postgraduate course at the Institute of Psychiatry, Psychology and Neuroscience, King’s College London, a Russell Group university in the United Kingdom. The teaching staff included a Teaching Fellow, a Lecturer, and two Research Associates. All participants had recently completed or marked a summative assessment within the course.

2.1. Stage 1: AI integrated Assessment

The research team collaborated with other members of the course teaching team to adapt an existing summative assessment already embedded in the curriculum. This assessment required students to write a blog post summarising and critically appraising an empirical research article on mental health. Framed as an authentic assessment, the task included the potential for selected blogs to be published on science communication platforms.

We used the Transforming Assessment in Higher Education framework developed by AdvanceHE to guide our approach to integrating generative AI tools into this assessment (Healey & Healey, 2019). The framework highlights the need for assessments that are authentic, inclusive, and aligned with learning outcomes, emphasising the importance of involving students in the assessment development process. This emphasis aligned with our approach of integrating AI tools to reflect real-world practices and to develop critical AI literacy.

Under the revised assessment approach, students were asked to use two AI tools to assist with drafting a blog based on an empirical article. The written assessment consisted of three components:

• Two AI-generated blog drafts, using two AI tools.
• A final blog that combined the strongest elements of the AI outputs with the student’s own revisions and original contributions, assessed for the accurate and critical appraisal of the empirical article.
• A commentary critically reflecting on the AI-generated content and explaining the rationale for revisions made, assessed for the depth of critical and ethical reflection.

The marking matrix was revised to retain the use of a standard critical appraisal checklist for assessing students’ understanding of the empirical article, alongside the programme-wide marking framework (stage 2). New criteria were introduced to evaluate students’ critical engagement with AI-generated content (stage 3). The adapted format built on the existing learning outcome of critically appraising empirical research, extending it to assess students’ ability to reflect on the role of AI in academic work, apply subject knowledge to evaluate AI outputs, and make informed editorial decisions.

2.2. Stage 2: Assessment Trial

All participants were invited to take part in a trial of the adapted assessment. They first attended a workshop designed to support students in their AI-assisted assessment. Microsoft Copilot, in both balanced and precise modes, was the mandated generative AI tool, selected for its free availability for the participants (ensuring equitable access) and to allow direct comparison between model outputs. While Copilot was used in this instance, the assessment was designed to be transferable to other AI tools.

The workshop was delivered in four stages. The first introduced Copilot’s core functions, including its strengths, limitations, and examples of effective prompt writing. In the second stage, students practised drafting prompts and used the AI models to generate and revise a mock blog post. The final two stages drew on Gibbs’ Reflective Cycle to guide structured learning (Gibbs, 1988). In stage three, students critically appraised an AI-generated blog, and compared the outputs produced by the two Copilot modes. This exercise supported a deeper understanding and analysis of AI-generated content. In the final stage, students reflected on their use of AI and developed an action plan for how they would apply AI tools in future academic work. This reflection aimed to consolidate learning and promote ethical, informed use of generative AI tools.

Feedback on the workshop was collected through qualitative discussions at the end of the session and a short survey. The survey included a Likert-scale question assessing whether the workshop would help students complete the assessment (responses: yes, somewhat, no) and two free-text questions: “What did you learn from the workshop?” and “What was missing from the workshop that would help you feel more prepared for the pilot assessment?”

Student participants were then randomly allocated to one of two groups. Those in the ‘honest’ group were instructed to follow the coursework brief precisely, using the designated AI tools as directed. Students in the ‘cheat’ group were given freedom to complete the assessment by any means, including generating the entire submission using AI tools. They were encouraged to be creative and to push the boundaries of the process. Teaching staff participants were asked to mark the submitted assessments and provide written student feedback using the adapted marking matrix. They also indicated whether they believed the student had completed the assessment as instructed (‘honest’ group) or had been in the unrestricted (‘cheat’) condition.

2.3. Stage 3: Evaluation

Participants participated in an iterative process of reviewing and refining the assessment materials, including the workshop content, coursework brief, and marking matrix.

To explore the feasibility, acceptability, and perceived integrity of the AI-integrated assessment approach, we conducted a series of semi-structured focus groups with students and individual interviews with teaching staff. This format was chosen to accommodate participant preferences and availability.

The discussion guides are reported in Appendix 2. They were developed to address the study’s research aims and to capture experiences across both groups regarding their engagement with generative AI in the context of assessment. Both focus groups and interviews lasted approximately 45 to 60 minutes and were structured in two parts: the first explored existing knowledge of generative AI and experiences of completing or marking the assessment; the second addressed reflections on the assessment design and its potential for future implementation. In addition, we explored perceptions of ‘cheating’ in the assessment, including whether students in the ‘honest’ and ‘cheating’ groups felt they had met the intended learning outcomes, and whether staff felt able to distinguish between the two. Particular attention was paid to whether the approach supported intended learning outcomes and provided a fair measure of student performance.

We also asked questions about the initial training workshop as part of the interviews. This feedback was reviewed alongside data from the survey questions completed by participants after the workshop and was used to revise and improve the training content.

Focus groups and interviews were conducted via Microsoft Teams. Thematic analysis was led by one researcher (AFM), following Braun and Clarke’s (2006) approach, including: familiarisation with the data, initial coding, theme identification, and iterative theme refinement (Braun & Clarke, 2006). Analysis was performed separately for students and teaching staff. Emerging themes were reviewed and refined through discussion within the research team and with participants who participated in subsequent workshops.

We held co-design workshops with students and teaching staff to further refine the assessment brief and marking matrix, respectively. The think aloud technique was used (Charters, 2003; Someren et al., 1994), whereby each section of the assessment materials was reviewed in turn. Participants took part in a facilitated group discussion, voicing their thoughts, suggestions, and reactions in real time as they engaged with the materials. Data saturation was considered to have occurred when no further substantial changes were proposed by the participants. Two workshops were held with students and one with teaching staff, which likely reflects the fact that more extensive feedback had already been gathered from teaching staff during earlier individual interviews and incorporated into the materials prior to the workshops.

Feedback gathered during these sessions was used to inform revisions to the assessment materials. We documented this process using a Table of Changes (ToC) from the Person-Based Approach (Bradbury et al., 2018). We also used a Custom GPT, which allows creation of a personalised version of ChatGPT tailored to specific tasks or knowledge domains, to review the final materials for accessibility and readability.

3. Results

3.1. Assessment Materials and Learning Outcomes

The modifications made to the assessment materials are summarised in the ToC (Table 1).

Feedback on the co-designed assessment materials produced by the Custom GPT indicated that, while both the assessment brief and marking matrix were generally well-structured and aligned with learning outcomes, several refinements could improve readability and accessibility. These included ensuring consistency of language and tone, using bullet points and clearer formatting to support navigation, and clarifying instructions around AI tool use and submission structure. Minor revisions were recommended to the learning outcomes and the reflection criteria to enhance alignment with marking expectations.

The final versions of the assessment materials (workshop proformas, assessment brief and marking matrix) and the amendments recommended by ChatGPT are included in Appendix C and at the Open Science Framework project: osf.io/ctewk/.

Feedback on the learning outcomes was generally positive or neutral, with no negative responses offered. Students and teaching staff appreciated the inclusion of learning outcomes and found them helpful for understanding the purpose of the assessment. Some suggested making the link between the learning outcomes and the specific assessment tasks more explicit, to improve alignment and clarify expectations. The major change that emerged from all feedback sources was the need to communicate that critical appraisal of the original empirical article is as important as appraisal of the ability of AI to generate seemingly useful content. One student noted that engaging with the AI output highlighted inaccuracies, such as fabricated participant details, which prompted them to critically verify the content against the original source. This process, while demanding, was seen as intellectually valuable: “It forces you to actually figure out whether you’re critically appraising the critical appraisal.” Another contributor reflected on the need to distinguish between assessing AI literacy and assessing critical thinking, suggesting that the learning objectives should clearly indicate which of these skills is being prioritised. This feedback informed revisions to the assessment brief and the revised learning outcomes.

Our revised learning outcomes became:

• Critical appraisal: Students will demonstrate the ability to critically appraise academic content by:
  • Evaluating an empirical research article using an established critical appraisal checklist.
  • Assessing the accuracy, relevance, and limitations of AI-generated content in relation to the original empirical article.
  • Comparing outputs from different AI tools, identifying their strengths and weaknesses in academic content generation.
• Generative AI literacy: Students will develop foundational AI literacy by using generative tools to support scientific blog writing. They will demonstrate an understanding of AI capabilities and limitations, including the ability to identify common errors such as fabrication or hallucination.
• Editorial and reflective judgment: Students will apply editorial judgement to revise AI-generated content, integrating critical analysis and original insight. They will reflect on their use of AI tools and articulate the rationale for content modifications in alignment with accuracy, academic standards, and ethical considerations.

3.2. Feasibility, Acceptability, and Integrity

Table 2 presents summaries of the key findings and illustrative quotes from the thematic analysis of the focus groups and interviews.

Student feedback highlighted that while AI tools could streamline aspects of the writing process, they did not reduce workload due to the effort required to refine outputs. Perceptions of feasibility, acceptability, and integrity varied, with students valuing the opportunity to build critical thinking skills, but also expressing concerns about fairness, skill development, and notably, ownership of their work. Some viewed equitable access and thoughtful integration of AI to be particularly important for maintaining academic standards. Teaching staff found the assessment structure clear, though marking was initially time-intensive because of the dual task of evaluating both AI and student contributions. Efficiency improved with familiarity, and staff recognised the assessment’s potential to support critical engagement. While challenges remained in distinguishing AI-generated from student-authored content, most staff endorsed transparent and pedagogically grounded use of AI in academic settings.

Students in the cheat group found that using AI to complete the entire assessment was challenging, with outputs, particularly the reflective commentary, requiring substantial oversight and correction. Some spent a similar amount of time on the task as those in the honest group, while others felt they used somewhat less. Most felt they had achieved the intended learning outcomes due to the time spent checking, appraising, and reflecting on the AI-generated content.

Assessment marks ranged from 35 (fail) to 78 (distinction). For most assessments, marks from different markers fell within a ten-point range, but for one assessment scores ranged more widely (58 to 78). Markers were not significantly better at correctly identifying students in the ‘honest’ group (6/14 accurately identified) compared to those in the ‘cheat’ group (3/6 accurately identified; X²(1, N = 20) = 0.09, p = .769). Markers’ views on identifying students in the cheat condition were polarised: some reported having no clear sense, while others felt very confident that they could recognise AI-generated submissions. However, these subjective impressions were not reflected in their actual ability to accurately distinguish between the groups.

4. Discussion

The findings from this study add to a nascent body of literature that highlights the dual role of AI-integrated assessments, as tools for digital literacy and as mechanisms for reflective, critical pedagogy. The blog format provided a unique opportunity for students to practise public-facing, accessible academic writing, aligning with real-world expectations in health communication. Pilot findings showed that students found the approach feasible and helpful for developing critical skills, although engaging with AI outputs was perceived to increase workload. Teaching staff initially found marking more demanding and had limited success distinguishing AI-generated content, but valued the assessment’s potential to promote ethical and critical AI use.

A key success of the project was the development of students’ critical AI literacy, with findings suggesting that the blog assessment promoted active engagement with AI outputs. Students were required not only to use AI tools but to critique their outputs, identify inaccuracies, and justify their editorial decisions. This process encouraged deeper critical engagement and helped students to view AI as a tool requiring human oversight rather than as a source of ready-made answers. The requirement to compare outputs across different AI models also supported the development of critical evaluation skills, as students reflected on the variability and limitations of AI-generated content. Importantly, these findings address concerns raised during the qualitative evaluation, and reflected issues highlighted in previous research, that overreliance on AI could undermine opportunities for deep learning and reflective practice (Kasneci et al., 2023; Zawacki-Richter et al., 2019). These findings align with recommendations that AI in education must go beyond functional skills to include AI literacy, as well as active learning skills and metacognition (Abdelghani et al., 2023).

Beyond promoting critical engagement with AI outputs, the study also highlighted strategies for maintaining assessment integrity and supporting academic skill development. Teaching staff expressed concerns that AI use could make it harder to distinguish original work from AI-generated content. This echoes broader challenges in the literature, where AI use may complicate traditional definitions of authorship and independent academic work (Yusuf et al., 2024). Although markers were sometimes confident, their accuracy in identifying AI-reliant submissions from the ‘cheat group’ was poor. This is likely to be because students in the cheat group generally described a similar editorial process to those in the honest group. Nevertheless, the assessment’s structure, requiring critical appraisal of the empirical article, critique of AI outputs, evidence of revision, and transparency may have helped to mitigate these risks, although this needs further testing.

By embedding critical evaluation and editorial judgement, the assessment addressed concerns that AI could weaken core academic skills such as critical thinking and reflective analysis (Bittle & El-Gayar, 2025). Students also recognised that genuine engagement, not uncritical acceptance of AI outputs, was needed to meet the learning outcomes. However, the extent to which students internalised critical evaluation versus simply complying with task requirements remains unclear. Future studies could explore students’ metacognitive strategies and critical reasoning during AI use through longitudinal or think-aloud methodologies (Charters, 2003; Someren et al., 1994). Overall, the findings suggest that carefully designed AI-integrated assessments can uphold academic integrity while supporting the development of essential academic competencies.

Involving students and teaching staff in the co-design and evaluation process was central to developing an assessment that was authentic, feasible, and acceptable. The participatory approach drew on academic, pedagogical, and lived experience to shape the teaching workshop and assessment materials, helping us spot practical challenges early and promote shared ownership of the development of the assessment (Fetterman et al., 2017; Teodorowski et al., 2023). This aligns with broader calls for more inclusive, responsive, and transparent innovation in educational assessment (Bovill et al., 2016; Healey & Healey, 2019). Future research should continue to embed participatory evaluation to ensure AI-integrated assessment remains student-centred and pedagogically sound.

Several limitations of the study should be acknowledged. First, students were not involved in the initial design phase of the assessment, falling short of authentic co-production (Cook-Sather et al., 2014). Although this was partly mitigated through later participatory evaluation, involving students earlier could have strengthened the creativity, relevance, and ethical responsiveness of the assessment. Second, qualitative feedback was collected following the initial pilot rather than after a full module-wide rollout. As such, findings may reflect early impressions rather than longer-term engagement. However, this timing allowed immediate adjustments and iterative revisions of the assessment materials based on student and staff experiences. Third, the study was conducted within a single institutional setting, which may limit generalisability to other universities or international contexts with different AI access, policies, and pedagogical cultures. However, the participatory methods used to co-design and co-evaluate the assessment are transferable and could be adapted to different educational settings to ensure relevance to local needs. Moreover, the assessment design mirrors real-world professional scenarios involving AI, making it relevant to clinical, research, and public health practice, and supporting the wider application of the skills developed.

5. Conclusions

This study explored the participatory development and evaluation of a generative AI-integrated assessment in postgraduate education. Findings suggest that integrating AI into assessment can promote both technical fluency and ethical reflection when scaffolded appropriately. Students engaged critically with AI outputs, while teaching staff recognised the potential for supporting critical thinking and maintaining academic integrity. Our approach supports growing calls for authentic assessment that mirrors real-world tasks, particularly in professions where AI is becoming more common. However, there remains a tension between preserving academic integrity and using AI to support skill development. Future iterations must continue to navigate this balance carefully, ensuring that critical engagement and ethical practice are at the core of AI-integrated learning.