Submitted:
23 June 2026
Posted:
24 June 2026
You are already at the latest version
Abstract
This study presents the design, implementation, and exploratory classroom evaluation of a web-based educational assistant built on Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs) for Vocational Education and Training (VET). The platform was designed to generate responses based on teacher-provided course materials, preserve source traceability, and return an abstention message when the retrieved evidence is insufficient. The assistant was deployed in an authentic classroom setting within the Higher Vocational Training programme in Network and Information Systems Administration (ASIR). Nineteen students and one instructor used the system during a practical session and completed a post-session questionnaire combining Likert-scale items with open-ended questions. The findings indicate positive student perceptions of usability, response clarity, perceived reliability, and learning support. Participants particularly valued the ability to obtain focused answers aligned with the instructional materials. The evaluation also revealed a relevant trade-off: restricting the assistant to a controlled corpus reinforced curricular consistency and perceived trustworthiness but limited its capacity to address questions insufficiently covered by the available resources. The absence of conversational memory emerged as the most frequently requested improvement. These preliminary findings suggest that course-constrained RAG assistants may constitute valuable complementary tools for transparent and pedagogically supervised AI-supported learning in technical VET contexts.
Keywords:
Retrieval-Augmented Generation (RAG)
; Large Language Models (LLMs)
; course-constrained educational assistant
; artificial intelligence in education
; semantic retrieval
; source traceability
; Vocational Education and Training (VET)
; autonomous learning
1. Introduction
1.1. Generative AI in Education: Opportunities and Challenges
The rapid adoption of Large Language Models (LLMs), particularly following the public release of ChatGPT, has generated considerable interest in their potential applications in education. These systems can produce natural-language explanations, answer questions, generate examples, and support problem-solving activities, enabling students to access information on demand and receive immediate assistance tailored to their queries [1,2].
Recent studies indicate that students increasingly use generative artificial intelligence (AI) tools as complementary learning resources. Common uses include clarifying concepts, resolving doubts, summarising information, supporting writing activities, and obtaining guidance during independent study [3,4,5]. These capabilities may be particularly valuable in technical disciplines, where students frequently interact with specialised documentation, procedural knowledge, and complex concepts that require accurate interpretation [6].
However, the educational integration of general-purpose LLMs also presents significant challenges. One of the most widely discussed limitations is their tendency to generate inaccurate, misleading, or fabricated information, commonly referred to as hallucinations [7,8]. Because these responses are often expressed fluently and confidently, students may find it difficult to identify errors, especially when they lack sufficient prior knowledge of the subject matter.
A further concern relates to curricular alignment. General-purpose LLMs draw on broad pre-trained knowledge and do not inherently distinguish between information that is relevant to a specific course and content that falls outside its intended scope. Consequently, their responses may introduce terminology, procedures, or explanations that differ from those used by the instructor or that are inconsistent with the educational materials provided to students. This issue is especially relevant in vocational and technical education, where learning activities often depend on the accurate interpretation of specific concepts, procedures, and professional documentation [9,10].
The use of generative AI in education also raises broader pedagogical and ethical concerns, including excessive reliance on automated assistance, difficulties in assessing the origin and reliability of generated content, the reproduction of biases, and potential risks to academic integrity [1,4,8]. These limitations do not negate the educational potential of LLMs, but they highlight the need for approaches that combine conversational support with greater control over the information used to generate responses.
In this context, educational assistants grounded in validated instructional resources represent a promising direction for providing students with immediate support while maintaining greater consistency with curricular objectives.
1.1. Retrieval-Augmented Generation as a Grounding Strategy
Retrieval-Augmented Generation (RAG) has emerged as a promising strategy for improving the grounding and contextual relevance of responses generated by Large Language Models (LLMs). Originally introduced by Lewis et al. [11], RAG combines the generative capabilities of a language model with access to an external knowledge repository. Rather than relying exclusively on the information encoded in the model’s parameters, the system retrieves relevant documents or text fragments at inference time and incorporates them into the context used to generate a response.
In a typical RAG workflow, a user query is transformed into a vector representation and compared with the representations of previously indexed documents. The most relevant fragments are retrieved through semantic similarity mechanisms and included in the prompt provided to the language model. This approach enables the model to generate responses grounded in a predefined corpus while also facilitating the traceability and updating of the information used during response generation [9,11].
In educational environments, this architecture offers a relevant advantage: the external repository can be restricted to instructional resources selected and validated by the teacher. Consequently, the assistant can be designed to provide responses that are more closely aligned with the course content and to indicate when the available materials do not contain sufficient information to answer a question. This form of grounding may help mitigate unsupported responses and improve users’ confidence in the information provided by the system [9,12].
Figure 1 illustrates the workflow implemented in the proposed educational assistant. Teacher-provided materials are processed, segmented, and stored as vector representations. When a student submits a question, the system retrieves the most relevant fragments and incorporates them into a controlled prompt. The language model is instructed to formulate its response on the basis of the retrieved evidence and to acknowledge when the available materials do not provide sufficient information.
RAG should not, however, be understood as a complete solution to the reliability challenges associated with LLMs. Its effectiveness depends on several factors, including the quality and coverage of the indexed materials, the relevance of the retrieved fragments, and the ability of the language model to use the retrieved evidence appropriately. If the corpus does not contain the required information or the retrieval process fails to identify the most relevant fragments, the quality of the generated response may be limited [9,12].
These characteristics make RAG particularly suitable for the development of educational assistants intended to provide immediate support while maintaining greater pedagogical control over the sources used during response generation. The present study applies this approach to a course-constrained assistant deployed in a real Vocational Education and Training classroom.
1.1. Research Gap in Vocational Education and Training
The growing interest in Retrieval-Augmented Generation (RAG) has led to the development of educational assistants designed to provide responses grounded in external instructional resources. Recent studies have examined RAG-based systems in higher education, online learning environments, and specialised educational contexts. For example, Németh et al. [13] explored an artificial intelligence-based tutor that answered students’ questions by referring to educational resources provided by course instructors. Miladi et al. [10] evaluated a RAG-enhanced conversational agent in a Massive Open Online Course (MOOC), while Thesen and Park [14] investigated the use of a constrained generative AI assistant grounded in instructor-curated materials in medical education. Systematic reviews have also highlighted the increasing relevance of RAG-based applications in education and the need to examine both their potential and their limitations in authentic learning environments [9,12].
Despite these advances, the available empirical evidence remains unevenly distributed across educational contexts. Existing studies have primarily focused on higher education, online learning, and specialised university-level programmes. By comparison, the application of course-constrained RAG assistants in Vocational Education and Training (VET) remains underexplored. This gap is relevant because VET programmes frequently involve the interpretation of technical documentation, the application of specific procedures, and the acquisition of competencies closely connected to professional practice. In such contexts, the accuracy, scope, and curricular consistency of the information provided to students are particularly important.
Research on artificial intelligence in vocational education has identified opportunities for supporting learning, personalising educational experiences, and developing professional competencies [15]. However, limited evidence is available regarding the classroom deployment of RAG-based assistants explicitly grounded in teacher-provided course materials within technical VET programmes. In particular, further research is required to understand how students and instructors perceive the usefulness, usability, and reliability of these tools, as well as the practical limitations that emerge when the available knowledge base is deliberately restricted to the instructional corpus.
To address this gap, the present study examines the design and exploratory classroom evaluation of a course-constrained RAG educational assistant in the Higher Vocational Training programme in Network and Information Systems Administration (ASIR). The system was deployed during a practical session in an authentic classroom setting and evaluated by 19 students and one instructor. Rather than assessing RAG as a novel technical architecture, this study contributes preliminary empirical evidence regarding its perceived educational value, limitations, and potential areas for improvement in a technical VET context.
1.1. Research Objective, Research Questions, and Hypothesis
Against this background, the aim of this exploratory study is to examine the feasibility and perceived educational value of a course-constrained Retrieval-Augmented Generation (RAG) assistant deployed in an authentic Vocational Education and Training (VET) classroom. The system was designed to provide students with immediate support grounded in teacher-provided course materials. The study focuses on participants’ perceptions after interacting with the assistant during a practical classroom session, as well as on the limitations and opportunities for improvement identified during its use.
The following research questions guide the study:
- RQ1. How do students perceive the usefulness, usability, clarity, and reliability of a course-constrained RAG educational assistant after using it in an authentic VET classroom?
- RQ2. To what extent do students perceive the assistant as supporting content understanding, the resolution of course-related questions, and autonomous learning?
- RQ3. What limitations and risks do students and the instructor identify during the classroom deployment of the assistant?
- RQ4. What pedagogical and technical improvements do participants suggest for future iterations of the system?
Based on previous research on educational assistants and RAG-based systems, the following exploratory hypothesis is proposed:
- H1. After the classroom session, students will report positive perceptions of the assistant’s usefulness, usability, clarity, and reliability as a complementary learning support tool.
Given the exploratory nature of the study, the hypothesis is examined descriptively rather than through causal or inferential analysis. The study does not aim to demonstrate improvements in academic performance or to quantify an objective reduction in hallucinations. Instead, it provides preliminary empirical evidence regarding the classroom feasibility, perceived educational value, and practical limitations of a course-constrained RAG assistant in a technical VET context.
The remainder of this paper is organised as follows. Section 2 reviews related work on generative AI and RAG-based educational assistants. Section 3 describes the design and implementation of the proposed system. Section 4 presents the evaluation methodology. Section 5 reports the quantitative and qualitative results. Section 6 discusses the findings in relation to the research questions, and Section 7 summarises the conclusions, limitations, and directions for future research.
1. Related Work
1.1. Generative AI Chatbots as Educational Support Tools
The emergence of Large Language Models (LLMs) has substantially expanded the capabilities of conversational systems used for educational purposes. Unlike earlier chatbots based primarily on predefined rules or scripted interactions, generative AI chatbots can provide flexible natural-language responses to a wide range of student queries. This capability has encouraged growing interest in their use as complementary tools for accessing explanations, obtaining immediate feedback, and supporting learning activities.
Recent systematic and scoping reviews have identified several recurring educational applications of generative AI chatbots. These systems are frequently used to clarify concepts, support writing and coding tasks, provide examples, assist with problem solving, and offer personalised guidance during independent study [4,5,6,16]. Their conversational nature allows students to formulate questions using natural language and receive rapid responses adapted to their immediate information needs.
Empirical evidence also indicates that students generally perceive AI chatbots as useful support tools, although their attitudes are not uniformly positive. Schei et al. [16], in a scoping review of 24 empirical studies on the use of AI chatbots in higher education, found that students valued their usefulness, immediacy, and capacity to assist with writing, coding, and academic tasks. However, the review also identified concerns regarding response accuracy, reliability, critical thinking, creativity, and the potential negative effects of excessive reliance on automated support.
Similarly, Baek et al. [3] examined the use and perceptions of generative AI among 1001 university students in the United States. Their findings revealed diverse attitudes towards ChatGPT and showed that patterns of use were influenced by factors such as students’ characteristics, academic disciplines, and institutional policies. These results suggest that the educational adoption of generative AI cannot be understood exclusively as a technological issue. Its value also depends on the educational context, the guidance provided to learners, and the conditions under which the tools are integrated into teaching practices.
Systematic reviews have also highlighted broader challenges associated with the educational use of generative AI. These include academic integrity, the risk of overreliance, the presence of inaccurate or biased information, data privacy concerns, and the need for institutional guidelines and AI literacy initiatives [4,5,6]. Consequently, the integration of conversational AI into education requires mechanisms that preserve its capacity for immediate and personalised support while addressing limitations related to reliability, transparency, and pedagogical control.
These concerns are particularly relevant when students use general-purpose chatbots to obtain information about course-specific content. Although such tools can generate plausible explanations, their responses are not inherently restricted to the materials selected by the instructor. This limitation has motivated growing interest in educational assistants grounded in predefined knowledge repositories, including systems based on Retrieval-Augmented Generation.
1.1. RAG-Based Educational Assistants
Retrieval-Augmented Generation (RAG) was originally introduced by Lewis et al. [11] as an approach that combines the generative capabilities of pre-trained language models with access to an external non-parametric memory. In contrast to models that rely exclusively on knowledge encoded in their parameters, RAG-based systems retrieve relevant information from an external repository and use it as additional context during response generation. This approach was initially developed for knowledge-intensive natural language processing tasks, where access to inspectable and updatable external information can improve the factual grounding of generated outputs.
The underlying principle has subsequently been adapted to a wide range of educational applications. In a typical educational RAG workflow, instructional materials are collected, preprocessed, divided into smaller text fragments, and converted into vector representations. When a learner submits a question, the system retrieves the fragments that are most semantically related to the query and incorporates them into the context provided to the language model. This process allows the assistant to formulate responses with reference to a predefined educational knowledge base rather than relying solely on the model’s general pre-trained knowledge [9].
This architecture offers several potential advantages in educational contexts. First, the knowledge base can be constructed from resources selected by instructors, including course notes, textbooks, slides, practical guides, and other validated materials. Second, its contents can be updated without retraining the underlying language model, enabling the system to reflect curricular changes or the incorporation of additional educational resources. Third, the retrieval process can improve transparency by allowing the information used to support an answer to be associated with identifiable source materials. These features are particularly relevant when assistants are intended to provide support that is consistent with the content taught in a specific course.
The educational adoption of RAG has expanded rapidly. Li et al. [9] identify applications related to interactive learning systems, the generation and assessment of educational content, and the integration of RAG into broader educational ecosystems. Similarly, Swacha and Gracel [12] review 47 publications describing educational RAG chatbots developed for different purposes and thematic areas. Their findings indicate growing interest in systems designed to support both students and educators through access to contextually relevant information.
However, RAG should not be interpreted as a guarantee of accuracy. The quality of the generated response depends on the completeness and timeliness of the knowledge base, the suitability of the segmentation and embedding strategies, the relevance of the retrieved fragments, and the ability of the language model to use the retrieved evidence appropriately. Retrieval failures, incomplete source materials, and insufficiently grounded generation may still lead to incomplete, inaccurate, or unsupported responses [9,12].
A further challenge concerns evaluation. Swacha and Gracel [12] observe that educational RAG chatbots can be assessed from multiple perspectives, including retrieval quality, response accuracy, usability, acceptance, and learning outcomes. Nevertheless, the literature still contains limited evidence demonstrating whether these systems achieve their intended educational objectives in authentic learning environments. This limitation reinforces the need for context-specific empirical studies that examine not only technical feasibility but also user perceptions, practical constraints, and educational value.
1.1. Empirical Applications of RAG in Educational Contexts
Beyond conceptual discussions and systematic reviews, recent empirical studies have examined how Retrieval-Augmented Generation (RAG) assistants operate in specific educational settings. These studies provide relevant evidence regarding the opportunities and limitations of grounding generative artificial intelligence systems in instructor-provided materials. However, they also reveal considerable variation in educational context, deployment strategy, and evaluation design.
Németh et al. [13] conducted a pilot study involving four courses from three academic programmes at two universities. For each course, the researchers developed an artificial intelligence tutor capable of answering students’ questions by referring to educational resources provided by the instructors. The study combined student and lecturer feedback with the qualitative and quantitative analysis of interaction records. Both students and lecturers reported positive experiences, although the authors observed that the same tool could function differently depending on the course context, learning objectives, and instructional materials. Based on expert coding of a random sample of question–answer pairs, 1.5% of the responses were classified as incorrect and 16.5% as outside the context provided to the language model. The authors concluded that RAG can reduce hallucinations when the relevant topics are sufficiently covered in the available materials, while also emphasising the importance of corpus quality and pedagogical context.
Miladi et al. [10] evaluated a conversational agent enhanced with GPT-4 and RAG in a Massive Open Online Course (MOOC). The system retrieved information from course documents to provide learners with real-time, contextually relevant support. The evaluation involved 25 learners and compared an experimental group using the conversational agent with a control group. The authors reported a statistically significant knowledge gain in the experimental group and a high System Usability Scale score. This study provides evidence of the potential educational value of RAG-enhanced conversational agents in online learning environments, particularly where learners have limited access to personalised instructor support.
A further relevant contribution is the study by Thesen and Park [14], who deployed a RAG-based teaching assistant in a medical school basic science course across two consecutive cohorts. The assistant generated responses grounded in instructor-curated materials and was designed to support self-directed learning. Usage patterns showed that students used the system strategically rather than continuously, with engagement increasing during the periods preceding examinations and with substantial use outside regular teaching hours. Students valued the immediate availability of source-grounded responses, but the study also identified a recurring tension between reliability and comprehensiveness. The same knowledge-base constraints that increased confidence in the system limited its capacity to address broader questions, leading some students to perceive the assistant as insufficiently comprehensive.
Taken together, these studies illustrate several important characteristics of educational RAG systems. First, grounding responses in instructional materials can contribute to greater curricular consistency and perceived trustworthiness. Second, the quality and coverage of the knowledge base play a central role in determining the usefulness of the assistant. Third, the educational impact of these tools cannot be assessed exclusively through technical performance metrics: their value also depends on the learning context, the way students incorporate them into their study practices, and the balance between controlled responses and sufficient informational coverage.
Nevertheless, the existing empirical evidence remains concentrated primarily on higher education, online learning, and specialised university-level programmes. Comparatively limited attention has been devoted to the classroom deployment of course-constrained RAG assistants in technical Vocational Education and Training (VET). The present study extends this emerging line of research by examining students’ and instructor perceptions after the use of a RAG educational assistant in an authentic classroom session within the Higher Vocational Training programme in Network and Information Systems Administration (ASIR).
1.1. Positioning of the Present Study
The empirical studies reviewed above demonstrate that Retrieval-Augmented Generation (RAG) can be integrated into educational assistants across different learning environments. However, their objectives, deployment conditions, and evaluation strategies vary considerably. Some studies focus primarily on technical reliability and interaction patterns, whereas others examine usability or learning outcomes through experimental designs. Table 1 summarises selected empirical studies that are particularly relevant to the present research. The comparison is intended to clarify the positioning of this study rather than to provide an exhaustive review of educational RAG applications.
As shown in Table 1, the present study does not propose RAG as a novel technical architecture. Its contribution lies in the exploratory deployment and evaluation of a course-constrained educational assistant in a technical Vocational Education and Training (VET) classroom. This context remains comparatively underrepresented in the empirical literature, despite the importance of accurate and curriculum-aligned information in programmes where students engage with specialised documentation, technical procedures, and professionally oriented competencies.
The study also differs from previous work in its evaluation focus. Unlike experimental research designed to measure learning gains, the present study examines how students and an instructor perceive the usefulness, usability, clarity, and reliability of the assistant after interacting with it during an authentic practical session. The analysis also considers the limitations identified by participants, including the restricted coverage of the instructional corpus and the absence of conversational memory.
Accordingly, the study provides preliminary evidence regarding the classroom feasibility and perceived educational value of course-constrained RAG assistants in technical VET settings. Its findings should be interpreted as exploratory and context-specific rather than as evidence of causal effects on learning outcomes or as an objective assessment of response accuracy.
1. Course-Constrained RAG Educational Assistant
1.1. Design Principles and System Architecture
The proposed educational assistant was developed as a web-based platform designed to provide students with immediate access to course-related support while maintaining pedagogical control over the information used during response generation. Rather than offering unrestricted interaction with a general-purpose Large Language Model (LLM), the system adopts a course-constrained Retrieval-Augmented Generation (RAG) approach in which responses are grounded in instructional materials selected and uploaded by the teacher.
The design of the platform was guided by four main principles. First, the assistant was intended to promote curricular alignment by retrieving information from a predefined repository of course materials. This approach reduces the likelihood of introducing explanations, procedures, or terminology that fall outside the instructional context. Second, the system was designed to improve transparency by preserving the source metadata associated with the retrieved text fragments. This enables users to consult the original educational resources supporting a generated response. Third, the platform incorporates an abstention mechanism: when the retrieved materials do not provide sufficient evidence to answer a question, the assistant informs the user that the requested information could not be located within the available course content. Finally, the application supports pedagogical oversight through role-based access, allowing instructors to manage resources and review student interactions.
The system follows a modular three-layer architecture complemented by external artificial intelligence services, as illustrated in Figure 2.
The presentation layer provides the web-based interface through which students, instructors, and administrators interact with the platform. Students can submit natural-language questions, receive generated responses, access the associated source materials, and review their previous standalone interactions. Instructors and administrators access additional management functionalities according to their assigned roles.
The application layer implements the core logic of the platform. Developed in PHP, it manages authentication, role-based access, classes, educational resources, document processing, query handling, and communication with external artificial intelligence services. When a student submits a question, this layer coordinates the generation of the query embedding, the retrieval of relevant instructional fragments, the construction of the controlled prompt, and the delivery of the generated response to the user interface.
The data and knowledge layer stores the information required for the operation of the assistant. This includes user accounts, roles, classes, uploaded documents, segmented text fragments, embeddings, source metadata, and interaction histories. The association between each fragment and its original document preserves traceability throughout the retrieval and response-generation process.
The platform integrates two external artificial intelligence services through the OpenAI API. The first generates embeddings for both indexed text fragments and user queries. The second generates natural-language responses from the user question, the retrieved evidence, and the control instructions defined by the application. The API therefore functions as an integrated service within the workflow rather than as an autonomous educational tool: the application determines which materials are processed, which fragments are retrieved, and which contextual evidence is provided to the language model.
This architecture separates user interaction, application logic, knowledge management, and external AI processing responsibilities. The following sections describe the ingestion of instructional materials, the semantic retrieval workflow, the controlled generation of responses, and the role-based functionalities provided by the platform.
1.1. Knowledge Ingestion and Semantic Indexing
The knowledge ingestion pipeline transforms teacher-provided instructional materials into a structured repository suitable for semantic retrieval. This process is performed whenever new educational resources are incorporated into the platform and provides the knowledge base used by the course-constrained Retrieval-Augmented Generation (RAG) workflow.
The platform supports commonly used instructional document formats, including PDF, DOCX, and TXT files. These resources may include course notes, technical guides, practical exercises, and other materials selected by the instructor. Once a document is uploaded, the backend automatically extracts its textual content and prepares it for subsequent processing.
The extracted content is segmented into smaller fragments, or chunks, using document-aware units. TXT files are processed by line, DOCX files by paragraph and detectable page breaks, and PDF files by page. The current implementation groups these units into fragments targeting approximately 1200 characters, with an overlap of approximately 150 characters between consecutive chunks. This strategy preserves contextual continuity while retaining references to the original location of the extracted content. Since document units may vary in length, the resulting chunk size is approximate rather than strictly fixed.
This configuration seeks to balance retrieval precision and contextual continuity. Smaller fragments can facilitate the identification of content directly related to a student’s question, whereas excessively short segments may omit information required to interpret the retrieved passage correctly. For this reason, the system preserves sufficient surrounding text while maintaining a level of granularity suitable for similarity-based retrieval.
Each generated fragment remains associated with its original source document and its corresponding course context. The platform also stores a sequential identifier and, when available, information about the original location of the fragment within the educational resource. These metadata support traceability and allow the interface to provide direct references to the instructional materials used during response generation.
Following segmentation, each text fragment is transformed into a numerical vector representation using the text-embedding-3-small model [17]. These embeddings encode semantic relationships between texts and allow the system to retrieve relevant content even when the wording of a student’s question does not exactly match the terminology used in the source materials. The generated vectors are stored together with the corresponding fragments and metadata in the knowledge repository.
The ingestion pipeline therefore produces a searchable, source-aware educational knowledge base. This repository constitutes the foundation of the subsequent retrieval process, in which student queries are compared with the indexed fragments to identify the contextual evidence provided to the language model.
1.1. Semantic Retrieval and Controlled Response Generation
When a student submits a question, the platform initiates a retrieval and response-generation workflow designed to identify relevant instructional evidence and provide a course-aligned answer. Each query is processed independently, without incorporating previous questions or responses into the generation context.
The first step consists of transforming the student’s question into a vector representation using the text-embedding-3-small model [17]. The resulting query embedding is compared with the embeddings associated with the instructional fragments stored for the selected class. In the current implementation, the comparison is performed locally within the application backend using cosine similarity.
The system ranks the available fragments according to their cosine similarity scores and selects the five highest-ranked results. Each retrieved fragment includes its textual content, the name of the source document, and the available reference metadata, such as page numbers, paragraph ranges, or line ranges. These fragments constitute the contextual evidence supplied to the language model. As illustrated in Figure 3, the workflow continues with the construction of a controlled prompt, response generation, and the presentation of either an answer based on the retrieved evidence or an abstention message.
The response is generated through the OpenAI API using the gpt-4o-mini model [18] with a temperature value of 0.2. The prompt instructs the model to formulate its answer exclusively from the retrieved fragments and to respond with a predefined abstention message when the available evidence does not contain sufficient information. The assistant is also instructed to provide clear and didactic responses in Markdown format and to associate factual statements with numbered source references.
The current prototype implements prompt-based abstention rather than a deterministic similarity threshold. The five highest-ranked fragments are provided to the language model for each query, and the model is instructed to indicate when they do not support an answer. Consequently, the quality of the response depends not only on the language model but also on the relevance of the retrieved fragments and the coverage of the instructional corpus.
After the answer has been generated, the application identifies the numbered references included in the response and maps them to the corresponding source documents and locations. The student interface can therefore display the educational resources associated with the generated answer. This traceability mechanism allows learners to consult the original materials and provides an additional layer of transparency.
The platform also stores each question, generated response, and associated source references in an interaction history. Students can review their previous standalone queries, while instructors can examine the questions submitted within their classes. However, these stored interactions are not currently incorporated into subsequent prompts. The present version of the assistant therefore supports query history but does not maintain conversational memory across successive turns.
1.1. User Roles and Implementation Summary
The platform incorporates role-based interfaces designed to support student interaction, instructional resource management, and pedagogical oversight. Three user roles are currently implemented: student, instructor, and administrator.
Students access the assistant through the web interface associated with their assigned class. They can submit natural-language questions, receive responses based on the retrieved instructional evidence, consult the associated source references, and review their previous standalone interactions. The stored history enables students to return to earlier questions and responses, although it is not used as conversational context for subsequent queries.
Instructors can upload and reprocess educational materials associated with their class. They can also review the questions submitted by students, together with the corresponding responses and citation metadata. This functionality provides visibility into the information needs expressed during use of the assistant and may help instructors identify recurring questions or concepts that require further clarification in class.
Administrators have access to broader management functionalities, including user, class, document, and query-history administration. The platform also allows uploaded documents to be reprocessed when the extraction logic, segmentation strategy, or embedding configuration is modified. Reprocessing regenerates the document fragments and their associated vector representations without requiring the original file to be uploaded again.
The implementation was deliberately designed as a web-based application requiring no local installation on student devices. The use of a browser-based interface facilitates access from different operating systems and devices while centralising document management, semantic indexing, and interaction histories. Table 2 summarises the main implementation characteristics of the prototype.
The resulting platform combines semantic retrieval, controlled response generation, source-aware interaction, and role-based management within a lightweight architecture intended for classroom use. The next section describes the methodology followed to evaluate the prototype in an authentic Vocational Education and Training context.
1. Evaluation Methodology
1.1. Research Design
A descriptive and exploratory case-study design was adopted to examine the feasibility and perceived educational value of the proposed course-constrained Retrieval-Augmented Generation (RAG) assistant in an authentic Vocational Education and Training (VET) classroom. The evaluation focused on participants’ perceptions after interacting with the system during a practical session.
The study combined quantitative and qualitative data collection techniques. Quantitative data were obtained through a post-session questionnaire comprising five-point Likert-scale items designed to examine perceived usefulness, usability, response clarity, reliability, and learning support. Qualitative data were collected through open-ended questions that allowed participants to describe positive aspects of the system, identify limitations and perceived risks, and suggest improvements for future iterations.
The quantitative and qualitative components were analysed complementarily. The structured items provided descriptive indicators of participants’ perceptions, while the open-ended responses offered additional context for interpreting those results and identifying issues that could not be captured through numerical scores alone. This approach was selected to address the research questions from both a descriptive and an interpretative perspective.
The study was not designed as an experimental evaluation. It did not include a control group, a pre-test or post-test of academic knowledge, or an objective assessment of response accuracy. Consequently, the findings should not be interpreted as evidence of causal effects on learning outcomes or as a quantitative demonstration that the system reduces hallucinations. Instead, the evaluation provides preliminary empirical evidence regarding classroom feasibility, perceived educational value, and practical limitations.
The exploratory hypothesis proposed in Section 1.4 is therefore examined descriptively through the questionnaire responses rather than through inferential statistical testing.
1.1. Educational Context and Participants
The study was conducted in March 2026 at IES Rosa Chacel, a public secondary education institution located in Madrid, Spain. The intervention took place within the Higher Vocational Training programme in Network and Information Systems Administration (ASIR), specifically in the Network Services and Internet module.
The module provides an appropriate context for evaluating a course-constrained Retrieval-Augmented Generation (RAG) assistant because students work with specialised technical documentation and apply theoretical knowledge to practical activities related to the configuration, administration, and management of network services. In this setting, learners frequently need to consult instructional resources, review technical concepts, and resolve questions arising during hands-on activities. The accuracy and curricular consistency of the information provided by an educational assistant are therefore particularly relevant.
A total of 20 participants took part in the classroom activity: 19 students enrolled in the module and the instructor responsible for the course. The sample was selected through intentional convenience sampling, as the participating group was directly connected to the instructional content incorporated into the assistant and formed part of the authentic teaching context in which the intervention was conducted.
Participation in the study was voluntary. The students attended the practical session, interacted with the assistant, and completed the post-session questionnaire. The instructor also used the platform during the activity and completed the same questionnaire from a teaching perspective, providing complementary evidence regarding the suitability of the generated responses and the potential educational value of the system.
Given the different roles of students and the instructor, their responses are considered separately in the analysis. The student questionnaire responses constitute the primary quantitative dataset used to examine the exploratory hypothesis and the student-focused research questions. The instructor’s response is treated as complementary contextual evidence and is considered alongside the qualitative findings.
1.1. Classroom Procedure
The classroom evaluation was organised in three phases: preparation of the educational assistant, practical interaction with the platform, and post-session data collection.
During the preparation phase, the course-constrained Retrieval-Augmented Generation (RAG) assistant was configured for use in the Network Services and Internet module. The instructional materials selected by the instructor were uploaded and processed to create the class-specific knowledge repository. Prior to the classroom deployment, technical tests were conducted to verify that the platform was functioning correctly and that the generated responses were connected to the available course materials.
The classroom intervention took place during a single session lasting approximately one hour. At the beginning of the activity, participants received a brief explanation of the purpose of the study and the basic operation of the assistant. Students accessed the platform using randomly generated credentials to prevent individual identification during the interaction and subsequent data collection.
Participants were then invited to use the assistant freely to formulate questions related to the module content. They could request clarifications, review technical concepts, and ask for explanations or examples associated with the instructional materials. The objective was to reproduce a realistic learning-support scenario rather than to impose a predefined sequence of questions or tasks.
The instructor also interacted with the assistant during the session from a teaching perspective. This enabled the instructor to examine the alignment of the responses with the course content and to provide complementary feedback regarding the potential educational value and limitations of the system.
During the final 15 minutes of the session, participants completed the online questionnaire voluntarily and anonymously. The questionnaire was administered immediately after the interaction period so that the responses reflected participants’ direct experience with the platform.
1.1. Data Collection Instrument
Data were collected through an online post-session questionnaire designed to examine participants’ perceptions after interacting with the educational assistant. The questionnaire was administered using Microsoft Forms and did not collect personally identifiable information.
The instrument consisted of two complementary sections. The first section included 16 statements rated on a five-point Likert scale, where 1 indicated strongly disagree and 5 indicated strongly agree. The items were designed to examine five thematic areas relevant to the classroom use of the assistant: usability, learning support and perceived autonomy, clarity and perceived reliability, perceived risks and limitations, and reflective use and source verification.
The usability items focused on ease of use, the clarity of the interface, and the need for technical assistance. The learning-support-and-perceived-autonomy items examined whether the assistant helped participants understand course content, resolve subject-related questions, obtain useful explanations, access clarifications rapidly, and study more independently. The clarity-and-perceived-reliability items addressed the comprehensibility of the generated responses, participants’ perceptions of their correctness, and the confidence associated with grounding answers in teacher-provided course materials. The perceived-risks-and-limitations items examined restricted corpus coverage, possible overreliance on the assistant, and the occurrence of incomplete or unhelpful responses. Finally, the reflective-use-and-source-verification items explored whether the assistant encouraged participants to consult the original instructional resources and formulate more effective questions.
Table 3.
Thematic organisation of the Likert-scale questionnaire items.
| Thematic area | Questionnaire items | Number of items |
|---|---|---|
| Usability | 4a–4c | 3 |
| Learning support and perceived autonomy | 5a–5e | 5 |
| Clarity and perceived reliability | 6a–6c | 3 |
| Perceived risks and limitations | 7a–7c | 3 |
| Reflective use and source verification | 8a–8b | 2 |
| Total | 16 |
The second section contained three open-ended questions inviting participants to identify positive aspects of the system, describe difficulties encountered during use, and suggest improvements for future versions. These questions were intended to capture strengths, limitations, and design priorities that could not be fully represented through closed-ended items.
The questionnaire was developed as an exploratory data-collection instrument for the present case study. The thematic grouping of items was used to organise the descriptive analysis and should not be interpreted as a psychometrically validated measurement model. The complete questionnaire is provided as Supplementary Material to facilitate transparency and reproducibility.
1.1. Data Analysis
The quantitative and qualitative data were analysed descriptively in accordance with the exploratory nature of the study. The primary quantitative analysis focused on the responses provided by the 19 students. Although the instructor completed the same questionnaire after interacting with the platform, the instructor’s response was examined separately as complementary contextual evidence because the questionnaire items were primarily formulated from the student perspective.
The Likert-scale responses were coded numerically from 1 (strongly disagree) to 5 (strongly agree). For each item, the mean, standard deviation, and distribution of responses were calculated. Item-level results were retained as the primary form of analysis to preserve the meaning of the individual statements and avoid overinterpreting the thematic groupings as validated psychometric scales.
Descriptive grouped scores were also calculated for the positively oriented thematic areas to facilitate interpretation. The negatively worded usability item concerning the need for technical assistance was reverse-coded before calculating the usability summary score. The three items addressing perceived risks and limitations were analysed individually rather than combined into a single aggregated score, as they represent distinct issues: restricted corpus coverage, possible overreliance on the assistant, and incomplete or unhelpful responses.
No inferential statistical tests were conducted because the study was designed as an exploratory classroom case study with a small convenience sample and did not include a control group or a pre-test/post-test design. The quantitative results are therefore interpreted as descriptive evidence of students’ perceptions rather than as generalisable estimates or evidence of causal effects.
The responses to the open-ended questions were reviewed and organised into recurring thematic categories. The analysis focused on identifying perceived strengths, practical limitations, and suggested improvements. Particular attention was paid to comments concerning course-material grounding, response clarity, conversational memory, corpus coverage, source traceability, and the handling of ambiguous or broad queries. The instructor’s open-ended responses were considered separately and used to complement the interpretation of the student feedback.
Participant quotations were translated from Spanish into English by the authors. Where minor edits are required to improve readability, these should preserve the original meaning and be indicated appropriately. Quotations are used illustratively rather than as evidence of the frequency or prevalence of a theme.
1.1. Ethical Considerations
The study was approved by the Research Ethics Committee of Universidad Rey Juan Carlos (URJC) (protocol code 031220250052026; date of approval: 9 March 2026). The educational institution in which the classroom activity was conducted also provided written authorisation for the collection of anonymised data.
Participation in the study was voluntary. Before the classroom session, participants received an information sheet explaining the purpose of the research, the activities involved, the use of the collected data, the measures adopted to preserve confidentiality, and their right to withdraw from the study without academic consequences. Written informed consent was obtained from all participants.
The questionnaire and the dataset used for the analysis did not contain personally identifiable information. Students accessed the platform using randomly generated credentials to prevent the association of individual interactions with identifiable participants. Signed consent forms were stored separately from the anonymised research dataset.
Open-ended responses were reviewed before publication to remove any contextual information that could permit indirect identification. The collected data were used exclusively for academic and educational research purposes.
1. Results
1.1. Participant Profile
A total of 20 participants took part in the classroom activity: 19 students enrolled in the Network Services and Internet module and the instructor responsible for the course. As explained in Section 4.5, the quantitative results reported in the following sections are based on the 19 student responses. The instructor’s questionnaire response is considered separately as complementary contextual evidence.
To preserve anonymity, the questionnaire did not collect personal demographic information such as age, gender, or academic record. The participant profile is therefore described in terms of role and previous experience with generative artificial intelligence tools.
The students reported a high degree of familiarity with tools such as ChatGPT, Gemini, and Copilot. Fifteen students (78.9%) indicated that they had used generative AI tools very often, while the remaining four students (21.1%) reported using them quite often. None of the students selected the less frequent response categories.
The use of generative AI as a learning resource was also widespread. Twelve students (63.2%) stated that they used these tools very often to resolve doubts or learn new content, six students (31.6%) selected quite often, and one student (5.3%) selected sometimes. These results indicate that the participants were already familiar with AI-assisted learning before the classroom deployment of the proposed assistant.
This prior experience provides relevant context for interpreting the subsequent findings. On the one hand, participants were able to compare the proposed course-constrained assistant with general-purpose AI tools that they already used regularly. On the other hand, their familiarity with generative AI may have influenced their expectations and their generally positive reception of the system.
1.1. Quantitative Results
Table 4 presents the descriptive results obtained from the 19 student questionnaires. All Likert-scale items received complete responses. The results show generally positive perceptions of the assistant, particularly in relation to usability, learning support, response clarity, and perceived reliability.
Usability received particularly positive ratings. All students agreed or strongly agreed that the assistant was easy to use (M = 4.63, SD = 0.50), and none indicated that they required technical or external assistance (M = 1.32, SD = 0.48). The clarity and comfort of the interface also received a favourable assessment (M = 4.05, SD = 0.71).
The results related to learning support were similarly positive. The highest-rated items concerned the provision of clear and useful explanations (M = 4.47, SD = 0.61), the resolution of course-related questions (M = 4.42, SD = 0.61), and the rapid provision of summaries or clarifications (M = 4.42, SD = 0.69). Students also expressed a positive perception of the assistant’s potential to support more autonomous study practices (M = 4.37, SD = 0.76).
The clarity and perceived reliability of the generated responses were also assessed favourably. All students agreed or strongly agreed that the responses seemed correct most of the time (M = 4.53, SD = 0.51), while 18 students (94.7%) considered them clear and easy to understand (M = 4.32, SD = 0.58). These results refer to students’ perceptions after using the system and should not be interpreted as an objective assessment of response accuracy. Grounding responses in course materials also increased confidence among most participants (M = 4.21, SD = 1.03), although the greater variability observed for this item indicates that this perception was not uniform across the group.
The negatively worded items reveal several practical limitations. Ten students (52.6%) agreed or strongly agreed that the assistant was sometimes unable to answer because it was restricted to the syllabus (M = 3.79, SD = 1.08). Perceptions regarding possible overreliance were more divided: seven students (36.8%) expressed concern that excessive dependence on the assistant could affect their study habits, while six students (31.6%) disagreed with this statement and six (31.6%) remained neutral. Unhelpful or incomplete responses were not perceived as the dominant experience, although five students (26.3%) reported encountering them.
Finally, the results indicate a moderately positive contribution to reflective use. Eleven students (57.9%) agreed or strongly agreed that the assistant encouraged them to consult the original course materials, and the same number considered that interacting with the system helped them formulate better questions. These results suggest that the assistant may support not only rapid access to information but also more reflective learning practices, although this effect was less pronounced than the perceived benefits related to usability and doubt resolution.
For interpretative purposes, descriptive grouped scores were also calculated for the positively oriented thematic areas. The usability score, calculated after reverse-coding item 4b, obtained the highest mean (M = 4.46, SD = 0.39), followed by learning support and perceived autonomy (M = 4.38, SD = 0.39), clarity and perceived reliability (M = 4.35, SD = 0.49), and reflective use and source verification (M = 3.74, SD = 0.75). These grouped values are used only as descriptive summaries and should not be interpreted as psychometrically validated scales.
1.1. Qualitative Findings
The open-ended responses provided additional context for interpreting the quantitative results. Overall, the student comments reflected a positive perception of the assistant, particularly in relation to course-material grounding, clarity, and ease of access to relevant information. At the same time, the responses revealed several practical limitations and priorities for future development.
Table 5 summarises the main themes identified in the student feedback. The quotations were translated from Spanish into English by the authors. Minor punctuation adjustments were made where necessary to improve readability without altering the original meaning.
1.1.1. Course-Material Grounding and Focused Learning Support
One of the most frequently valued characteristics of the assistant was its reliance on teacher-provided course materials. Students perceived this restriction as an advantage over general-purpose artificial intelligence tools because it reduced the likelihood of receiving explanations or procedures that differed from those used during classroom instruction.
Several comments explicitly highlighted this aspect. One student stated: “I like that it works only with the notes we provide.” Another valued the fact that the assistant “only extracts information from the syllabus previously stored by the teacher.” A further participant explained that restricting the system to the course materials was useful because general-purpose AI tools sometimes propose solutions that differ from those taught in class.
Students also appreciated the clarity, conciseness, and organisation of the responses. The feedback included comments such as: “It gives me a specific answer to my question”; “It does not take long to respond, and it focuses on the notes”; and “It does not go on at length; it gets straight to the point, which saves you time and visual fatigue.” These observations are consistent with the positive quantitative ratings obtained for doubt resolution, response clarity, and perceived usability.
Some participants also referred positively to the presentation of the information and the source references included at the end of the answers. These comments suggest that the value of the system was not limited to rapid response generation: students also appreciated receiving structured answers connected to the instructional resources used in the module.
1.1.1. Conversational Continuity as the Main Improvement Priority
The most recurrent improvement request concerned the absence of conversational memory. Eleven of the 19 students explicitly suggested that the assistant should preserve the context of previous interactions, support follow-up questions, or allow users to resume earlier conversations.
Students described the current system as closer to a standalone query interface than to a continuous chat. One participant suggested that “the chat should be saved and continue, so that the previous question and answer are not deleted.” Another requested “memory within the chat and the ability to link one question to another.” A further student explained that the tool should allow users to “continue a previous query, or extend it instead of having only one response per question.”
This limitation is particularly relevant in educational contexts. Learning-related questions frequently emerge progressively: an initial answer can generate a follow-up doubt, require clarification, or lead the learner to explore a related concept. The absence of conversational continuity therefore limits the natural development of these interactions, even though students can consult a stored history of previous standalone questions and responses.
1.1.1. Corpus Coverage, Response Boundaries, and Additional Improvements
The responses also revealed a trade-off associated with restricting the assistant to teacher-provided resources. Although this design decision was widely valued, some students noted that it could prevent the system from resolving technical questions when the relevant information was not included in the available materials. One participant summarised this limitation by stating that the assistant was “very limited to the syllabus.” Another observed that an unexpected technical error not covered by the notes could leave the learner unable to progress with a practical activity.
A related concern involved the boundaries of the restricted corpus. One student reported that the assistant correctly abstained when a question was clearly unrelated to the course content, but occasionally answered questions that were closely related to the topic even when the relevant information was not contained in the syllabus. Other participants noted that broad or insufficiently specific questions could lead to partial answers or omit relevant procedural details. These comments indicate that future versions should improve the handling of ambiguous queries and the distinction between sufficiently supported and insufficiently supported responses.
Students also proposed several additional enhancements. These included allowing users to attach screenshots, images, or supplementary documents to their queries; providing links that navigate directly to the relevant location within the referenced source; improving the visual organisation of the interface and the accessibility of the interaction history; and enabling optional mechanisms for detecting potentially outdated course materials. Some participants also identified limitations when requesting extensive multiple-choice tests or more creative practical exercises.
Taken together, the qualitative findings reinforce the positive perceptions observed in the Likert-scale responses while revealing concrete priorities for future development. The most important priorities are the incorporation of conversational memory, improved handling of broad or ambiguous questions, more robust response-boundary control, and enhanced interaction with source materials.
1.1. Instructor Feedback
The instructor’s response was analysed separately from the student questionnaire data because the instrument was primarily designed to examine the student experience. Although the feedback represents the perspective of a single instructor and should not be generalised, it provides complementary contextual evidence regarding the educational suitability of the assistant and potential priorities for future development.
The instructor assessed the overall experience positively and particularly valued the presentation of the generated content, stating that “the responses were very well summarised and structured.” This observation is consistent with the student feedback regarding the clarity, conciseness, and organisation of the answers.
The instructor also identified two areas for improvement. First, the instructor suggested enabling users to upload additional documentation created or selected by the user. This functionality could extend the scope of the assistant for specific learning activities while preserving the distinction between official course materials and supplementary resources.
Second, the instructor noted that the system did not provide a useful answer when presented with an ambiguous question and suggested that it could request additional details before generating a response. This observation highlights the value of incorporating clarification mechanisms for queries that are insufficiently specific. Rather than immediately returning an abstention message or producing a partial answer, a future version of the assistant could guide the learner towards reformulating the question.
Although based on a single teaching perspective, these observations reinforce issues also identified in the student feedback: the need to improve conversational interaction, extend the controlled use of supplementary materials, and handle broad or ambiguous queries more effectively.
Overall, the results reveal a favourable perception of the proposed assistant as a course-aligned learning support tool. Students particularly valued its ease of use, the clarity and structure of the generated responses, and its ability to provide focused answers based on teacher-provided materials. At the same time, the findings identify a relevant design trade-off: restricting the assistant to the instructional corpus increases curricular consistency and perceived trustworthiness, but may limit its capacity to address questions that are broad, ambiguous, or insufficiently covered by the available resources. The absence of conversational memory emerged as the most frequently requested improvement, while both students and the instructor also highlighted the need for more effective handling of ambiguous queries and supplementary materials. These findings are interpreted in relation to the research questions in the following section.
1. Discussion
1.1. Perceived Usefulness, Usability, and Reliability: RQ1 and H1
The first research question examined how students perceived the usefulness, usability, clarity, and reliability of the course-constrained Retrieval-Augmented Generation (RAG) assistant after using it in an authentic Vocational Education and Training (VET) classroom. Overall, the findings provide preliminary descriptive support for H1. Students reported positive perceptions across the four aspects considered, particularly in relation to ease of use, the clarity of the generated responses, and their perceived correctness.
Usability emerged as one of the strongest aspects of the prototype. All 19 students agreed or strongly agreed that the assistant was easy to use, and none indicated that technical or external assistance was required. These results suggest that the browser-based interface and the straightforward question-and-answer workflow enabled students to interact with the system without encountering significant access barriers. This is particularly relevant for the classroom integration of educational AI tools, as their potential value may be reduced if students require extensive training or technical support before they can use them effectively.
The clarity of the generated responses was also assessed positively. Most students considered the answers easy to understand and valued their concise and structured presentation. The open-ended comments reinforce this interpretation: participants highlighted that the assistant provided focused answers, avoided unnecessarily lengthy explanations, and presented information in a way that facilitated rapid consultation. The instructor similarly observed that the responses were well summarised and structured. These findings are consistent with previous research showing that students often value AI-based educational assistants for their immediacy, accessibility, and ability to provide targeted support for academic tasks [16].
The results also indicate a positive perception of reliability. All students agreed or strongly agreed that the responses seemed correct most of the time. However, this result must be interpreted carefully. The study did not include an expert assessment of response accuracy or a comparison with a general-purpose Large Language Model (LLM). Consequently, the findings do not demonstrate that the system objectively generated correct answers or reduced hallucinations. They show that students perceived the responses as generally reliable after interacting with the prototype during the classroom session.
Grounding the responses in teacher-provided materials appears to have contributed to this perceived trustworthiness. Most students reported that knowing the assistant used the course materials increased their confidence in the system. The qualitative responses provide further context: several participants valued receiving explanations aligned with the procedures and terminology used in class rather than alternative solutions drawn from unrestricted external sources. This result is consistent with the rationale underlying course-constrained RAG systems: restricting the retrieval repository to instructor-selected materials can improve curricular consistency and make the origin of the information more transparent.
Similar observations have been reported in previous educational deployments of RAG-based assistants. Németh et al. [13] found that grounding an AI tutor in lecturer-provided educational resources contributed to positive student and instructor experiences while also reducing unsupported responses when the relevant content was sufficiently represented in the available materials. In medical education, Thesen and Park [14] similarly observed that source-grounded responses could reinforce students’ trust in the system, while also identifying a tension between reliability and comprehensiveness when learners expected answers beyond the boundaries of the curated corpus.
The present findings reproduce this tension in a technical VET setting. Although most students valued the use of official course materials, the confidence item showed greater variability than the other positively oriented items. The open-ended responses suggest that students recognised an important limitation: the quality of the assistant ultimately depends on the quality, completeness, and currency of the instructional resources incorporated into the repository. Restricting the system to a controlled corpus therefore improves pedagogical alignment, but it does not guarantee that every answer will be complete or objectively correct.
The students’ extensive previous experience with general-purpose AI tools also provides relevant context for interpreting these results. Most participants regularly used tools such as ChatGPT, Gemini, or Copilot before the intervention. Their positive assessment may therefore reflect not only the usability of the proposed assistant but also their ability to compare its course-specific behaviour with the broader and less controlled responses commonly produced by general-purpose systems. At the same time, this familiarity limits the extent to which the findings can be transferred to learners with little or no prior experience of generative AI.
Taken together, the results provide preliminary descriptive support for H1: after the classroom session, students perceived the assistant as a useful, accessible, clear, and generally reliable complementary learning-support tool. Nevertheless, the findings concern perceived reliability rather than objectively measured response accuracy. Future studies should complement user perceptions with expert evaluation of generated responses and controlled comparisons with unconstrained LLM-based tools.
1.1. Perceived Learning Support and Autonomous Learning: RQ2
The second research question examined the extent to which students perceived the assistant as supporting content understanding, the resolution of course-related questions, and more autonomous learning practices. The findings indicate that the strongest perceived contribution of the system concerned immediate learning support. Students valued the possibility of obtaining focused explanations and clarifications without manually searching through the complete set of instructional materials.
The quantitative results show consistently positive ratings across the learning-support items. Students particularly valued the provision of clear and useful explanations (M = 4.47, SD = 0.61), the resolution of course-related questions (M = 4.42, SD = 0.61), and the rapid access to summaries or clarifications (M = 4.42, SD = 0.69). The assistant was also perceived as helpful for improving understanding of the course content (M = 4.21, SD = 0.71). These findings are consistent with the qualitative responses, in which students highlighted the value of receiving concise, structured, and specific answers connected to their immediate doubts.
This type of support is particularly relevant in technical Vocational Education and Training (VET). Students in the Network Services and Internet module work with specialised documentation, practical procedures, and configuration tasks that may generate specific questions during learning activities. In this context, an assistant capable of locating relevant information within the course materials can reduce the effort required to search manually through multiple resources and provide timely guidance when a doubt arises. The system therefore acts as a complementary access layer to the instructional corpus rather than as a replacement for the original materials or for teacher support.
Previous studies have similarly highlighted the potential of educational chatbots to provide immediate, context-sensitive assistance. Schei et al. [16] identified immediacy and accessibility as recurring factors contributing to students’ positive perceptions of generative AI tools in educational contexts. Miladi et al. [10] also reported the potential of a RAG-enhanced conversational agent to support learning activities in an online environment, while Németh et al. [13] emphasised that the usefulness of RAG-based tutoring systems depends on the relationship between the learner’s question and the available course resources. The present study extends these observations to an authentic technical VET classroom.
The results also indicate a positive perception of the assistant’s potential to support more autonomous study practices. Sixteen of the 19 students agreed or strongly agreed that the system could help them study more independently (M = 4.37, SD = 0.76). This finding suggests that participants recognised value in being able to consult the assistant without requiring immediate teacher intervention for every question. However, the result should be interpreted as a perception of potential rather than as evidence of increased learner autonomy. The classroom evaluation was limited to a single session and did not examine whether students subsequently used the tool independently, improved their study habits, or achieved better academic outcomes.
A more nuanced pattern emerges from the items related to reflective use and source verification. Eleven students agreed or strongly agreed that the assistant encouraged them to consult the original course materials (M = 3.79, SD = 1.18), and the same number considered that interacting with the system helped them formulate better questions (M = 3.68, SD = 0.82). Although these results are moderately positive, they are lower than the scores obtained for immediate doubt resolution and explanation quality. This difference suggests that the assistant was perceived primarily as a rapid consultation tool rather than as an instrument that automatically promotes reflective learning behaviours.
The distinction is important from a pedagogical perspective. Providing source references creates an opportunity for students to verify information and return to the original instructional resources, but the availability of these references does not ensure that learners will use them actively. Similarly, access to a conversational interface may encourage students to refine their questions over time, but the current lack of conversational memory limits the possibility of developing extended inquiry sequences. The assistant’s educational value therefore depends not only on its technical features but also on how it is incorporated into learning activities.
Future classroom deployments could strengthen this reflective dimension through guided activities. For example, students could be asked to compare an answer with its cited source, identify the fragment that supports a specific explanation, reformulate an ambiguous query, or evaluate whether the retrieved evidence is sufficient to answer a practical question. These activities could help transform source traceability from a transparency feature into an explicit learning strategy.
Taken together, the findings provide a positive response to RQ2. Students perceived the assistant as a useful tool for understanding content, resolving doubts, and obtaining rapid explanations. They also recognised its potential to support more autonomous study practices. Nevertheless, the results do not demonstrate an objective improvement in learning autonomy or academic performance. Further longitudinal and experimental research is required to determine whether repeated use of the assistant produces measurable changes in study behaviour, knowledge acquisition, or learning outcomes.
1.1. Design Trade-Offs and Improvement Priorities: RQ3 and RQ4
The third and fourth research questions addressed the limitations identified during the classroom deployment and the pedagogical and technical improvements suggested for future iterations of the assistant. The findings reveal that the most relevant challenges are closely connected to the core design decision underlying the platform: restricting responses to teacher-provided instructional materials.
1.1.1. The Trade-Off between Curricular Control and Informational Coverage
The course-constrained approach was widely perceived as a strength. Students valued receiving answers aligned with the materials used during classroom instruction and appreciated avoiding alternative procedures or explanations that could differ from those taught in the module. However, the same restriction also limited the assistant’s capacity to answer questions when the required information was absent from the repository.
This trade-off is visible in the quantitative results. Ten of the 19 students agreed or strongly agreed that the assistant was sometimes unable to answer because it was limited to the syllabus. The qualitative responses provide additional context: some participants noted that unexpected technical errors or highly specific practical problems might not be covered in the available materials. Others observed that broad questions could lead to partial answers or omit relevant procedural details.
This tension should not be interpreted as a reason to abandon the course-constrained design. Allowing unrestricted access to external sources could weaken the curricular alignment and pedagogical control that participants valued. Instead, future versions should seek a more flexible but transparent approach. One possibility would be to distinguish clearly between different types of sources: teacher-validated course materials, supplementary resources uploaded by the learner, and optional external information subject to explicit user request and appropriate labelling. This layered model could extend the scope of the assistant while preserving the provenance of the information used in each response.
The quality of the repository also remains an important consideration. A course-constrained assistant can reproduce incomplete, outdated, or inaccurate information if these limitations are already present in the instructional materials. The system should therefore support document updating, version management, and periodic review by instructors. Source grounding improves traceability, but it does not eliminate the need to maintain the quality of the underlying corpus.
1.1.1. Conversational Continuity as the Main Development Priority
The absence of conversational memory emerged as the most frequently requested improvement. Eleven of the 19 students explicitly suggested preserving the context of previous interactions, supporting follow-up questions, or allowing users to resume earlier conversations. Although the current platform stores a history of standalone queries and responses, these interactions are not incorporated into the context of subsequent questions.
This limitation reduces the naturalness of the interaction. Learning-related questions often develop progressively: an initial response can generate a new doubt, require clarification, or lead the student to explore a related concept. Requiring learners to reformulate the complete context in every query introduces unnecessary friction and may limit deeper inquiry.
Future versions should therefore incorporate conversational memory within each course-specific interaction. However, continuity should be implemented carefully. Previous questions and responses should be used to preserve the conversational context without allowing the dialogue to drift away from the validated instructional corpus. Each new response should remain grounded in retrieved evidence, and the system should continue to abstain when sufficient support is unavailable.
1.1.1. Handling Ambiguous Queries and Response Boundaries
A further priority concerns the treatment of broad, ambiguous, or insufficiently specific questions. Some students indicated that complete answers required highly detailed queries, while the instructor observed that an ambiguous question produced no useful response and suggested that the assistant could request additional information.
This observation points to an important distinction between abstention and clarification. Returning an abstention message is appropriate when the available materials do not contain sufficient evidence. However, when the difficulty arises because the user’s intention is unclear, the assistant should first attempt to reformulate or narrow the question through a clarification request. For example, a broad query about installing a tool could lead the system to ask which operating system, version, or deployment scenario the learner is using before generating an answer.
Future iterations should therefore incorporate a clarification mechanism before final response generation. This improvement could increase the usefulness of the assistant without weakening its course-constrained nature. It would also support the development of better question-formulation practices by guiding students towards more precise and contextually appropriate queries.
The classroom feedback also suggests that response-boundary control should be strengthened. One participant observed that the assistant correctly abstained when a question was clearly unrelated to the course but occasionally responded to closely related questions even when the relevant information was not present in the materials. In the current prototype, abstention is primarily managed through prompt instructions. A more robust implementation could combine prompt-based abstention with retrieval-level safeguards, such as minimum similarity thresholds or evidence-sufficiency checks. These mechanisms should be calibrated empirically in future evaluations rather than assumed to guarantee accuracy.
1.1.1. Additional Interaction and Traceability Improvements
The participants also suggested several secondary improvements. These included the ability to attach screenshots, images, or supplementary documents; more direct navigation to the exact location of the referenced information within the original source; improved access to previous interactions; and refinements to the visual interface.
The possibility of attaching screenshots may be particularly relevant in technical VET contexts, where students frequently encounter configuration errors, command-line messages, or interface states that are difficult to describe precisely in text. Nevertheless, multimodal interaction would require additional evaluation to ensure that the assistant continued to provide responses aligned with the intended pedagogical scope.
Improved source navigation would also strengthen the educational value of the platform. The current system associates responses with source documents and available location metadata. Future versions could provide more direct links to the relevant page, paragraph, or line range whenever the original file format allows it. This enhancement would make it easier for learners to verify the generated explanation and return to the original instructional resource.
Taken together, the findings provide a clear response to RQ3 and RQ4. The main limitations concern restricted corpus coverage, the lack of conversational continuity, the handling of ambiguous queries, and the need for more robust response-boundary control. The most relevant future improvements are therefore not aimed at removing pedagogical constraints, but at making those constraints more transparent, flexible, and useful for learners.
1.1. Implications for Vocational Education and Training
The findings have several implications for the use of course-constrained Retrieval-Augmented Generation (RAG) assistants in Vocational Education and Training (VET). Technical VET programmes frequently require students to interpret specialised documentation, apply procedural knowledge, and resolve practical questions arising during configuration and troubleshooting activities. In this context, rapid access to explanations grounded in the instructional materials can provide valuable support without replacing the original resources or the instructor’s role.
The proposed assistant should therefore be understood as a complementary learning-support tool rather than as an autonomous source of knowledge. Its main educational value lies in facilitating access to the materials selected by the teacher, helping students locate relevant information, clarify concepts, and obtain concise explanations when a doubt arises. This function may be particularly useful during practical sessions, independent study, or revision activities, where immediate teacher intervention is not always available.
The course-constrained approach also supports a form of pedagogical oversight that is especially relevant in vocational education. By restricting the retrieval repository to instructional resources selected by the teacher, the assistant can remain more closely aligned with the terminology, procedures, and level of complexity expected in the module. This characteristic may reduce the risk of exposing students to alternative solutions that are technically plausible but inconsistent with the learning objectives or with the procedures introduced in class.
However, pedagogical control should not be confused with completeness. The findings show that a restricted repository can also limit the assistant’s ability to address unexpected technical problems, ambiguous queries, or questions that extend beyond the available course materials. For this reason, instructors should view the knowledge base as an evolving educational resource requiring periodic review, updating, and extension. The design of the assistant should make these boundaries visible to learners and clearly distinguish between supported answers, insufficient evidence, and optional supplementary information.
The interaction history available to instructors also creates opportunities for pedagogical monitoring. Reviewing the questions submitted by students may help identify recurring doubts, misunderstood concepts, or parts of the instructional materials that require clarification. In this sense, the assistant can support not only individual consultation but also formative reflection on the teaching process. Future versions could incorporate dashboards or summaries of frequently asked questions to facilitate this type of analysis.
The classroom integration of these tools should also include explicit guidance on their appropriate use. Students need to understand that source-grounded responses remain dependent on the quality of the indexed materials and that the availability of citations does not eliminate the need for critical verification. Activities that require learners to compare answers with the original sources, reformulate ambiguous questions, or evaluate whether the retrieved evidence is sufficient could contribute to the development of artificial intelligence literacy and responsible study practices.
These implications suggest that course-constrained RAG assistants may be particularly suitable for technical VET environments when they are integrated as supervised, transparent, and curriculum-aligned support tools. Their educational value does not depend solely on response generation, but also on how teachers curate the knowledge base, design learning activities, interpret interaction data, and encourage students to use the system critically.
1.1. Study Limitations
The findings should be interpreted in light of several limitations related to the exploratory nature of the study.
First, the evaluation was conducted with a small convenience sample comprising 19 students from a single module within one Vocational Education and Training (VET) institution. The results therefore provide context-specific evidence and should not be generalised to other educational levels, subject areas, institutions, or student profiles without further investigation. Future studies should include larger and more diverse samples across different modules, VET programmes, and educational settings.
Second, the classroom deployment was limited to a single practical session. The study captures participants’ immediate perceptions after interacting with the assistant, but it does not examine sustained use over time. Consequently, the results do not provide evidence regarding long-term adoption, changes in study habits, continued engagement, or the extent to which the assistant could support autonomous learning outside the classroom. Longitudinal evaluations are required to assess how students use the system during regular study activities and whether their perceptions change after repeated interaction.
Third, the evaluation focused primarily on self-reported perceptions. The questionnaire examined usability, clarity, perceived reliability, learning support, and practical limitations, but the study did not include objective measures of learning outcomes. No pre-test/post-test design, control group, or assessment of academic performance was incorporated. The findings therefore indicate that students perceived the assistant as useful, but they do not demonstrate that its use improved knowledge acquisition, retention, or academic achievement.
Fourth, perceived reliability should not be interpreted as objective response accuracy. Although students generally considered the generated answers correct, the responses were not systematically reviewed by independent experts using a predefined evaluation protocol. The study also did not compare the course-constrained assistant with a general-purpose Large Language Model (LLM) or quantify the occurrence of unsupported or hallucinated responses. Future research should incorporate expert assessment of generated answers, retrieval-quality metrics, predefined question sets, and controlled comparisons between constrained and unconstrained systems.
Fifth, the questionnaire was developed specifically for this exploratory classroom evaluation. The thematic organisation of the items facilitated descriptive interpretation, but the instrument has not undergone psychometric validation. The grouped scores should therefore be understood as descriptive summaries rather than as validated measurement scales. Future studies with larger samples could refine the instrument and examine its reliability and construct validity.
Sixth, the participants reported extensive previous experience with generative artificial intelligence tools such as ChatGPT, Gemini, and Copilot. This familiarity may have facilitated interaction with the assistant and influenced their expectations and evaluations. The generally positive reception observed in this study may not be replicated in groups with lower levels of AI literacy or different patterns of technology use.
Seventh, the instructor feedback provides a useful complementary perspective but is based on the response of a single teacher. It should therefore be interpreted as contextual evidence rather than as a representative evaluation of instructor acceptance. Future deployments should involve multiple instructors and examine how teachers use interaction histories, manage course materials, and integrate the assistant into different pedagogical strategies.
Finally, the evaluation concerns a specific version of the prototype and its current technical configuration. The system retrieves the five highest-ranked fragments using local cosine similarity and relies primarily on prompt-based abstention when the available evidence is insufficient. The study did not assess retrieval precision, experiment with different chunking strategies, or calibrate similarity thresholds. Subsequent technical evaluations should examine how these design decisions influence response quality, corpus-boundary control, and the balance between reliability and informational coverage.
Despite these limitations, the study provides preliminary empirical evidence from an authentic technical VET classroom. Its value lies in identifying both the perceived benefits of a course-constrained RAG assistant and the practical challenges that should guide more extensive technical and pedagogical evaluations.
1. Conclusions and Future Work
This study presented the design, implementation, and exploratory classroom evaluation of a web-based educational assistant built on Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs). The platform was developed as a course-constrained learning-support tool designed to generate responses from teacher-provided instructional materials, preserve source traceability, and provide an abstention message when the retrieved evidence is insufficient.
The assistant was deployed in an authentic classroom setting within the Higher Vocational Training programme in Network and Information Systems Administration (ASIR). Nineteen students used the platform during a practical session, and the instructor also interacted with the system and provided complementary feedback. The evaluation combined Likert-scale questionnaire items with open-ended questions to examine perceived usability, clarity, reliability, learning support, risks, and potential improvements.
The findings provide preliminary descriptive support for the exploratory hypothesis. Students perceived the assistant as easy to use and valued its ability to provide concise, structured, and course-aligned responses. The strongest perceived benefits concerned doubt resolution, the rapid provision of explanations and clarifications, and the confidence associated with grounding answers in teacher-provided materials. These results suggest that course-constrained RAG assistants may constitute valuable complementary tools for facilitating access to instructional resources in technical Vocational Education and Training contexts.
The evaluation also revealed an important design trade-off. Restricting the assistant to a controlled instructional corpus can reinforce curricular alignment, source transparency, and perceived trustworthiness. However, the same restriction may limit the system’s ability to answer broad questions, resolve unexpected technical problems, or provide complete responses when the required information is not sufficiently represented in the available materials. The educational value of the assistant therefore depends not only on the language model but also on the quality, coverage, and maintenance of the instructional repository.
Conversational continuity emerged as the most frequently requested improvement. Although the current prototype stores previous standalone queries and responses, it does not preserve the context required to support follow-up questions. Future versions should incorporate course-specific conversational memory while ensuring that each answer remains supported by retrieved evidence. The handling of broad or ambiguous queries should also be improved through clarification requests, allowing the assistant to ask for additional information before returning an abstention message or generating a partial response.
Further technical development should examine retrieval-level safeguards, including evidence-sufficiency checks and empirically calibrated similarity thresholds. Additional improvements may include more direct navigation to the cited location within source documents, the controlled incorporation of supplementary user-provided materials, optional multimodal inputs such as screenshots, and instructor dashboards for identifying recurring questions or potential gaps in the instructional resources.
Future research should complement the present exploratory findings with more extensive pedagogical and technical evaluations. Relevant directions include longitudinal deployments across multiple modules and educational institutions, expert assessment of generated answers, objective measurement of retrieval quality, controlled comparisons with general-purpose AI assistants, and experimental designs incorporating learning-outcome measures. Studies involving larger and more diverse groups of students and instructors would also help determine the extent to which the findings can be transferred to other vocational, technical, and educational contexts.
Overall, the study provides preliminary evidence that course-constrained RAG assistants can support a transparent and pedagogically supervised approach to the integration of generative AI in technical VET environments. Their value lies not in replacing instructors or instructional resources, but in improving access to teacher-selected materials while making the boundaries, sources, and limitations of AI-generated support more visible to learners.
Supplementary Materials
The following supporting information can be downloaded at the website of this paper posted on Preprints.org, Table S1: Complete post-session questionnaire administered to participants.
Author Contributions
Conceptualization, J.D.B., R.H.N, O. B.G,; methodology, J.D.B., R.H.N; software, J.D.B.; validation, J.D.B., R.H.N.; formal analysis, J.D.B.; investigation, J.D.B., R.H.N., O.B.G; resources, J.D.B., R.H.N.; data curation, J.D.B.; writing—original draft preparation, J.D.B, R.H.N.; writing—review and editing, J.D.B. , R.H.N., O.B.G.; visualization, J.D.B, R.H.N.; supervision, R.H.N., O.B.G.; All authors have read and agreed to the published version of the manuscript.
Funding
This research was funded by research grants PID2022-137849OB-I00 funded by MI CIU/AEI/10.13039/501100011033 and by the ERDF, EU.
Institutional Review Board Statement
The study was conducted in accordance with the Declaration of Helsinki and approved by the Research Ethics Committee of Universidad Rey Juan Carlos (URJC) (protocol code 031220250052026; date of approval: 9 March 2026).
Informed Consent Statement
Written informed consent was obtained from all subjects involved in the study.
Data Availability Statement
The de-identified quantitative data supporting the findings of this study and a redacted version of the qualitative responses are available from the corresponding author upon reasonable request. The complete raw dataset is not publicly available due to privacy and ethical considerations related to the small classroom cohort and the potential risk of indirect identification.
Acknowledgments
The authors would like to thank IES Rosa Chacel and the participants who took part in the classroom evaluation for their collaboration.During the preparation of this manuscript, the authors used ChatGPT (OpenAI) for language editing, structural revision, the refinement of the academic writing, and the preparation of selected visual materials. The authors reviewed and edited the output and take full responsibility for the content of this publication.
Conflicts of Interest
The authors declare no conflicts of interest.
Abbreviations
The following abbreviations are used in this manuscript:
| AI | Artificial Intelligence |
| API | Application Programming Interface |
| ASIR | Network and Information Systems Administration |
| DOCX | Microsoft Word Open XML Document |
| LLM | Large Language Model |
| MOOC | Massive Open Online Course |
| Portable Document Format | |
| RAG | Retrieval-Augmented Generation |
| TXT | Plain Text File |
| VET | Vocational Education and Training |
References
- Kasneci, E.; Sessler, K.; Küchemann, S.; Bannert, M.; Dementieva, D.; Fischer, F.; Gasser, U.; Groh, G.; Günnemann, S.; Hüllermeier, E.; et al. ChatGPT for good? On opportunities and challenges of large language models for education. Learn. Individ. Differ. 2023, 103, 102274. [Google Scholar] [CrossRef]
- von Garrel, J.; Mayer, J. Artificial intelligence in studies—Use of ChatGPT and AI-based tools among students in Germany. Humanit. Soc. Sci. Commun. 2023, 10, 799. [Google Scholar] [CrossRef]
- Baek, C.; Tate, T.; Warschauer, M. “ChatGPT seems too good to be true”: College students’ use and perceptions of generative AI. Comput. Educ. Artif. Intell. 2024, 7, 100294. [Google Scholar] [CrossRef]
- Mai, D.T.T.; Da, C.V.; Hanh, N.V. The use of ChatGPT in teaching and learning: A systematic review through SWOT analysis approach. Front. Educ. 2024, 9, 1328769. [Google Scholar] [CrossRef]
- Naznin, K.; Al Mahmud, A.; Nguyen, M.T.; Chua, C. ChatGPT integration in higher education for personalized learning, academic writing, and coding tasks: A systematic review. Computers 2025, 14, 53. [Google Scholar] [CrossRef]
- Munaye, Y.Y.; Admass, W.; Belayneh, Y.; Molla, A.; Asmare, M. ChatGPT in education: A systematic review on opportunities, challenges, and future directions. Algorithms 2025, 18, 352. [Google Scholar] [CrossRef]
- Lo, C.K. What is the impact of ChatGPT on education? A rapid review of the literature. Educ. Sci. 2023, 13, 410. [Google Scholar] [CrossRef]
- Montenegro-Rueda, M.; Fernández-Cerero, J.; Fernández-Batanero, J.M.; López-Meneses, E. Impact of the implementation of ChatGPT in education: A systematic review. Computers 2023, 12, 153. [Google Scholar] [CrossRef]
- Li, Z.; Wang, Z.; Wang, W.; Hung, K.; Xie, H.; Wang, F.L. Retrieval-augmented generation for educational application: A systematic survey. Comput. Educ. Artif. Intell. 2025, 8, 100417. [Google Scholar] [CrossRef]
- Miladi, F.; Psyché, V.; Diattara, A.; El Mawas, N.; Lemire, D. Evaluating a GPT-4 and retrieval-augmented generation-based conversational agent to enhance learning experience in a MOOC. In Proceedings of the 17th International Conference on Computer Supported Education (CSEDU 2025), Porto, Portugal, 1–3 April 2025; Volume 2, pp. 347–354. [Google Scholar] [CrossRef]
- Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; Küttler, H.; Lewis, M.; Yih, W.; Rocktäschel, T.; et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems; 2020; Volume 33, NeurIPS 2020, pp. 9459–9474. [Google Scholar]
- Swacha, J.; Gracel, M. Retrieval-Augmented Generation (RAG) chatbots for education: A survey of applications. Appl. Sci. 2025, 15, 4234. [Google Scholar] [CrossRef]
- Németh, R.; Tátrai, A.; Szabó, M.; Zaletnyik, P.T.; Tamási, Á. Exploring the use of retrieval-augmented generation models in higher education: A pilot study on artificial intelligence-based tutoring. Soc. Sci. Humanit. Open 2025, 12, 101751. [Google Scholar] [CrossRef]
- Thesen, T.; Park, S.H. A generative AI teaching assistant for personalized learning in medical education. npj Digit. Med. 2025, 8, 627. [Google Scholar] [CrossRef] [PubMed]
- Rosyadi, M.I.; Kustiawan, I.; Tetehfio, E.O.; Joshua, Q. The role of AI in vocational education: A systematic literature review. J. Vocat. Educ. Stud. 2023, 6, 244–263. [Google Scholar] [CrossRef]
- Schei, O.M.; Møgelvang, A.; Ludvigsen, K. Perceptions and use of AI chatbots among students in higher education: A scoping review of empirical studies. Educ. Sci. 2024, 14, 922. [Google Scholar] [CrossRef]
- OpenAI. text-embedding-3-small Model. Available online: https://developers.openai.com/api/docs/models/text-embedding-3-small (accessed on 8 June 2026).
- OpenAI. GPT-4o mini Model. Available online: https://developers.openai.com/api/docs/models/gpt-4o-mini (accessed on 8 June 2026).
Figure 1.
Workflow of the proposed course-constrained RAG educational assistant.
Figure 2.
System architecture of the proposed course-constrained RAG educational assistant.
Figure 3.
Semantic retrieval and controlled response-generation workflow implemented in the proposed educational assistant.
Figure 3.
Semantic retrieval and controlled response-generation workflow implemented in the proposed educational assistant.
Table 1.
Selected empirical studies on RAG-based educational assistants and positioning of the present study.
Table 1.
Selected empirical studies on RAG-based educational assistants and positioning of the present study.
| Study | Educational context and participants | Knowledge base and deployment | Evaluation focus |
|---|---|---|---|
| Németh et al. [13] | Four university courses; 118 students and 46 survey responses | Lecturer-provided course resources; authentic course deployment | Interaction patterns, response quality, student feedback, and lecturer perceptions |
| Miladi et al. [10] | Artificial intelligence MOOC; 25 university students | MOOC resources; three-day experimental activity with control (n = 12) and experimental (n = 13) groups | Knowledge gain and usability |
| Thesen and Park [14] | Medical education course; two cohorts (n = 92 and n = 98) | Instructor-curated repository; optional access during a 14-week course | Adoption patterns, perceived usefulness, and the balance between reliability and comprehensiveness |
| Present study | Technical VET programme (ASIR); 19 students and one instructor | Teacher-provided course materials; practical classroom session | Perceived usefulness, usability, clarity, reliability, learning support, and practical limitations |
Table 2.
Main implementation characteristics of the proposed educational assistant.
| Component | Implementation |
| Application model | Web-based platform |
| User roles | Student, instructor, and administrator |
| Backend | PHP |
| User interface | Browser-based interface developed with Bootstrap |
| Data storage | MySQL-compatible relational database |
| Supported document formats | PDF, DOCX, and TXT |
| Text processing | Document-aware segmentation targeting approximately 1200 characters per fragment with an overlap of approximately 150 characters |
| Embedding model | text-embedding-3-small [17] |
| Retrieval scope | Class-specific instructional repository |
| Similarity calculation | Local cosine similarity calculation in the application backend |
| Retrieved context | Five highest-ranked fragments |
| Response-generation model | gpt-4o-mini [18] |
| Temperature | 0.2 |
| Abstention strategy | Prompt-based abstention when the retrieved evidence is insufficient |
| Interaction storage | Standalone questions, generated responses, and citation metadata |
| Conversational memory | Not implemented in the current prototype |
Table 4.
Student responses to the Likert-scale questionnaire (n = 19).
| Item | Abbreviated statement | Mean | SD | Agreement, n (%) |
|---|---|---|---|---|
| Usability | ||||
| 4a | The assistant was easy to use | 4.63 | 0.50 | 19 (100.0%) |
| 4b | I needed technical or external assistance to use the assistant | 1.32 | 0.48 | 0 (0.0%) |
| 4c | The interface was clear and comfortable to use | 4.05 | 0.71 | 15 (78.9%) |
| Learning support and perceived autonomy | ||||
| 5a | The assistant helped me understand the course content better | 4.21 | 0.71 | 16 (84.2%) |
| 5b | The assistant helped me resolve questions related to the course | 4.42 | 0.61 | 18 (94.7%) |
| 5c | The assistant provided clear and useful explanations | 4.47 | 0.61 | 18 (94.7%) |
| 5d | The assistant enabled me to obtain summaries or clarifications rapidly | 4.42 | 0.69 | 17 (89.5%) |
| 5e | The assistant could help me study more autonomously | 4.37 | 0.76 | 16 (84.2%) |
| Clarity and perceived reliability | ||||
| 6a | The responses were clear and easy to understand | 4.32 | 0.58 | 18 (94.7%) |
| 6b | The responses seemed correct most of the time | 4.53 | 0.51 | 19 (100.0%) |
| 6c | Using only course materials increased my confidence in the assistant | 4.21 | 1.03 | 15 (78.9%) |
| Perceived risks and limitations | ||||
| 7a | The assistant was sometimes unable to answer because it was limited to the syllabus | 3.79 | 1.08 | 10 (52.6%) |
| 7b | Excessive reliance on the assistant could affect my study habits | 3.00 | 1.11 | 7 (36.8%) |
| 7c | The assistant sometimes provided an unhelpful or incomplete response | 2.53 | 1.17 | 5 (26.3%) |
| Reflective use and source verification | ||||
| 8a | The assistant encouraged me to check the original course materials | 3.79 | 1.18 | 11 (57.9%) |
| 8b | Interacting with the assistant helped me formulate better questions | 3.68 | 0.82 | 11 (57.9%) |
Note:Agreement includes responses of 4 (agree) and 5 (strongly agree). Item 4b is negatively worded; lower values therefore indicate a more positive usability result. Items 7a–7c are also negatively worded; higher values indicate a greater perceived concern. The complete questionnaire wording is provided inTable S1 in the Supplementary Material.
Table 5.
Main themes identified in the open-ended student responses.
| Theme | Main observation | Illustrative quotation |
|---|---|---|
| Course-material grounding and trust | Students valued receiving answers based on the instructional resources selected by the teacher | “I like that it works only with the notes we provide.” |
| Focused and accessible learning support | Participants highlighted concise, structured, and rapidly available responses | “It does not take long to respond, and it focuses on the notes.” |
| Lack of conversational continuity | Students frequently requested memory within the chat and the ability to continue previous questions | “It should have memory within the chat and be able to link one question to another.” |
| Corpus coverage and response boundaries | The restricted repository increased curricular alignment but could limit the completeness or scope of certain answers | “If you do not specify enough, it gives partial rather than complete results.” |
| Additional usability and functionality improvements | Students suggested more precise source navigation, file or image attachments, and interface refinements | “The link to the reference file should take you to that exact point in the file, not to the file in general.” |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
