CONF.i: A Confidence-Informed AI Feedback Framework for Canvas LMS in Engineering Education

1. Introduction

The rapid digital transformation of higher education has positioned Learning Management Systems (LMS) as central infrastructures for teaching, learning, communication, and assessment. Platforms such as Canvas, Moodle, and Blackboard are now widely adopted across universities worldwide, supporting millions of students and instructors through content delivery, assignment management, communication tools, and grading systems [1,2]. Despite their broad adoption, many LMS implementations continue to function primarily as administrative and organizational platforms rather than intelligent learning environments capable of supporting personalized instruction, adaptive assessment, and metacognitive development [3,4].

At the same time, advances in educational measurement and learning analytics have expanded possibilities for more sophisticated approaches to assessment. Item Response Theory (IRT) and related multidimensional assessment approaches have demonstrated the value of moving beyond simple aggregate scores toward models that capture richer dimensions of learner performance, including response behavior, confidence, and engagement patterns [5,6]. Recent research has emphasized that assessment should not only evaluate whether students answer correctly but also examine how learners perceive and regulate their own understanding [7,8]. Confidence-informed assessment approaches are particularly relevant in this context because they provide insights into metacognitive calibration, helping identify patterns such as overconfidence, under confidence, and uncertainty during learning processes [9,10].

Parallel to these developments, generative artificial intelligence (AI) has emerged as a transformative force in education. Large Language Models (LLMs), including OpenAI’s GPT models and Google’s Gemini family, have demonstrated capabilities in natural language generation, contextual reasoning, tutoring support, and personalized feedback generation [11,12]. In educational settings, generative AI has been explored for intelligent tutoring systems, automated feedback, content creation, adaptive learning support, and self-regulated learning interventions [13,14]. Recent studies suggest that AI-generated feedback can improve learner engagement and provide scalable individualized support when aligned with pedagogical principles and instructor oversight [15,16]. However, concerns remain regarding reliability, pedagogical alignment, transparency, response quality, and institutional integration [17,18].

Although LMS technologies, learning analytics, and generative AI have evolved significantly in parallel, their integration within mainstream educational environments remains limited. Many institutions seeking adaptive assessment or AI-enhanced learning support depend on external proprietary systems that require substantial financial investment, technical infrastructure, or institutional reconfiguration [19,20]. Consequently, there is increasing interest in approaches that extend existing institutional platforms through interoperable and low-cost solutions capable of supporting intelligent and personalized learning experiences.

This study addresses this challenge through the development of CONF.i, a confidence-informed assessment and AI feedback framework integrated with Canvas LMS using Google Apps Script and Google Gemini AI. Rather than proposing a fully psychometric IRT implementation, the framework combines traditional assessment scores with student self-reported confidence ratings to support formative assessment, metacognitive reflection, and personalized feedback generation. The system was designed to operate within existing institutional infrastructures using Canvas LTI standards and Google Workspace services, minimizing deployment barriers and avoiding additional licensing costs.

The conceptual foundation of this work originates from prior research on intelligent tutoring systems, confidence-informed assessment, and multidimensional learner analysis conducted through collaboration between Tecnologico de Monterrey and the University of São Paulo (USP). Building on these foundations, the present study focuses on the practical integration of confidence-aware assessment and generative AI within a widely adopted LMS environment. The project also contributes to ongoing discussions regarding sustainable and accessible educational innovation, particularly in the context of higher education institutions seeking scalable approaches to personalized learning support.

Using design-based research methodology, this study developed and evaluated a functional prototype capable of integrating Canvas LMS, confidence-informed assessment mechanisms, and AI-generated feedback services. Prototype testing employed simulated learner profiles to explore system functionality, assessment visualization, and feedback generation processes. While the study does not claim validated learning effectiveness outcomes, it aims to demonstrate the feasibility and potential pedagogical value of integrating confidence-aware analytics and generative AI within existing LMS ecosystems.

The study was guided by the following research questions:

RQ1: How can Canvas LMS be integrated with confidence-informed assessment mechanisms and generative AI services using existing institutional infrastructure and authentication systems?

RQ2: What types of diagnostic insights can emerge from combining assessment performance with student confidence ratings within an AI-enhanced LMS environment?

This work contributes to literature in three ways. First, it presents a practical framework for integrating confidence-informed assessment and AI-generated feedback into Canvas LMS using accessible institutional technologies. Second, it explores how confidence-aware assessment may support richer interpretations of learner performance beyond traditional grading approaches. Third, it offers an exploratory model for extending LMS platforms with personalized feedback and metacognitive support capabilities while maintaining instructor oversight and compatibility with existing educational workflows.

2. Literature Review

2.1. Learning Management Systems as Evolving Platforms

Learning Management Systems (LMS) have become foundational digital infrastructures in higher education, supporting instructional delivery, communication, assessment, and academic administration across diverse educational contexts [19,20]. Initially designed primarily as repositories for course materials and administrative management tools, LMS platforms have progressively evolved into broader digital ecosystems capable of supporting collaborative learning, analytics, competency tracking, and adaptive educational experiences [21,22]. Among the most widely adopted platforms, Canvas LMS has gained significant institutional acceptance due to its cloud-based architecture, usability, interoperability standards, and robust Application Programming Interface (API) support [21,22]. In particular, Canvas supports Learning Tools Interoperability (LTI) standards, enabling institutions to integrate external applications and services without requiring modifications to the core platform infrastructure.

Despite these technological capabilities, research consistently indicates that LMS platforms remain underutilized in pedagogical practice [1,23]. In many institutions, faculty primarily employ LMS environments for administrative functions such as syllabus distribution, content hosting, assignment submission, and grade recording, while advanced features related to analytics, adaptive learning, personalized feedback, and competency-based assessment are used less frequently [24,25]. This limited adoption of intelligent learning functionalities has been described as a “feature gap,” where the technical potential of LMS ecosystems exceeds their actual pedagogical implementation [26].

Recent developments in learning analytics and educational data mining have increased interest in transforming LMS platforms into more intelligent and learner-centered environments [8,15]. By capturing and analyzing interaction data, LMS platforms can potentially support early identification of learning difficulties, adaptive interventions, and personalized feedback mechanisms [27]. However, implementing such capabilities within institutional environments remains challenging due to issues related to technical complexity, scalability, faculty readiness, privacy concerns, and integration with existing teaching practices [14,17].

Institutional experiences reported in recent literature further illustrate these opportunities and challenges. For example, the University of Waterloo’s implementation of advanced analytics tools within the D2L platform demonstrated that instructors valued enhanced visibility into student engagement and performance patterns, but also reported difficulties associated with inconsistent course structures and limited customization options. Similarly, dashboard redesign initiatives at Graz University of Technology emphasized the importance of usability, transparency, and pedagogical alignment when presenting learning analytics to students and instructors [27]. These findings suggest that effective LMS enhancement requires not only technical integration but also careful consideration of educational design and user experience.

In parallel, increasing interest in artificial intelligence (AI) within higher education has created new opportunities to extend LMS capabilities through automated feedback, intelligent tutoring, adaptive assessment, and personalized learning support [10,11]. Generative AI technologies, particularly Large Language Models (LLMs), offer the possibility of embedding scalable conversational and feedback mechanisms directly into LMS workflows. However, many existing AI-enhanced educational solutions depend on proprietary commercial systems that may present financial, technical, or institutional adoption barriers [13,18].

Within this context, there is growing interest in approaches that augment existing LMS platforms through interoperable and low-cost integration strategies rather than replacing institutional infrastructures entirely [41,42]. This perspective aligns with broader educational technology research emphasizing sustainable innovation, institutional compatibility, and incremental digital transformation. The CONF.i project follows this approach by integrating confidence-informed assessment and generative AI capabilities within Canvas LMS using existing institutional credentials, Google Workspace services, and LTI interoperability standards. Rather than functioning as a standalone assessment platform, the framework seeks to extend the pedagogical capabilities of the LMS environment while preserving familiar workflows for instructors and students.

In the Latin American context, institutions such as Tecnologico de Monterrey and the University of São Paulo have increasingly explored innovative educational technology initiatives aimed at improving personalized learning, analytics, and digital assessment practices. The collaboration between these institutions reflects a broader movement toward international and interdisciplinary educational innovation, combining expertise in educational technology implementation, intelligent systems, and learner analytics to address emerging challenges in higher education.

2.2. Item Response Theory in Digital Assessment: The Brazilian Theoretical Foundation

Item Response Theory (IRT) has significantly influenced contemporary approaches to educational measurement and digital assessment by shifting the focus from aggregate test scores to probabilistic models of learner performance at the item level [3,4]. Unlike Classical Test Theory, which assumes that measurement error is constant across populations and test conditions, IRT models estimate the probability of a correct response based on interactions between learner ability and item characteristics such as difficulty, discrimination, and guessing parameters [5,6]. These characteristics have made IRT particularly valuable in adaptive testing, large-scale assessment, and computerized educational systems where more precise measurement and individualized assessment pathways are required [28,29].

In digital learning environments, IRT-inspired approaches have contributed to the development of Computerized Adaptive Testing (CAT), intelligent tutoring systems, and learning analytics frameworks capable of dynamically adjusting assessment difficulty according to learner performance [7,8]. Research has demonstrated that adaptive assessment models can improve measurement efficiency, reduce test fatigue, and provide more accurate estimates of learner ability when compared with traditional fixed-form assessments [29,30]. At the same time, contemporary educational research increasingly recognizes that effective assessment should consider not only accuracy, but also learner behaviors, metacognitive awareness, response strategies, and self-regulated learning processes [13,34].

Recent studies have therefore explored extensions of traditional assessment models by incorporating additional learner dimensions such as response time, engagement patterns, confidence ratings, and process-oriented indicators [31,32,33]. Confidence-informed assessment approaches are particularly relevant because they provide insights into learners’ metacognitive calibration, revealing whether students appropriately recognize the limits of their own understanding [9,10]. Prior research has shown that students who demonstrate high accuracy with low confidence may possess fragile or unstable knowledge structures, while learners displaying high confidence despite incorrect responses may be more resistant to remediation due to misconceptions or overconfidence biases [32,35].

The conceptual foundation for the CONF.i framework emerged from prior research conducted in Brazil on intelligent tutoring systems, multidimensional learner analysis, and confidence-informed assessment approaches. This research explored how digital assessment environments could incorporate metacognitive indicators alongside traditional performance measures to provide richer interpretations of learner behavior. Rather than relying exclusively on correctness-based evaluation, the proposed perspective emphasized the importance of combining cognitive and metacognitive dimensions during assessment processes.

Building on these foundations, the CONF.i framework adopts a simplified confidence-informed assessment model composed of three interconnected variables:

Grade: the percentage of correct responses representing summative assessment performance.

Confidence: learner self-reported certainty regarding assessment responses.

Performance: a composite indicator combining correctness and confidence information to support diagnostic interpretation.

This approach is informed by multidimensional assessment traditions [34], while intentionally avoiding the complexity of full psychometric IRT implementation. The framework does not estimate latent traits or employ formal probabilistic item calibration models. Instead, it applies a practical confidence-aware assessment strategy designed for formative and exploratory educational contexts within LMS environments. The objective is not to replace established psychometric methodologies, but rather to extend digital assessment practices through the integration of metacognitive indicators capable of supporting reflection, personalized feedback, and learner analytics.

The inclusion of confidence as an assessment dimension aligns with broader research on self-regulated learning and metacognition, which emphasizes the importance of learners’ ability to evaluate their own knowledge, monitor uncertainty, and regulate learning strategies [7,13,35]. Confidence-aware assessment can therefore contribute not only to performance measurement, but also to the identification of learning patterns that may otherwise remain hidden in conventional grading systems. For example, two students achieving identical scores may demonstrate substantially different confidence profiles, suggesting distinct instructional needs and intervention strategies.

In the context of digital education, these approaches have become increasingly relevant as institutions seek assessment models capable of supporting personalized learning experiences, adaptive feedback, and learner-centered analytics [8,15]. The integration of confidence-informed assessment with generative AI technologies further expands these possibilities by enabling automated interpretation of learner patterns and the generation of individualized formative feedback at scale. Within the CONF.i project, this integration seeks to combine confidence-aware assessment insights with AI-generated educational support while maintaining compatibility with existing LMS infrastructures and instructional workflows.

2.3. Generative AI in Education: Opportunities and Challenges

The emergence of generative artificial intelligence (AI) has significantly transformed contemporary discussions surrounding educational technology, teaching practices, and personalized learning in higher education. Since the public release of large-scale conversational AI systems such as ChatGPT in 2022, educational institutions have increasingly explored the potential of Large Language Models (LLMs) to support tutoring, assessment, feedback generation, content creation, and adaptive learning experiences [11,12]. These systems demonstrate advanced capabilities in natural language understanding, contextual reasoning, summarization, and dialogue generation, enabling new forms of interaction between learners, instructors, and digital educational environments [10,13].

Recent systematic reviews identify a wide range of educational applications for generative AI, including intelligent tutoring systems, automated formative feedback, personalized learning recommendations, assessment support, language learning assistance, and administrative automation [10,14]. Studies have reported potential benefits such as increased learner engagement, improved accessibility to educational support, scalable individualized feedback, and enhanced opportunities for self-regulated learning [15,16]. In particular, AI-generated feedback has attracted significant attention because it can provide immediate and context-sensitive guidance that would otherwise require substantial instructor time and effort [11,12].

Generative AI also offers new possibilities for supporting metacognitive learning processes. By analyzing learner responses and interaction patterns, AI systems may help students reflect on misunderstandings, identify areas requiring improvement, and develop greater awareness of their own learning strategies [7,35]. These capabilities align with broader educational goals associated with self-regulated learning, formative assessment, and learner-centered pedagogies [13]. In digital assessment contexts, AI systems may therefore function not only as automated response generators but also as tools that support reflection, confidence calibration, and personalized academic guidance.

Within this evolving landscape, Google’s Gemini family and related educational initiatives such as LearnLM represent examples of generative AI systems designed with explicit educational applications in mind. LearnLM has been described as an experimental educational model informed by learning science principles, including active learning, cognitive load management, curiosity stimulation, and reflective questioning strategies [36,37]. Such developments illustrate growing interest in aligning generative AI technologies with pedagogical frameworks rather than treating them solely as general-purpose conversational systems.

Despite these opportunities, the rapid adoption of generative AI in education has also raised significant pedagogical, ethical, and institutional concerns [14,17]. Researchers have highlighted challenges related to reliability, hallucinated content, bias, assessment integrity, overreliance on AI-generated responses, and unequal access to digital resources [15,18]. Instructors and institutions also face uncertainty regarding how to integrate AI technologies responsibly within existing teaching practices while maintaining transparency, academic integrity, and meaningful human oversight [11,17].

Ethical and regulatory concerns have become increasingly central in educational AI discussions. A 2023 Brookings Institution report emphasized the importance of addressing bias and fairness in educational AI systems to avoid reinforcing inequalities or disadvantaging underrepresented groups [38]. Similarly, emerging regulatory frameworks such as the European Union Artificial Intelligence Act classify educational AI applications as high-risk contexts requiring transparency, accountability, human supervision, and careful evaluation [8]. These considerations underscore the importance of designing AI-enhanced educational systems that prioritize pedagogical alignment, instructor control, and responsible data practices.

Another challenge involves the practical integration of generative AI within institutional educational ecosystems. Many existing AI-enhanced learning tools operate as standalone platforms disconnected from institutional LMS infrastructures, creating barriers related to authentication, workflow continuity, data management, and faculty adoption [19,20]. Consequently, there is growing interest in interoperable solutions capable of embedding AI functionalities directly into familiar LMS environments while minimizing disruption to established instructional practices.

The integration of generative AI with confidence-informed assessment represents a particularly promising area for educational innovation. Confidence-aware assessment provides contextual information regarding learner certainty and metacognitive calibration, while AI systems can use these indicators to generate more personalized and reflective feedback. For example, students demonstrating low confidence despite correct responses may benefit from reinforcement-oriented feedback, whereas overconfident incorrect responses may require conceptual clarification and metacognitive intervention. Combining assessment performance with confidence information may therefore enable AI systems to provide richer and more pedagogically meaningful feedback than correctness-based evaluation alone.

The CONF.i framework was developed within this context as an exploratory approach to integrating generative AI, confidence-informed assessment, and LMS interoperability. Rather than replacing instructors or automating grading decisions, the framework seeks to augment formative assessment practices through AI-supported personalized feedback and learner analytics while preserving instructor oversight and compatibility with existing institutional infrastructure. This design perspective aligns with emerging educational AI research emphasizing augmentation, collaboration, and pedagogically guided integration rather than technological substitution [15,17].

2.4. Integration Challenges and Opportunities

The integration of Learning Management Systems (LMS), learning analytics, confidence-informed assessment, and generative artificial intelligence (AI) presents significant opportunities for enhancing higher education, while also introducing important technical, pedagogical, and institutional challenges [21,40]. As universities increasingly seek personalized and data-informed learning environments, the ability to connect assessment systems, analytics platforms, and AI-driven feedback mechanisms within existing institutional infrastructures has become an important area of educational technology research [10,15].

One of the primary technical challenges involves interoperability between educational platforms and external intelligent services. Modern LMS platforms such as Canvas provide support for Learning Tools Interoperability (LTI) standards and Application Programming Interfaces (APIs), enabling external tools to exchange authentication, course, and assessment data securely [21,22]. These interoperability standards create opportunities for institutions to extend LMS functionality without modifying core platform architectures. However, implementing advanced assessment analytics and AI-enhanced services still requires careful coordination of authentication systems, data flows, interface design, and institutional security policies [19,20].

In practice, many institutions face difficulties integrating intelligent educational technologies due to infrastructure complexity, licensing costs, limited technical support, and concerns regarding scalability and maintainability [14,18]. Commercial adaptive learning and AI tutoring platforms often require proprietary ecosystems or specialized deployments that may exceed institutional financial or technical capacities [13]. As a result, there is growing interest in lightweight and interoperable development approaches capable of leveraging existing institutional resources while minimizing implementation barriers.

Cloud-based scripting and automation environments, such as Google Apps Script, provide one possible pathway for rapid educational technology prototyping and institutional integration. Google Apps Script enables the development of web applications, automated workflows, and database interactions using existing Google Workspace infrastructures, thereby simplifying authentication and user management within educational institutions. In contexts where institutions already employ Google Workspace services alongside LMS platforms such as Canvas, these environments may support relatively low-cost integration strategies capable of connecting assessment systems, analytics dashboards, and AI services without requiring dedicated server infrastructure.

At the same time, integrating generative AI into educational workflows introduces additional considerations related to reliability, response quality, latency, and pedagogical appropriateness [11,15]. AI-generated feedback systems must balance automation with educational value, ensuring that generated responses are relevant, understandable, supportive, and aligned with instructional goals [16,17]. Inconsistent or overly generic feedback may reduce learner trust and limit pedagogical effectiveness. Consequently, prompt engineering, contextual information design, and instructor oversight become critical elements in AI-enhanced educational systems.

Another important challenge concerns the interpretation and use of learner analytics. Although educational platforms increasingly collect large volumes of interaction and assessment data, institutions often struggle to transform this information into actionable pedagogical insights [8,27]. Confidence-informed assessment introduces additional complexity because it combines cognitive performance indicators with metacognitive dimensions such as learner certainty and self-perception. While this richer data can support more personalized interventions, it also requires careful interpretation to avoid oversimplification or misclassification of learner behaviors.

The integration of confidence-informed assessment and generative AI nevertheless offers important opportunities for more reflective and personalized educational experiences. Confidence ratings can provide contextual information that allows AI systems to tailor feedback according to learner certainty, helping distinguish between fragile understanding, conceptual misconceptions, and appropriate self-awareness. For example, students demonstrating low confidence despite correct responses may benefit from reinforcement and confidence-building guidance, whereas learners exhibiting high confidence with incorrect responses may require conceptual clarification and metacognitive support. Such distinctions are difficult to identify through traditional grading approaches alone.

Recent developments in AI-enhanced educational products illustrate increasing institutional and commercial interest in these capabilities. Google’s Gemini educational initiatives and AI-supported assessment tools demonstrate a growing movement toward embedding personalized tutoring and adaptive support within digital learning ecosystems. These developments suggest that the future of LMS environments may increasingly involve integrated AI functionalities capable of supporting individualized feedback, learner analytics, and adaptive instructional assistance [36,37].

Within this context, the CONF.i project adopts an integration strategy focused on accessibility, interoperability, and institutional compatibility. The framework was designed to operate using existing Canvas LMS infrastructure, Google Workspace services, and open AI APIs without requiring major institutional reconfiguration or commercial licensing costs. This approach reflects broader educational technology perspectives advocating sustainable and incremental innovation rather than complete platform replacement [41,42]. By leveraging LTI standards, cloud-based scripting environments, and AI feedback services, the project seeks to demonstrate how institutions can augment existing LMS ecosystems with confidence-informed analytics and personalized feedback capabilities while maintaining familiar workflows for instructors and students.

The collaboration between Tecnologico de Monterrey and the University of São Paulo further contributes to this integration perspective by combining expertise in educational technology implementation, intelligent systems, and confidence-informed assessment research. This international collaboration supports both theoretical alignment and practical experimentation, illustrating how cross-institutional partnerships can contribute to the development of scalable and pedagogically grounded educational innovations in higher education.

3. Methodology

3.1. Research Design

This study employed a design-based research (DBR) methodology to guide the development and exploratory evaluation of the CONF.i framework. Design-based research is particularly appropriate for investigating complex educational innovations situated within authentic learning environments because it combines iterative technological development with pedagogical analysis and theory-informed refinement [43]. Unlike purely experimental approaches, DBR emphasizes the interaction between educational theory, technological implementation, and contextual practice, making it well suited for studies involving emerging educational technologies such as learning analytics, adaptive assessment, and generative AI integration [43].

The selection of DBR was motivated by three primary considerations. First, the study aimed to develop and integrate a functional educational technology prototype within an existing institutional LMS environment rather than evaluate a finalized commercial system. Second, the project sought to explore how confidence-informed assessment and generative AI could be operationalized within realistic educational workflows while preserving compatibility with institutional infrastructure. Third, the exploratory nature of the study required an iterative process capable of accommodating technical adjustments, interface refinement, and pedagogical interpretation during development and testing phases.

The research process was organized into four iterative phases:

Phase 1: Analysis and Conceptual Design (Months 1–2)

The initial phase focused on defining pedagogical objectives, technical requirements, and system architecture. During this stage, the research team reviewed literature related to LMS interoperability, confidence-informed assessment, learning analytics, metacognition, and generative AI in education [10,11,34]. The conceptual design of the confidence-informed assessment framework was refined based on prior research concerning multidimensional learner analysis and intelligent tutoring systems.

Collaboration with researchers from the University of São Paulo (USP) played an important role during this phase. Through structured virtual meetings, the research teams discussed the conceptual alignment of the proposed three-variable model (Grade–Confidence–Performance) with existing multidimensional assessment and metacognitive learning frameworks [34,35]. These discussions focused on the educational interpretation of confidence ratings, the practical feasibility of real-time implementation within LMS environments, and the pedagogical role of AI-generated feedback.

Phase 2: Prototype Development and System Integration (Months 3–4)

The second phase involved the development of the CONF.i prototype using Google Apps Script as the primary backend environment. During this stage, the research team implemented:

• Canvas LMS integration through LTI standards;
• Google Workspace authentication workflows;
• confidence-informed assessment interfaces;
• spreadsheet-based data management;
• Gemini AI API integration for automated feedback generation;
• instructor and learner dashboards.

The development process emphasized compatibility with existing institutional infrastructure while minimizing deployment complexity. Iterative internal testing was conducted throughout this phase to refine interface usability, data flows, prompt engineering strategies, and system stability.

The collaborators from USP periodically reviewed the evolving implementation to ensure conceptual consistency between the practical system design and the confidence-informed assessment principles derived from earlier theoretical work. These reviews focused primarily on the interpretation of learner confidence patterns and the educational coherence of the feedback generation process.

Phase 3: Exploratory Prototype Testing (Month 5)

The third phase consisted of exploration prototype testing using simulated learner profiles within an authentic Canvas LMS course environment at Tecnologico de Monterrey. The use of simulated profiles allowed the research team to evaluate system functionality, workflow integration, confidence visualization, and AI feedback generation without involving real student assessment data during this preliminary stage.

Twenty-three simulated learner profiles representing different combinations of performance and confidence patterns were created to test the diagnostic behavior of the framework. These profiles were designed to explore illustrative scenarios such as aligned mastery, underconfident competence, overconfident struggle, and aligned struggle. The testing process examined:

• assessment data capture,
• confidence tracking,
• automated score calculation,
• dashboard visualization,
• AI-generated personalized feedback,
• Canvas grade pass back functionality.

Although the learner performance data were simulated, human evaluators interacted directly with the platform and provided usability observations regarding interface experience, workflow integration, and perceived usefulness of AI-generated feedback.

Phase 4: Evaluation and Iterative Refinement (Month 6)

The final phase focused on exploratory evaluation and refinement of the prototype. Quantitative descriptive data and qualitative usability observations were analyzed to identify strengths, limitations, and improvement opportunities related to system integration, interface usability, confidence-informed assessment visualization, and AI feedback quality.

The DBR process during this phase emphasized reflective analysis rather than hypothesis testing or formal validation. Findings were used to assess the feasibility of integrating confidence-informed assessment and generative AI within existing LMS ecosystems and to identify directions for future research involving authentic learner populations, longitudinal implementation, and larger-scale deployment.

The collaborators from USP participated in the interpretation of the observed confidence-performance patterns and contributed theoretical perspectives regarding metacognition, learner calibration, and assessment-informed feedback processes. This collaborative analysis supported the refinement of the framework while maintaining alignment with the study’s educational objectives and theoretical foundations.

3.2. System Architecture

The CONF.i framework was designed as a modular and interoperable architecture capable of extending the pedagogical functionality of Canvas LMS through the integration of confidence-informed assessment, learning analytics, and generative AI services. The architecture emphasizes institutional compatibility, low deployment complexity, and scalability through the use of existing educational technologies and cloud-based services [19,21,22]. The system follows a layered service-oriented design composed of four interconnected components: (1) Canvas LMS integration layer, (2) Google Apps Script application layer, (3) confidence-informed assessment engine, and (4) Gemini AI feedback services.

Figure 1. presents the overall interaction architecture of the CONF.i framework and the data flow between LMS services, assessment processing, and AI-generated feedback components.

Figure 1. presents the overall interaction architecture of the CONF.i framework and the data flow between LMS services, assessment processing, and AI-generated feedback components.

Component 1: Canvas LMS Integration Layer

The first architectural layer is responsible for interoperability between the CONF.i framework and the institutional Canvas LMS environment. The integration was implemented using Learning Tools Interoperability (LTI 1.3) standards, enabling secure communication between Canvas and external educational services without requiring modifications to the LMS core infrastructure [21,22].

This layer performs four primary functions:

• Secure authentication using institutional OAuth2 credentials;
• Context transfer between Canvas and the external application;
• Role identification (student or instructor);
• Grade synchronization with the Canvas gradebook.

When users launch the CONF.i activity within Canvas, the LMS transmits contextual metadata including course identification, assignment information, institutional user credentials, and role permissions. This process enables the framework to preserve continuity within existing instructional workflows while minimizing additional login procedures and reducing usability friction for instructors and students [19,20].

The use of LTI interoperability standards also supports institutional sustainability by allowing the framework to function as an extensible educational service rather than an isolated platform replacement. This design aligns with recent educational technology perspectives emphasizing incremental LMS augmentation instead of large-scale infrastructure substitution [41,42].

Component 2: Google Apps Script Application Layer

The second architectural layer consists of a cloud-based backend developed using Google Apps Script. This environment was selected because Tecnologico de Monterrey already integrates Google Workspace services institutionally, enabling unified authentication and simplified deployment across educational systems.

The Google Apps Script backend performs several operational functions:

• Dynamic rendering of HTML and JavaScript interfaces;
• Session and authentication management;
• Data collection and processing;
• Communication with Canvas APIs and Gemini AI services;
• Storage and retrieval of assessment records;
• Dashboard generation for students and instructors.

Assessment data, learner responses, confidence selections, and analytics outputs are temporarily stored using Google Sheets databases during the prototype phase. Although spreadsheet-based storage is not intended for large-scale production deployment, it provides a lightweight and accessible infrastructure suitable for rapid educational prototyping and design-based experimentation [43].

The backend also exposes REST-style communication endpoints that enable asynchronous interaction between the user interface and AI services. This asynchronous architecture supports real-time feedback generation while maintaining compatibility with standard web browsers and mobile devices.

The selection of Google Apps Script reflects a broader strategy of leveraging existing institutional infrastructures to reduce implementation costs and technical barriers associated with educational innovation [14,18].

Component 3: Confidence-Informed Assessment Engine

The third layer implements the confidence-informed assessment framework responsible for combining learner performance indicators with self-reported confidence ratings. Unlike traditional LMS grading systems that rely exclusively on correctness-based scoring, this component incorporates metacognitive indicators designed to support richer diagnostic interpretation of learner behavior [7,9,35].

The assessment engine captures three interconnected variables:

Grade: percentage of correct responses representing summative performance;

Confidence: learner self-reported certainty regarding responses;

Performance: composite indicator derived from grade and confidence information.

Confidence values are operationalized using a simplified three-level structure:

• Low confidence = 0%;
• Average confidence = 50%;
• High confidence = 100%.

The composite performance metric is calculated in real time using the following formulation:

$$Performance={\displaystyle \frac{Grade+Confidence}{2}}$$

This formulation was intentionally designed as a lightweight confidence-aware assessment mechanism suitable for formative educational contexts rather than a full psychometric Item Response Theory implementation [5,6]. The objective is to support metacognitive reflection and confidence calibration within authentic LMS workflows while maintaining computational simplicity and rapid processing capability.

The assessment engine additionally manages:

• Question verification;
• Real-time scoring;
• Confidence aggregation;
• Diagnostic pattern classification;
• Dashboard visualization generation.

This component enables the identification of learner profiles such as aligned mastery, underconfident competence, overconfident struggle, and aligned struggle. These patterns provide instructors with additional contextual information that may not be visible through traditional grading approaches alone [32,35].

Component 4: Gemini AI Feedback and Analytics Layer

The fourth architectural layer integrates Google Gemini AI services to generate personalized educational feedback and instructional analytics. The framework employs Gemini API models (gemini-2.0-flash and learnlm-2.0-flash-experimental) to process assessment results, confidence patterns, and contextual learner data.

This layer supports two primary feedback processes:

Student-Oriented Feedback

For students, the AI system generates individualized formative feedback based on:

• correctness of responses,
• confidence alignment,
• detected misconceptions,
• observed metacognitive patterns.

The feedback generated includes:

• reinforcement for correct responses,
• conceptual clarification for incorrect answers,
• comments regarding confidence calibration,
• recommendations for additional learning resources.

This process aligns with research emphasizing the importance of immediate and personalized formative feedback for supporting self-regulated learning and metacognitive development [13,15,16].

Instructor-Oriented Analytics

For instructors, the AI layer synthesizes aggregated class-level analytics including:

• topic-level performance patterns,
• confidence anomalies,
• identification of learners requiring intervention,
• instructional recommendations.

This functionality extends conventional LMS analytics by incorporating metacognitive indicators alongside assessment accuracy, enabling more nuanced interpretation of learner performance [8,27].

To improve pedagogical alignment and reduce generic AI responses, the framework employs structured prompt engineering strategies informed by educational tutoring principles and reflective learning approaches [36,37]. Human instructor oversight remains central to the system design, with AI-generated feedback functioning as decision support rather than autonomous instructional replacement [15,17].

Architectural Design Considerations

The overall CONF.i architecture was guided by four design principles:

• Interoperability: compatibility with existing LMS ecosystems through LTI standards;
• Accessibility: use of institutionally available technologies and services;
• Pedagogical augmentation: support for instructors rather than instructional automation;
• Scalable experimentation: rapid prototyping and iterative refinement through cloud-based services.

By combining LMS interoperability, confidence-informed assessment, and generative AI feedback within a unified workflow, the framework demonstrates how existing educational infrastructures can be extended to support more personalized, reflective, and data-informed learning experiences without requiring substantial institutional reconfiguration [41,42].

3.3. The Assessment Framework and Three-Variable Model Specification

The CONF.i framework adopts a confidence-informed assessment approach designed to extend traditional correctness-based evaluation by incorporating learner self-reported confidence as a complementary metacognitive indicator. The model is conceptually informed by prior research on multidimensional assessment, metacognition, and self-regulated learning [7,13,34,35], while intentionally avoiding the complexity of full psychometric Item Response Theory (IRT) implementations [5,6].

Rather than estimating latent learner traits through probabilistic item calibration, the framework operationalizes a simplified formative assessment model capable of supporting real-time interpretation of learner performance within standard LMS environments. The objective is not to replace established psychometric methodologies, but to provide instructors and learners with richer contextual information regarding certainty, confidence calibration, and potential misconceptions during assessment activities.

The assessment framework is based on three interconnected variables:

• Grade: correctness-based assessment performance;
• Confidence: learner self-reported certainty regarding responses;
• Performance: composite indicator combining correctness and confidence information.

This three-variable structure enables the framework to capture both cognitive and metacognitive dimensions of learner behavior, supporting more nuanced interpretations than conventional percentage scoring alone [9,32,35].

Response Capture Structure

For each assessment item, the system records two independent dimensions:

1. Response Correctness (Grade Component)

Learner responses are automatically compared with instructor-defined correct answers stored within the assessment database. Responses are scored dichotomously:

$${Grade}_i=\left\{\begin{array}{c}1,iftheresponseiscorrect\\ 0.iftheresponseisincorrect\end{array}\right.$$

This component represents the traditional summative assessment dimension focused on observable correctness and content mastery.

2. Confidence Rating (Metacognitive Component)

After answering each question, learners report their perceived confidence using a three-level self-assessment scale:

• Low confidence: “Not sure” → 0.0
• Average confidence: “Reasonably sure” → 0.5
• High confidence: “Very confident” → 1.0

The confidence scale was intentionally simplified to reduce cognitive overload during assessment interaction while still capturing meaningful differences in learner certainty [7,35]. The three-category structure also supports rapid interaction within LMS environments and facilitates real-time computation during formative activities.

This confidence dimension functions as a metacognitive indicator reflecting learners’ perceptions of their own understanding, uncertainty, and decision-making processes during assessment tasks [13,35].

Composite Performance Indicator

To combine cognitive accuracy and metacognitive certainty, the framework calculates a composite performance indicator for each assessment item:

$${Performance}_i={\displaystyle \frac{{Grade}_i+{Confidence}_i}{2}}$$

where:

• Grade_i: represents correctness (0 or 1);
• Confidence_i: represents the normalized confidence value (0.0, 0.5, or 1.0).

The resulting value produces a continuous indicator ranging from 0 to 1 that reflects both learner accuracy and confidence alignment.

Unlike traditional assessment scores, this formulation allows the framework to distinguish between learners with similar correctness outcomes but substantially different metacognitive profiles. For example:

• A learner answering correctly with low confidence may demonstrate fragile or uncertain understanding;
• A learner answering incorrectly with high confidence may exhibit conceptual misconceptions or overconfidence bias;
• A learner demonstrating both high accuracy and appropriate confidence may indicate stronger calibration and conceptual mastery.

These distinctions align with prior research emphasizing the importance of confidence calibration and self-regulated learning processes in educational assessment [9,32,35].

Aggregate Metrics

At the assessment level, the framework computes three aggregated indicators:

$$AverageGrade={\displaystyle \frac{{\sum Grade}_i}{2}}$$

This metric represents the percentage of correct responses across all assessment items.

$$AverageConfidence={\displaystyle \frac{{\sum Confidence}_i}{2}}$$

This metric reflects the learner’s overall self-reported certainty during the assessment process.

$$OverallPerformance={\displaystyle \frac{AverageGrade+AverageConfidence}{2}}$$

This composite metric synthesizes cognitive and metacognitive dimensions into a unified diagnostic indicator suitable for formative interpretation and AI-assisted feedback generation.

Diagnostic Interpretation of Confidence Patterns

A central objective of the framework is to support the identification of confidence-performance alignment patterns that may inform instructional intervention and learner reflection. Based on combinations of correctness and confidence values, the system identifies four illustrative diagnostic profiles:

5.

Aligned Mastery

High correctness with appropriately high confidence, suggesting stable conceptual understanding.

6.

Underconfident Competence

High correctness with low confidence, suggesting uncertainty despite adequate performance and potential need for confidence reinforcement.

7.

Overconfident Struggle

Low correctness with high confidence, suggesting misconceptions, inaccurate self-assessment, or insufficient conceptual monitoring.

8.

Aligned Struggle

Low correctness with appropriately low confidence, suggesting learner awareness of conceptual difficulty and potential openness to remediation.

These patterns are not intended as psychometric classifications or formal learner diagnoses. Instead, they function as exploratory formative indicators capable of supporting metacognitive reflection, personalized feedback, and instructor interpretation within digital learning environments [13,35].

Pedagogical Rationale

The integration of confidence ratings into assessment workflows is grounded in broader research on self-regulated learning and metacognition, which emphasizes learners’ abilities to monitor understanding, evaluate uncertainty, and regulate learning strategies [7,13,35]. Traditional LMS grading systems typically capture only correctness outcomes, limiting the visibility of important cognitive and affective dimensions associated with learner decision-making.

By incorporating confidence-aware indicators, the CONF.i framework seeks to:

• support reflective learning processes;
• improve learner awareness of knowledge limitations;
• provide richer contextual information for AI-generated feedback;
• assist instructors in identifying hidden misconceptions and fragile understanding.

This approach also aligns with emerging perspectives in learning analytics that advocate multidimensional interpretation of learner behavior rather than reliance on single-score performance metrics alone [8,27].

Computational and Practical Considerations

The simplified three-variable formulation was intentionally designed to support:

• real-time processing within Google Apps Script environments;
• compatibility with LMS workflows;
• low computational overhead;
• accessibility for instructors without advanced psychometric training.

The model therefore prioritizes interpretability, usability, and pedagogical integration over psychometric sophistication. This design choice reflects the exploratory and formative objectives of the CONF.i framework and supports scalable experimentation within institutional educational settings [41,42,43].

3.4. Gemini AI Prompt Engineering

The CONF.i framework employs structured prompt-engineering strategies to support the generation of pedagogically aligned formative feedback through Google Gemini AI services. Prompt engineering was treated as a central design component because the quality, specificity, and educational usefulness of AI-generated responses depend heavily on the contextual structure and instructional guidance provided to the model [11,15,16]. The framework integrates Gemini models (gemini-2.0-flash and learnlm-2.0-flash-experimental) through the Google AI API to generate both student-level personalized feedback and instructor-level learning analytics summaries. The design of prompts was informed by prior research on intelligent tutoring systems, formative feedback, self-regulated learning, and metacognitive support [7,13,15,35], as well as recent educational AI initiatives emphasizing reflective and learner-centered interaction strategies [36,37].

Rather than relying on open-ended conversational prompting, the system employs structured instructional prompts that constrain output behavior and guide the AI model toward educationally appropriate responses. This strategy was implemented to reduce generic feedback, improve contextual relevance, and maintain alignment with formative assessment principles [15,17]. The prompt-engineering process followed five core pedagogical design principles: contextualization, constructive tone, metacognitive reflection, actionable guidance, and instructor augmentation.

The first principle, contextualization, emphasized that feedback should incorporate both assessment correctness and learner confidence patterns rather than relying exclusively on numerical scores. The second principle focused on maintaining a constructive and supportive tone consistent with formative assessment practices and growth-oriented instructional feedback [13]. The third principle emphasized metacognitive reflection by encouraging learners to examine confidence calibration, uncertainty, misconceptions, and reasoning processes [7,35]. The fourth principle required that AI-generated responses provide actionable guidance, including concrete suggestions for improvement and recommendations for additional learning resources. Finally, the fifth principle, instructor augmentation, established that AI-generated outputs should support instructor decision-making rather than replace human pedagogical judgment or instructional oversight [15,17]. These principles guided iterative refinement of prompts throughout prototype development and internal testing phases.

For individualized learner feedback, the system generates structured prompts incorporating learner responses, correctness indicators, confidence ratings, assessment metadata, and topic-specific information. The Gemini model receives instructional guidance directing it to provide encouraging and constructive feedback that acknowledges correct responses, explains incorrect responses, comments on confidence-accuracy alignment patterns, recommends additional learning resources, and maintains a supportive educational tone. Responses are additionally formatted in structured HTML to support readability and integration within the Canvas LMS interface.

This prompt structure was intentionally designed to encourage the AI model to integrate both cognitive and metacognitive dimensions during feedback generation rather than merely reporting correctness outcomes. For example, correct answers combined with low confidence trigger reinforcement-oriented responses intended to strengthen learner self-efficacy, while incorrect answers associated with high confidence generate reflective prompts focused on conceptual recalibration and awareness of misconception. This approach aligns with research emphasizing that effective formative feedback should support both conceptual understanding and learner self-awareness [13,15,35].

In addition to individualized learner feedback, the framework generates aggregated instructional summaries intended to support instructor interpretation of class-wide assessment patterns. These summaries synthesize assessment performance trends, confidence distributions, recurring misconceptions, and potential intervention needs. The instructor-oriented prompts direct the Gemini model to identify class strengths, areas requiring improvement, problematic confidence patterns, learners potentially requiring additional support, suggested instructional adjustments, and recommended resources for class-wide review activities.

The instructor-facing prompts were intentionally designed to produce concise pedagogical summaries rather than fully automated instructional decisions. Consequently, the system functions as an analytics support mechanism that assists in identifying patterns requiring pedagogical attention while preserving human oversight, instructional autonomy, and professional judgment [15,17]. This augmentation-oriented approach reflects broader perspectives in educational AI research advocating responsible integration of generative AI systems within instructional environments.

During prototype development, multiple prompt iterations were tested internally to improve feedback specificity, pedagogical coherence, response consistency, HTML formatting quality, and reduction of repetitive phrasing. Initial prompt versions frequently generated overly generic encouragement, broad resource recommendations, insufficient conceptual explanations, and inconsistent interpretation of confidence patterns. To address these limitations, additional contextual constraints and explicit instructional guidance were progressively incorporated into the prompts.

These refinements substantially improved the model’s ability to distinguish between under confidence and overconfidence, provide more targeted conceptual guidance, maintain supportive instructional language, and generate more structured educational responses. The iterative refinement process reflects broader findings in educational AI research demonstrating that pedagogically informed prompt design significantly influences the usefulness, consistency, and instructional reliability of AI-generated outputs [11,15,36].

The CONF.i framework additionally adopts an augmentation-oriented approach to AI integration within educational environments. Gemini-generated feedback is intended to complement instructional processes rather than replace educator expertise or automated grading systems [15,17]. Consequently, instructors retain full control over grading decisions, while AI-generated responses function as formative support mechanisms subject to instructor review and interpretation. This design perspective also addresses emerging concerns regarding transparency, reliability, accountability, and responsible AI use in educational contexts [14,17,38].

Because generative AI systems may occasionally produce inaccurate, incomplete, or overly generic responses, the framework emphasizes instructor oversight, reflective use of AI-generated feedback, and formative rather than high-stakes application. Furthermore, the integration of confidence-informed assessment data into prompt generation introduces an additional layer of pedagogical contextualization that may improve the relevance and specificity of AI-generated feedback compared with correctness-only assessment systems. By incorporating learner certainty and confidence calibration into the prompting process, the framework supports more nuanced educational responses aligned with metacognitive learning principles [7,35].

Operationally, prompt generation follows a structured technical workflow integrated within the Canvas LMS environment. Assessment responses and confidence ratings are first captured through the LMS interface and transmitted to the Google Apps Script backend. The backend subsequently aggregates learner data and dynamically constructs structured prompts for submission to the Gemini API services. API requests are transmitted asynchronously, after which AI-generated feedback is parsed and rendered directly within the LMS interface. This architecture enables near real-time personalized feedback generation while maintaining interoperability with existing institutional infrastructure and LMS workflows [21,22,41].

3.6. Data Collection Instruments

This exploratory study employed a mixed-methods data collection strategy to examine the technical feasibility, usability, and pedagogical potential of the CONF.i framework within an authentic LMS environment. Consistent with design-based research (DBR) methodology, the objective of data collection was not to establish causal learning effects, but rather to generate evidence regarding system functionality, user interaction, confidence-informed assessment patterns, and perceptions of AI-generated feedback during prototype implementation [43]. The data collection process combined quantitative system-generated indicators with qualitative observational and reflective instruments, enabling triangulation between platform interaction data, usability perceptions, and interpretive feedback regarding the confidence-informed assessment experience [13,15].

Quantitative Instruments

Usability and User Experience Survey

A structured post-interaction usability questionnaire was developed to capture participant perceptions regarding interface usability, clarity of assessment workflows, perceived usefulness of AI-generated feedback, confidence-rating interaction, and overall platform experience. The instrument employed a five-point Likert scale ranging from 1 (“strongly disagree”) to 5 (“strongly agree”).

The questionnaire included ten evaluative items organized into four principal dimensions. The first dimension focused on interface usability, including ease of navigation, clarity of interaction, and perceived integration within the Canvas LMS environment. The second dimension examined the assessment experience, particularly the ease of confidence selection and clarity of assessment presentation. The third dimension evaluated AI feedback quality through participant perceptions of relevance, usefulness, and personalization of the generated feedback. The final dimension explored general satisfaction, including overall experience, perceived educational value, and willingness to use the system in future courses.

The survey instrument was intentionally designed as an exploratory usability measure appropriate for prototype evaluation rather than as a validated psychometric scale. Its primary purpose was to identify usability tendencies, perceived strengths, and areas requiring refinement during early-stage system development [43].

System Usage Logs

The CONF.i platform automatically recorded interaction and operational data through the Google Apps Script backend. These system-generated logs provided objective information regarding user interaction patterns and technical performance during the pilot implementation.

Captured variables included session initiation and completion timestamps, assessment submission times, confidence selections, API response latency, dashboard access frequency, grade synchronization events, and AI feedback generation status. These logs were used primarily to evaluate workflow continuity, integration stability, response performance, and the reliability of asynchronous communication between LMS services and Gemini AI components.

The automated collection of interaction data additionally supported verification of successful Learning Tools Interoperability (LTI) integration and assessment processing workflows within the Canvas LMS environment [21,22]. The resulting operational data provided important evidence regarding the practical feasibility of integrating generative AI services and confidence-informed analytics into institutional learning management infrastructures.

Confidence-Informed Assessment Metrics

The assessment engine generated quantitative learner indicators derived from the three-variable model described previously, specifically Grade, Confidence, and Performance. For each simulated learner profile, the framework calculated item-level correctness, average confidence ratings, composite performance indicators, and confidence-performance alignment patterns.

These metrics enabled identification of illustrative learner archetypes including aligned mastery, underconfident competence, overconfident struggle, and aligned struggle. The resulting quantitative outputs were used descriptively to explore how confidence-informed assessment may reveal metacognitive dimensions not visible through conventional grading approaches alone [9,32,35]. The metrics additionally supported interpretation of learner calibration patterns and informed the generation of adaptive AI-supported formative feedback.

Qualitative Instruments

Open-Ended Reflective Questions

To complement the quantitative usability measures, participants were invited to provide written reflections regarding their experience using the CONF.i system. Four open-ended prompts were included to encourage qualitative feedback concerning perceptions of the confidence-rating mechanism, usefulness of AI-generated feedback, interface usability, and recommendations for future improvement.

These prompts allowed participants to elaborate on perceived strengths and weaknesses, emotional reactions, metacognitive reflections, and instructional suggestions associated with the platform experience. The inclusion of open-ended responses aligns with DBR approaches that prioritize iterative refinement informed by authentic user experience and contextual interpretation [43].

Instructor Observation Notes

Throughout the pilot implementation, observational notes were documented by members of the research team during assessment administration and feedback review sessions. These observations focused on user interaction behavior, navigation difficulties, participant questions, workflow interruptions, perceptions regarding confidence reporting, and reactions to AI-generated feedback.

The observational process also captured implementation-related considerations including explanation requirements for confidence ratings, instructor interpretation of analytics dashboards, and operational limitations associated with system latency. These field observations contributed contextual information that complemented system logs and participant survey responses while supporting iterative refinement of platform architecture and instructional workflows.

Written Feedback Comments

Participants were additionally invited to provide voluntary written comments regarding feedback specificity, perceived usefulness, AI response quality, and desired future functionalities. Sixteen participants submitted narrative comments that contributed directly to the thematic analysis process described in Section 3.8.

These comments were particularly useful for identifying perceptions of personalization, concerns regarding response delays, reactions to confidence-informed assessment, and expectations for future AI-enhanced learning support. The narrative feedback additionally revealed varying participant attitudes toward the role of generative AI within LMS-supported educational environments.

Cross-Institutional Validation Notes

As part of the collaborative research process between Tecnologico de Monterrey and the University of São Paulo (USP), theoretical interpretation notes were documented during collaborative review sessions involving the Brazilian research partners. These discussions focused on conceptual consistency of the three-variable framework, interpretation of confidence-performance patterns, alignment with prior multidimensional assessment research, and pedagogical coherence of AI-generated feedback structures.

The resulting validation notes supported reflective refinement of the assessment framework and contributed theoretical perspectives regarding metacognitive interpretation and learner calibration [34,35]. This collaborative process also strengthened alignment between the exploratory implementation findings and broader theoretical perspectives in learning analytics and confidence-informed assessment research.

Instrumentation Scope and Study Limitations

Because this study focused on exploration prototype evaluation rather than large-scale effectiveness testing, the selected instruments prioritized usability exploration, workflow analysis, formative interpretation, and system feasibility. Consequently, the instruments were not intended to produce validated psychometric measures, inferential statistical comparisons, or longitudinal evidence of learning outcomes.

The collected data should therefore be interpreted as preliminary evidence intended to support iterative refinement and future large-scale investigation rather than as definitive validation of educational effectiveness [43].

3.7. Qualitative Instruments

Qualitative data collection was incorporated to support reflective analysis of user interaction, perceived pedagogical value, and implementation dynamics associated with the CONF.i framework. In alignment with design-based research (DBR) principles, the qualitative component emphasized contextual interpretation, iterative refinement, and exploratory understanding of how participants interacted with confidence-informed assessment and AI-generated feedback within an authentic LMS environment [43]. Because the study focused on prototype feasibility rather than formal evaluation of educational effectiveness, the qualitative instruments were designed to capture participant perceptions, usability observations, metacognitive reflections, instructional implementation insights, and recommendations for system improvement.

The qualitative strategy combined multiple data sources, including open-ended participant responses, instructor observation notes, written narrative comments, and collaborative interpretive discussions involving the Brazilian research collaborators. This multimodal qualitative approach supported triangulation of participant experiences and implementation observations while contributing to iterative refinement of the framework.

Open-Ended Reflective Responses

Participants were invited to provide written reflections following their interaction with the CONF.i platform. The reflective prompts were intentionally broad to encourage authentic responses regarding the confidence-rating experience, perceptions of AI-generated feedback, usability and workflow interaction, perceived educational usefulness, and suggestions for future improvement. Participants were encouraged to discuss both positive and negative aspects of the experience while reflecting on how confidence reporting influenced their thinking during assessment tasks.

This instrument was particularly important for exploring metacognitive dimensions associated with confidence-informed assessment, including awareness of uncertainty, self-evaluation behaviors, confidence calibration, and perceptions of conceptual understanding. The use of reflective prompts aligns with prior research emphasizing the importance of learner self-awareness and metacognitive interpretation within formative assessment environments [7,13,35].

Instructor Observation Notes

During pilot implementation, structured observational notes were recorded by members of the research team while participants interacted with the system. Observations focused on participant navigation behavior, interface usability difficulties, reactions to confidence selection, engagement with AI-generated feedback, workflow interruptions, and verbal comments expressed during interaction sessions. Special attention was given to how participants interpreted the confidence-reporting mechanism and whether additional explanation or clarification was required during assessment administration.

The observational process also documented instructor interaction with analytics dashboards, feedback summaries, learner profile visualizations, and Canvas LMS integration workflows. These observations provided contextual information regarding the practical integration of the framework within authentic instructional processes and contributed directly to iterative system refinement [43].

Written Narrative Feedback

Participants were additionally encouraged to submit optional written comments describing perceived strengths of the framework, limitations of the AI-generated responses, recommendations for improving feedback quality, and desired future functionalities. Sixteen participants provided narrative feedback comments, which contributed substantially to identifying perceptions of personalization, concerns regarding response latency, reactions to confidence-informed assessment, and expectations regarding adaptive learning support.

Narrative responses additionally revealed differences in how participants interpreted the educational role of generative AI within LMS environments, particularly regarding trust in AI-generated explanations, expectations for feedback specificity, and perceptions of instructional usefulness. These responses provided valuable contextual insight into participant expectations concerning AI-supported formative assessment systems.

Cross-Institutional Reflective Validation

As part of the collaborative research process between Tecnologico de Monterrey and the University of São Paulo (USP), qualitative interpretation sessions were conducted with the Brazilian collaborators following the pilot implementation phase. These discussions focused on interpretation of confidence-performance alignment patterns, pedagogical implications of overconfidence and under confidence profiles, consistency with prior multidimensional assessment research, and educational coherence of AI-generated feedback structures.

The collaborative review process supported theoretical triangulation between practical implementation observations and the conceptual foundations underlying the confidence-informed assessment framework [34,35]. The involvement of collaborators with expertise in multidimensional assessment and intelligent tutoring systems contributed additional interpretive depth while supporting refinement of the theoretical framework guiding the CONF.i model.

Pilot Procedure

The pilot implementation was conducted within a Canvas LMS course environment at Tecnologico de Monterrey using simulated learner profiles developed by the research team. The objective of the pilot was to evaluate workflow integration, assessment functionality, confidence-informed visualization, and AI-generated feedback behavior under realistic instructional conditions.

The pilot procedure was organized into five sequential stages. During the first stage, participants received an orientation introducing the purpose of the confidence-informed assessment model, the role of confidence ratings, and interaction procedures within the CONF.i interface. Emphasis was placed on clarifying that confidence ratings would not directly affect grades but instead would support reflective feedback and diagnostic interpretation. This clarification was intended to encourage honest confidence reporting and reduce anxiety associated with self-assessment activities [35]. Participants were also introduced to the conceptual distinction between correctness, confidence, and composite performance indicators.

In the second stage, participants completed a five-question mathematics assessment using the CONF.i interface embedded within Canvas LMS. For each item, participants selected an answer and reported their confidence level using the three-option confidence scale integrated into the assessment workflow. The interface presented assessment questions, confidence options, response submission controls, and LMS navigation features within a unified environment. Figure 2, Figure 3 and Figure 4 illustrate representative examples of the assessment interface and confidence-selection workflow.

The third stage involved AI feedback generation. Following assessment submission, the Google Apps Script backend transmitted response data and confidence indicators to the Gemini AI services. The AI system subsequently generated individualized formative feedback, confidence-alignment observations, conceptual guidance, and resource recommendations. Generated feedback was rendered directly within the Canvas-integrated interface to preserve workflow continuity and maintain the perception of a unified learning environment [21,22].

During the fourth stage, participants completed the usability questionnaire and open-ended reflection prompts after reviewing the generated feedback. Simultaneously, members of the research team documented interaction observations, participant comments, navigation issues, technical limitations, and reactions to AI-generated responses. These observations were used to identify usability strengths and opportunities for refinement during later development iterations.

The pilot concluded with a brief debrief and reflective discussion facilitated by the instructor-researcher. Participants were encouraged to discuss their perceptions of confidence-informed assessment, the usefulness of AI-generated feedback, potential classroom applications, and concerns regarding AI integration in education. These discussions generated additional contextual insights regarding participant expectations, concerns, and perceived educational value.

Scope and Interpretive Considerations

It is important to emphasize that the assessment profiles and scoring data used during the pilot were simulated for exploratory testing purposes. Although real participants interacted with the interface and provided usability feedback, no authentic student academic records or high-stakes assessment data were collected during this phase. Consequently, the qualitative findings should be interpreted as exploratory usability observations, preliminary implementation insights, and reflective perceptions regarding AI-supported formative assessment rather than validated evidence of learning effectiveness or long-term pedagogical impact [43].

3.8. Data Analysis

Data analysis followed a mixed-methods exploration strategy consistent with the design-based research (DBR) orientation adopted in this study [43]. Because the primary objective was to evaluate the feasibility, usability, and pedagogical potential of the CONF.i framework rather than to test causal hypotheses or measure instructional effectiveness, the analysis emphasized descriptive interpretation, pattern identification, and iterative system refinement. Quantitative and qualitative data sources were analyzed separately and subsequently integrated through interpretive triangulation to support a broader understanding of system behavior, participant perceptions, and confidence-informed assessment patterns.

Quantitative Data Analysis

Quantitative analysis focused on three principal categories of data: usability questionnaire responses, system interaction logs, and confidence-informed assessment indicators. Given the exploratory nature of the pilot implementation, the use of simulated learner profiles, and the limited participant sample, inferential statistical testing was not considered appropriate. Instead, descriptive statistical procedures and visual analytic techniques were employed to identify trends, distributions, and illustrative behavioral patterns [43].

Responses from the post-interaction Likert-scale usability questionnaire were analyzed using descriptive statistics including means, standard deviations, response frequencies, and percentage distributions. The analysis examined participant perceptions related to interface usability, ease of confidence reporting, usefulness of AI-generated feedback, and overall satisfaction with the LMS integration workflow. Positive-response aggregation was additionally calculated to summarize participant agreement regarding the perceived educational value and usability of the framework. Because the instrument functioned as an exploratory usability measure rather than a formally validated psychometric scale, results were interpreted descriptively and used primarily to identify strengths and areas requiring refinement during subsequent development iterations.

System-generated logs collected through the Google Apps Script backend were analyzed to evaluate workflow continuity, platform responsiveness, AI feedback generation performance, and operational stability of the LMS integration environment. The analysis included response latency measurements, frequency of API failures or retries, assessment submission completion rates, dashboard access patterns, and grade synchronization success rates. Particular attention was given to response times associated with Gemini API feedback generation and spreadsheet-based database operations, as these represented potential technical constraints for future scalability. These analyses supported evaluation of the practical feasibility of integrating Canvas LMS, Google Apps Script services, confidence-informed analytics, and generative AI feedback mechanisms within existing institutional infrastructure [21,22].

The assessment framework additionally generated quantitative indicators derived from the three-variable models described in Section 3.3, specifically Grade, Confidence, and Performance. For each simulated learner profile, the system calculated percentage correctness, average confidence level, composite performance scores, and confidence-performance alignment patterns. The analysis focused on identifying illustrative learner profiles associated with aligned mastery, underconfident competence, overconfident struggle, and aligned struggle. These patterns were interpreted descriptively to explore how confidence-informed assessment may reveal metacognitive dimensions that are not observable through conventional correctness-based grading alone [9,32,35]. Visual dashboards and comparative profile displays were also used to examine relationships among learner correctness, confidence calibration, and AI-generated feedback responses. The objective of this analysis was exploratory and interpretive rather than predictive or diagnostic in a formal psychometric sense.

Qualitative Data Analysis

Qualitative data consisted of open-ended participant responses, narrative usability comments, instructor observation notes, reflective debriefing discussions, and cross-institutional validation notes. These data were analyzed using thematic analysis following the six-phase interpretive framework commonly employed in qualitative educational research [43]. The analytical process included data familiarization, initial code generation, theme identification, theme review, theme refinement and definition, and interpretive synthesis.

Two members of the research team independently reviewed and coded participant comments and observational notes to identify recurring themes associated with metacognitive awareness, perceptions of AI-generated feedback, usability and workflow interaction, confidence-reporting behaviors, technical limitations, and perceived instructional value. Initial coding combined semantic coding, focused on explicit participant statements, with interpretive coding directed toward underlying perceptions and implications. Following independent coding, the researchers compared code assignments and discussed discrepancies to refine category definitions and improve interpretive consistency. Inter-coder agreement reached approximately 87% during the reconciliation process, while remaining differences were resolved through collaborative discussion and consensus-building among the researchers. Because the study emphasized exploratory interpretation rather than formal qualitative validation, the thematic analysis was intended to support reflective understanding and iterative refinement rather than establish definitive categorical generalizations.

Codes were subsequently grouped into broader interpretive themes reflecting recurring participant experiences and implementation observations. Particular attention was given to themes associated with confidence calibration awareness, perceptions of personalization, reactions to AI-generated explanations, technical responsiveness, and expectations regarding adaptive learning support. Themes were iteratively refined to ensure conceptual coherence and alignment with the study’s research questions concerning confidence-informed assessment, AI-supported formative feedback, and LMS integration feasibility. Representative participant comments were selected to illustrate dominant perceptions while preserving anonymity and contextual interpretation.

Cross-Institutional Interpretive Validation

Following the initial analysis phase, collaborative interpretation sessions were conducted with research collaborators from the University of São Paulo (USP). These sessions focused on interpretation of confidence-performance patterns, pedagogical implications of learner calibration profiles, conceptual consistency with prior multidimensional assessment research, and coherence of AI-generated feedback structures. This collaborative review process functioned as a form of theoretical triangulation, helping align the practical implementation findings with broader metacognitive and confidence-informed assessment perspectives [34,35].

The involvement of collaborators with prior expertise in multidimensional assessment and intelligent tutoring systems contributed additional interpretive depth to the analysis while supporting refinement of the conceptual framework underlying the CONF.i model. This process strengthened analytical consistency and improved alignment between the exploratory implementation findings and established theoretical perspectives in learning analytics and metacognitive assessment research.

Integration of Quantitative and Qualitative Findings

The final stage of analysis involved interpretive integration of quantitative and qualitative findings. This triangulation process examined how usability perceptions, confidence-informed metrics, participant reflections, and technical performance observations collectively informed understanding of the framework’s feasibility and pedagogical potential. The integrated analysis supported evaluation of the practicality of confidence-informed assessment within LMS workflows, participant perceptions regarding AI-generated formative feedback, operational strengths and limitations of the prototype architecture, and opportunities for future refinement and large-scale investigation.

Consistent with DBR methodology, the purpose of this integrated analysis was not to establish definitive claims regarding educational effectiveness, but rather to generate evidence capable of informing iterative system improvement, conceptual refinement, and future research directions [43].

4. Results

4.1. Technical Integration Feasibility (RQ1)

The implementation and pilot testing of the CONF.i framework demonstrated the technical feasibility of integrating confidence-informed assessment and generative AI services within an existing Canvas LMS environment using institutionally available infrastructure. The results indicate that interoperability between Canvas LMS, Google Workspace services, Google Apps Script, and Gemini AI APIs can be achieved without requiring major institutional reconfiguration, dedicated server deployment, or proprietary adaptive learning platforms [19,20,21]. The technical evaluation focused on four operational dimensions: LMS interoperability and authentication, backend application performance, confidence-informed assessment processing, and AI-generated feedback integration. Across all dimensions, the prototype successfully completed the intended instructional workflow during pilot testing.

The CONF.i framework was successfully integrated into Canvas LMS using Learning Tools Interoperability (LTI 1.3) standards. The external tool configuration enabled instructors to create and deploy CONF.i assessments directly within existing Canvas assignments while preserving standard course workflows and institutional authentication processes [21,22]. The integration process supported seamless assignment launching from Canvas, automatic transfer of course and assignment context, institutional single sign-on authentication, user role identification, and grade synchronization with the Canvas gradebook. When participants accessed the assessment activity through Canvas, the LMS correctly transmitted course identifiers, assignment metadata, user roles, and institutional email credentials. Role-based rendering functioned consistently during the pilot implementation, presenting instructor dashboards for faculty users and assessment interfaces for learner users. No additional account creation or external authentication procedures were required, reducing interaction complexity and preserving continuity within the institutional LMS ecosystem. These results support prior research suggesting that interoperability and workflow compatibility are critical factors influencing institutional adoption of educational technologies [19,20,41]. Figure 2 illustrates the Canvas assignment creation interface with the CONF.i external tool integrated through the LMS workflow.

The Google Apps Script backend successfully managed the operational workflow of the prototype during pilot testing. The cloud-based application handles session management, data collection and storage, confidence tracking, dashboard rendering, and communication with Gemini AI services. The backend architecture maintained stable functionality throughout the pilot implementation involving twenty-three simulated learner profiles and concurrent interactions within the Canvas environment. Spreadsheet-based storage mechanisms correctly recorded assessment responses, confidence selections, calculated performance indicators, and AI-generated feedback outputs. The backend additionally supported asynchronous communication between Canvas LMS, Google Workspace services, Gemini AI APIs, and dashboard visualization interfaces. This architecture enabled near real-time assessment processing and feedback generation while avoiding the need for dedicated institutional server infrastructure.

Although the system remained operational throughout the pilot, several technical limitations were observed. Spreadsheet write operations occasionally introduced latency, API response times varied depending on Gemini service availability, and isolated interface refresh events were required following delayed responses. Observed response delays ranged between approximately 3- and 7-seconds during database operations and AI feedback generation. These findings are consistent with previously reported scalability and responsiveness limitations associated with Google Apps Script environments in educational prototyping contexts [14,18]. Despite these limitations, the prototype-maintained workflow continuity and demonstrated the practical viability of low-cost cloud-based integration strategies for educational innovation.

The confidence-informed assessment engine also functioned consistently during all pilot assessment sessions. The system correctly captured and processed learner responses, confidence selections, grade calculations, and composite performance indicators. The three-variable framework described in Section 3.3 was calculated dynamically in real time, allowing immediate generation of confidence metrics, performance visualizations, and diagnostic learner profiles. The interface consistently supported confidence selection using the three-level structure of low confidence, average confidence, and high confidence. All assessment variables were correctly aggregated and displayed within both learner and instructor dashboards.

In addition to real-time scoring, the assessment engine supported item-level correctness verification, automated calculation of confidence-performance alignment patterns, and generation of visual analytics summaries. The resulting dashboards enabled the identification of illustrative patterns including aligned mastery, underconfident competence, overconfident struggle, and aligned struggle. These findings demonstrate that confidence-informed assessment workflows can be integrated into standard LMS environments without requiring advanced psychometric infrastructure or specialized assessment platforms [9,32,35]. The Canvas grade pass back mechanism also functioned correctly throughout the pilot implementation, allowing assignment grades generated within CONF.i to synchronize automatically with the Canvas gradebook. This feature preserved the LMS as the institutional “source of truth” for academic records while extending assessment functionality through the external application layer.

The Gemini AI integration layer successfully generated personalized formative feedback for all simulated learner profiles during pilot testing. The framework employed Gemini API services (gemini-2.0-flash and learnlm-2.0-flash-experimental) to process assessment correctness, confidence indicators, learner performance patterns, and instructional prompt templates. AI-generated responses consistently incorporated reinforcement for correct answers, conceptual guidance for incorrect responses, comments regarding confidence alignment, and resource recommendations for further review. Average AI response time during pilot testing was approximately 4.2 seconds (SD = 1.8), although occasional response variability and isolated timeout events required retry logic within the backend architecture.

Despite these inconsistencies, the prompt-engineering framework described in Section 3.4 enabled the system to generate feedback that was contextually relevant, pedagogically structured, responsive to confidence patterns, and integrated within the LMS workflow. The AI-generated feedback was rendered directly inside the Canvas-integrated interface, preserving interaction continuity and reducing the perception of transitioning to external systems. This integration approach aligns with broader educational technology research emphasizing the importance of embedded and workflow-compatible AI support systems [15,17].

From a systems integration perspective, the pilot demonstrated that the complete instructional workflow could be executed successfully within existing institutional infrastructure. The workflow included assessment launching through Canvas LMS, institutional authentication transfer, real-time capture of responses and confidence ratings, dynamic calculation of composite metrics, asynchronous Gemini AI feedback generation, dashboard rendering for learners and instructors, and automatic grade synchronization back to Canvas. This end-to-end process demonstrates the feasibility of combining LMS interoperability, confidence-informed assessment, cloud-based scripting environments, and generative AI services within a unified educational ecosystem.

The results further suggest that institutions already employing Canvas LMS, Google Workspace infrastructure, and LTI-compatible services may be able to implement similar AI-enhanced formative assessment frameworks with relatively low deployment barriers and reduced infrastructure costs [41,42]. Instructor observations additionally highlighted the practical advantages of maintaining compatibility with familiar LMS workflows. The instructor responsible for pilot implementation reported that assignment creation and deployment remained consistent with standard Canvas procedures, requiring minimal additional technical configuration. From an instructional perspective, the integrated workflow reduced administrative friction while enabling access to confidence-informed learner analytics, automated formative feedback, and class-level performance summaries. These observations support the broader design principle underlying the CONF.i framework: educational AI systems may achieve greater institutional acceptance when they augment existing instructional ecosystems rather than requiring complete platform replacement [15,17,41].

The Figure 2 shows the Canvas assignment creation interface with the CONF.i external tool selected. The image demonstrates how instructors can create a new assignment and select the “IRT task type” option, illustrating the integration with the existing Canvas workflow.

Instructor quote: “The integration felt seamless from my perspective. I created the assignment in Canvas as usual, selected the CONF.i tool, and the system managed everything else. No IT support tickets required.”

4.2. The Three-Variable Model Insights (RQ2)

The second research question examined the types of diagnostic and pedagogical insights that emerged from the implementation of the confidence-informed three-variable assessment model. The results suggest that integrating learner confidence alongside conventional correctness-based grading enabled the framework to capture additional metacognitive dimensions that were not visible through traditional assessment metrics alone. Specifically, the inclusion of confidence indicators allowed the system to differentiate between learners who achieved similar correctness outcomes but demonstrated substantially different levels of certainty, calibration, and perceived understanding [7,9,35].

The three-variable model generated three interconnected indicators during assessment processing: Grade, Confidence, and Performance. Grade represented correctness-based achievement, Confidence reflected learner self-reported certainty, and Performance synthesized both dimensions into a composite indicator. The resulting outputs provided a broader interpretive view of learner behavior by incorporating both cognitive performance and metacognitive self-assessment within the same analytical framework.

$$Performance={\displaystyle \frac{Grade+Confidence}{2}}$$

During pilot implementation, the framework consistently identified distinct confidence-performance alignment patterns across the simulated learner profiles. These patterns demonstrated that correctness alone was insufficient to fully characterize learner understanding and assessment behavior. For example, some profiles achieved high scores while simultaneously reporting low confidence levels. In conventional LMS grading systems, these learners would likely appear indistinguishable from highly confident high-performing learners. However, the confidence-informed framework revealed potential uncertainty, fragile understanding, or reduced self-efficacy despite acceptable correctness outcomes [13,35].

Conversely, the system also identified learner profiles characterized by high confidence combined with low correctness. These cases were interpreted as illustrative examples of possible overconfidence or conceptual misunderstanding. From an instructional perspective, these patterns may be particularly relevant because highly confident incorrect responses can indicate misconceptions that learners may not independently recognize or seek to correct [9,32]. The framework therefore enabled instructors to distinguish between learners requiring conceptual remediation and learners primarily requiring confidence reinforcement or metacognitive support.

The resulting diagnostic patterns were categorized into four illustrative learner profiles: aligned mastery, underconfident competence, overconfident struggle, and aligned struggle. Learners classified within the aligned mastery category demonstrated both high correctness and high confidence, suggesting stable conceptual understanding and effective confidence calibration. Underconfident competence profiles demonstrated strong correctness outcomes combined with low confidence, potentially indicating uncertainty despite adequate content mastery. Overconfident struggle profiles combined low correctness with high confidence, suggesting possible misconceptions or inaccurate self-assessment behaviors. Finally, aligned struggle profiles demonstrated both low correctness and low confidence, indicating awareness of conceptual difficulty and potential readiness for instructional intervention [35].

Table 1. Distribution of Simulated Learner Archetypes.

Archetype Profile	Definition Criteria	Count	Percentage
A: Aligned Mastery	High Grade (>80%), High Confidence	7	30.4%
B: Underconfident	Competence High Grade (>80%), Low Confidence	6	26.1%
C: Overconfident Struggle	Low Grade (<60%), High Confidence	6	26.1%
D: Aligned Struggle	Low Grade (<60%), Low Confidence	4	17.4%

Table 1. Distribution of Simulated Learner Archetypes.

Archetype Profile	Definition Criteria	Count	Percentage
A: Aligned Mastery	High Grade (>80%), High Confidence	7	30.4%
B: Underconfident	Competence High Grade (>80%), Low Confidence	6	26.1%
C: Overconfident Struggle	Low Grade (<60%), High Confidence	6	26.1%
D: Aligned Struggle	Low Grade (<60%), Low Confidence	4	17.4%

These distinctions illustrate how confidence-informed assessment may provide richer contextual information than traditional grading systems that rely exclusively on correctness-based evaluation. The framework therefore supports emerging perspectives in learning analytics and metacognitive assessment that advocate multidimensional interpretation of learner behavior rather than single-score performance models [8,27,35].

The confidence dimension additionally influenced the AI-generated feedback process described in Section 3.4. Because the Gemini prompt-engineering structure incorporated confidence indicators alongside correctness data, the system generated differentiated formative feedback based on confidence-performance alignment patterns. For example, correct responses associated with low confidence triggered reinforcement-oriented feedback designed to encourage learner self-efficacy and confidence calibration. In contrast, incorrect responses associated with high confidence generated feedback emphasizing conceptual reflection, misconception review, and reconsideration of reasoning strategies.

This integration of metacognitive indicators into AI feedback generation contributed to more context-sensitive formative responses compared with correctness-only feedback systems. Participant comments collected during the pilot suggested that confidence-aware feedback was perceived as more personalized and reflective because it acknowledged not only whether responses were correct, but also how certain learners felt about their understanding. These findings align with prior research emphasizing the importance of formative feedback processes that support both cognitive and metacognitive development [13,15,35].

The instructor-facing analytics dashboards also benefited from the inclusion of confidence-informed indicators. Rather than presenting only aggregate correctness statistics, the dashboards visualized distributions of confidence-performance alignment across learner profiles. These visualizations enabled instructors to identify:

• learners potentially experiencing hidden uncertainty;
• learners demonstrating possible misconceptions;
• topics associated with widespread confidence miscalibration;
• areas requiring additional instructional clarification.

From an instructional analytics perspective, this additional interpretive layer may support more targeted pedagogical decision-making than traditional LMS reporting systems [8,27].

The pilot implementation further demonstrated that the three-variable model could be operationalized within standard LMS workflows without requiring advanced psychometric infrastructure or complex adaptive testing systems. The simplified confidence structure and lightweight computational model enabled real-time processing within the Google Apps Script environment while remaining understandable for instructors without specialized training in psychometrics or learning analytics. This practical accessibility represents an important design consideration because many institutional assessment environments lack the technical infrastructure required for large-scale multidimensional assessment systems [41,42].

At the same time, the findings should be interpreted cautiously due to the exploratory nature of the study and the use of simulated learner profiles. The observed patterns do not constitute psychometric validation of the framework, nor do they establish causal relationships between confidence calibration and learning outcomes. Rather, the results provide preliminary evidence suggesting that confidence-informed assessment may support richer formative interpretation and more nuanced AI-assisted feedback processes within LMS environments [43].

The pilot additionally revealed several operational considerations associated with confidence reporting. Some participants initially required clarification regarding the distinction between confidence and correctness, particularly when confidence ratings did not directly affect grades. However, after orientation and interaction with the platform, participants generally reported that the confidence-selection process encouraged greater reflection regarding their own certainty and understanding. These observations are consistent with metacognitive research suggesting that self-assessment activities may promote learner reflection and awareness of knowledge limitations [7,13,35].

Overall, the findings associated with RQ2 suggest that the three-variable framework provided meaningful exploratory insights into learner confidence calibration, metacognitive awareness, and formative assessment interpretation. By integrating correctness, confidence, and AI-supported feedback within a unified LMS workflow, the CONF.i framework demonstrated the potential value of confidence-informed assessment approaches for extending conventional digital learning analytics beyond correctness-based evaluation alone [8,27,35].

Figure 3 shows the student view of the assessment interface, including the question presentation and the three confidence options (Low, Average, High). The image illustrates how students interact with the assessment framework and confidence mechanism while answering questions.

More revealing were individual student profiles. Four hypothetical archetypes were modeled:

Archetype A: Aligned Mastery (8 of 23 simulated profiles, 34.8%)

High Grade (>80%) with aligned Confidence (±15%). Example: One student scored 100% correct with 65% confidence (Performance=82.5%). AI feedback: “Excellent work! Your confidence was appropriately high for correct answers. Consider whether you might increase confidence in basic facts you’ve mastered.”

Archetype B: Underconfident Competence (5 of 23 simulated profiles, 21.7%)

High Grade (>80%) with low Confidence (<50%). Example: Student scoring 100% correct with 33% average confidence (Performance=66.5%). AI feedback: “You answered everything correctly but seemed unsure. Trust your knowledge more, you clearly understand these concepts!”

Archetype C: Overconfident Struggle (6 of 23 simulated profiles, 26.1%)

Low Grade (<60%) with high Confidence (>80%). Example: Student scoring 33% correct with 87% confidence (Performance=60%). AI feedback: “Some answers were incorrect despite high confidence. This suggests reviewing foundational concepts carefully, verification is especially important when you feel certain.”

Archetype D: Aligned Struggle (4 of 23 simulated profiles, 17.4%)

Low Grade (<60%) with appropriately low Confidence (<50%). Example: Student scoring 50% correct with 42% confidence (Performance=46%). AI feedback: “You’re aware of areas needing improvement, which is an important first step. Let’s focus on those specific concepts.”

These archetypes illustrate the diagnostic potential of confidence measurement. Traditional grading would identify only high/low performers, missing the metacognitive insights that inform targeted intervention.

It is important to emphasize that these categories describe the performance of participants in this pilot study; larger-scale research is needed to determine whether these patterns generalize across populations and contexts.

Figure 4 shows an individual simulated learner profile results, including the Assign Results display with question-by-question breakdown of answers, correctness, confidence levels, and the calculated average Grade, Confidence, and Performance metrics.

The collaborators noted that these patterns align with theoretical predictions from the foundational research, validating the three-variable model’s diagnostic utility. The identification of overconfident struggle patterns (Pattern C) was particularly significant, as this group represents students most resistant to traditional remediation, they believe they understand material they have not mastered.

4.3. Simulated Learner Profile AI Feedback

The pilot implementation demonstrated that the CONF.i framework was capable of generating differentiated AI-assisted formative feedback based on simulated learner profiles that combined correctness and confidence indicators. The results suggest that integrating confidence-informed assessment variables into the prompt-engineering workflow enabled the Gemini AI services to produce feedback that extended beyond conventional correctness-based response generation. Rather than simply identifying whether answers were correct or incorrect, the system generated responses that incorporated learner confidence, perceived certainty, and confidence-performance alignment patterns into the feedback process [13,15,35].

The AI-generated feedback process employed the structured prompting framework described in Section 3.4. Assessment responses, confidence selections, and composite performance indicators were dynamically transmitted to the Gemini API services through the Google Apps Script backend. The AI models then generated individualized formative feedback designed to:

• reinforce conceptual understanding;
• identify misconceptions;
• encourage metacognitive reflection;
• provide actionable learning recommendations.

This process enabled the framework to generate differentiated feedback across multiple simulated learner scenarios while maintaining continuity within the Canvas LMS workflow [21,22].

One of the most significant findings observed during pilot implementation was the ability of the AI system to adapt its feedback tone and instructional emphasis according to confidence-performance alignment patterns. Learner profiles demonstrating high correctness combined with high confidence typically received reinforcement-oriented feedback emphasizing conceptual mastery and continued engagement. In these cases, the AI-generated responses acknowledged successful performance while encouraging learners to maintain effective study strategies and continue advancing toward more complex material.

In contrast, learner profiles are characterized by high correctness, but low confidence generates feedback focused on confidence reinforcement and self-efficacy support. Rather than treating these learners identically to highly confident high-performing learners, the system produced responses encouraging learners to trust their reasoning processes and recognize their demonstrated understanding. This distinction reflects broader educational research suggesting that under confidence may influence learner motivation, participation, and willingness to engage with challenging material despite adequate academic performance [7,13,35].

Profiles demonstrating low correctness combined with high confidence produced substantially different feedback patterns. In these scenarios, the AI system emphasized conceptual clarification, reflective questioning, and reconsideration of reasoning strategies. The generated responses frequently encouraged learners to review specific concepts, revisit learning materials, and examine possible misconceptions underlying their confidence but incorrect responses. These findings are pedagogically significant because overconfident incorrect responses may indicate hidden misconceptions that are not immediately visible through correctness-based grading alone [9,32].

For learner profiles demonstrating both low correctness and low confidence, the generated feedback adopted a more supportive and scaffolded instructional tone. These responses are commonly included:

• simplified conceptual explanations;
• encouragement to revisit foundational concepts;
• recommendations for supplementary resources;
• reassurance regarding the learning process.

This adaptive feedback behavior illustrates how confidence-informed prompting may support differentiated formative feedback strategies within AI-enhanced educational systems [15,17].

Participant reflections collected during the pilot suggested that the confidence-aware feedback was perceived as more personalized and contextually relevant than conventional automated grading comments. Several participants noted that the system appeared to “understand” not only whether they answered correctly, but also how certain they felt regarding their responses. These observations align with prior studies emphasizing that learners often value feedback that acknowledges uncertainty, confidence, and reflective learning processes in addition to correctness outcomes [13,35].

The AI-generated feedback also demonstrated the operational feasibility of combining metacognitive indicators with generative AI systems in near real-time educational workflows. Average feedback generation remained sufficiently low to preserve interaction continuity within the LMS environment, allowing participants to review personalized feedback immediately after assessment submission. This immediacy is particularly relevant within formative assessment contexts, where timely feedback is considered, an important factor supporting learner reflection and self-regulated learning [13,15].

From an instructional perspective, the system additionally generated aggregated analytics summaries for instructors based on the simulated learner profiles. These summaries are synthesized:

• confidence distributions;
• performance trends;
• recurring misconceptions;
• potential intervention areas.

The instructor-facing outputs provided more nuanced interpretations than conventional grade summaries because they incorporated confidence-performance alignment information alongside correctness metrics. For example, instructors could identify groups of learners demonstrating widespread uncertainty despite acceptable correctness outcomes or detect patterns of overconfidence associated with specific concepts or assessment items. These findings support broader learning analytics research advocating multidimensional interpretation of learner behavior rather than reliance on isolated performance scores [8,27].

The prompt-engineering framework also contributed to improving the pedagogical coherence of AI-generated responses. Early prototype iterations occasionally produced generic or repetitive comments that lacked sufficient contextualization. However, the inclusion of explicit prompt constraints regarding:

• confidence interpretation;
• metacognitive reflection;
• constructive tone;
• instructional recommendations

improved the specificity and perceived educational usefulness of the generated feedback. These observations are consistent with emerging research demonstrating that pedagogically informed prompt engineering significantly influences the quality and instructional relevance of generative AI outputs in educational environments [11,15,36].

Despite these positive findings, several limitations were identified during the pilot implementation. Some generated responses occasionally demonstrated:

• repetitive phrasing;
• overly broad resource recommendations;
• excessive encouragement language;
• insufficient conceptual specificity for complex misconceptions.

Additionally, AI response quality varied depending on the clarity of learner response patterns and the contextual detail included within the generated prompts. These findings reinforce broader concerns regarding the consistency, reliability, and pedagogical precision of large language model outputs in educational settings [14,17,38].

It is also important to emphasize that the learner profiles used during this study were simulated for exploratory testing purposes. Consequently, the feedback responses generated should not be interpreted as validated evidence of instructional effectiveness or authentic learner impact. Instead, the results demonstrate the technical and pedagogical feasibility of integrating confidence-informed assessment variables into AI-assisted formative feedback workflows within LMS environments [43].

The pilot nevertheless suggests that combining confidence-informed analytics with structured generative AI prompting may support richer and more adaptive formative feedback processes than correctness-only assessment systems. By incorporating metacognitive indicators into feedback generation, the CONF.i framework demonstrates a potential pathway for extending LMS-based assessment systems toward more reflective, personalized, and context-aware educational support mechanisms [7,13,35].

Figure 5 shows the Gemini AI-generated feedback presented after assessment completion. The feedback includes personalized comments on each response, suggestions for improvement, and recommended resources including bibliographic references and video links.

Interface experience received 82% positive ratings, appreciating the familiar Canvas context. One commented: “It felt like part of Canvas, not a separate tool. I didn’t have to learn anything new.”

AI feedback helpfulness received 78% positive ratings. Observation valued personalization: “The feedback addressed the specific wrong answers, not generic comments.” However, 22% observed neutrality or dissatisfaction, primarily citing response delays.

Resource recommendations received the lowest ratings (74% positive). Some observed points: “The AI recommended entire books when I missed one multiplication question. More specific resources would help.” Another point: “Links to Khan Academy were useful but some resources seemed aimed at younger students.”

Table 2. Preliminary Usability and Platform Experience Ratings.

Evaluation Dimension	Metric Evaluated	Positive Rating (%)	Primary User Feedback Tendency
Interface Experience	Integration within the familiar Canvas context	82%	Felt native to the LMS; low adoption friction
AI Feedback Helpfulness	Granular personalization of responses	78%	Addressed specific wrong answers over generic comments
Resource Recommendations	Relevance of suggested reference materials	74%	Lowest rating; occasionally suggested broad textbooks for granular errors

Table 2. Preliminary Usability and Platform Experience Ratings.

Evaluation Dimension	Metric Evaluated	Positive Rating (%)	Primary User Feedback Tendency
Interface Experience	Integration within the familiar Canvas context	82%	Felt native to the LMS; low adoption friction
AI Feedback Helpfulness	Granular personalization of responses	78%	Addressed specific wrong answers over generic comments
Resource Recommendations	Relevance of suggested reference materials	74%	Lowest rating; occasionally suggested broad textbooks for granular errors

Qualitative Thematic Analysis:

Observations yielding four themes:

Theme 1: Metacognitive Awareness (n=12)

Reported that confidence rating prompted reflection. “Having to say how confident I was made me think about what I really know versus guess. I realized I’m often overconfident.” Another observation: “The confidence part is interesting, too sure about incorrect answers. That’s something to work on.”

Theme 2: Feedback Specificity (n=10)

Some observed points of attention. “It felt like having a tutor look at my specific work, not just a score.” However, some desired more detail: “The AI said, ‘review multiplication’ but didn’t explain why 9×9 equals 81 specifically.”

Theme 3: Technical Performance (n=8)

Latency concerns emerged. “It took maybe 30 seconds to get feedback. Fine for homework but stressful for timed tests.” Another point: “Was it necessary to refresh once when it seemed stuck.” These comments align with observed Google Apps Script limitations.

Theme 4: Future Applications (n=7)

Some suggestions for expanded uses. “This would be great for practicing quizzes before exams, see what you don’t know.” Another: “Could it generate similar practice questions automatically? Like if I miss 9×9, give me more multiplication practice.”

Instructor Observations

The instructor noted three implementation insights. First, explaining the confidence rating purpose was essential, the team initially wondered if confidence affected grades. Second, the instructor dashboard enabled initiative-taking outreach: “I identified three overconfident simulated profile struggling and scheduled brief meetings before the next exam.” Third, grading efficiency improved: “I didn’t grade anything manually. The system managed scoring and feedback, I just reviewed and posted grades.”

Figure 6 shows the instructor’s comprehensive view of simulated profile results, including the IRT02 interface with the option to view individual student reports and send grades to Canvas. The dashboard aggregates data from multiple student submissions.

5. Discussion

5.1. Principal Findings

This study explored the feasibility and pedagogical potential of integrating confidence-informed assessment and generative AI feedback within an existing Canvas LMS environment through the CONF.i framework. The principal findings suggest that it is technically and operationally feasible to combine LMS interoperability, cloud-based scripting services, confidence-aware assessment mechanisms, and AI-generated formative feedback within a unified educational workflow using institutionally available infrastructure [19,21,22]. More importantly, the results indicate that incorporating learner confidence into assessment processes may provide additional metacognitive insights that are not captured through conventional correctness-based grading systems alone [7,9,35].

From a technical perspective, the study demonstrated successful interoperability between Canvas LMS, Google Apps Script services, Google Workspace authentication, and Gemini AI APIs. The framework maintained stable operational functionality during pilot implementation, including assessment deployment, confidence data collection, AI-generated feedback delivery, dashboard visualization, and automatic grade synchronization with Canvas. These findings are significant because they suggest that institutions already employing LMS ecosystems and cloud-based educational infrastructure may be able to implement similar AI-enhanced formative assessment systems without requiring major institutional reconfiguration or proprietary adaptive learning platforms [41,42].

The pilot implementation also demonstrated the practical feasibility of embedding confidence-informed assessment workflows directly into standard LMS interactions. The three-variable model enabled the system to dynamically combine correctness, confidence, and composite performance indicators during assessment processing. This integration allowed the framework to identify distinct confidence-performance alignment patterns, including aligned mastery, underconfident competence, overconfident struggle, and aligned struggle. These patterns revealed differences in learner behavior that would likely remain invisible in conventional LMS grade reporting environments [9,32,35].

One of the most important findings emerging from the study concerns the pedagogical value of confidence-informed interpretation. Learners demonstrating similar correctness outcomes often exhibited substantially different confidence profiles, suggesting that correctness alone may not provide sufficient insight into learner certainty, conceptual stability, or self-regulated learning behavior. For example, highly accurate but underconfident learners may require reinforcement and confidence-building support, whereas highly confident but incorrect learners may require conceptual clarification and metacognitive recalibration. These distinctions align with broader research emphasizing the importance of confidence calibration and reflective learning processes within formative assessment environments [7,13,35].

The integration of confidence indicators into the AI feedback generation process also produced meaningful differences in the structure and tone of generated responses. Rather than delivering generic correctness-based comments, the Gemini AI system generated differentiated feedback that incorporated learner certainty, confidence-performance alignment, and metacognitive interpretation into the feedback workflow. Correct but uncertain learner profiles received reinforcement-oriented feedback, while incorrect yet highly confident profiles generated responses emphasizing conceptual reflection and misconception review. This finding suggests that confidence-informed prompt engineering may support more context-sensitive and pedagogically nuanced AI-assisted feedback processes [15,17,36].

Participant reflections collected during the pilot additionally indicated that confidence-aware feedback was often perceived as more personalized and reflective than traditional automated assessment responses. Participants reported that the system appeared to recognize not only whether responses were correct, but also how certain learners felt regarding their understanding. These observations support prior educational research suggesting that effective formative feedback should address both cognitive performance and metacognitive awareness [13,15,35].

From an instructional analytics perspective, the framework also demonstrated the potential value of combining confidence-informed metrics with AI-generated class summaries. Instructor dashboards provided multidimensional visualizations of learner performance, confidence distributions, and possible misconception patterns. These analytics extended beyond traditional LMS reporting by enabling instructors to identify:

• learners potentially experiencing hidden uncertainty;
• concepts associated with widespread overconfidence;
• confidence misalignment trends across assessment topics.

Such multidimensional analytics may support more targeted instructional intervention and richer interpretation of learner behavior within digital learning environments [8,27].

The findings additionally highlight the importance of prompt engineering in educational AI systems. Early prototype iterations occasionally generated overly generic or repetitive responses. However, iterative refinement of the instructional prompts improved contextual relevance, metacognitive interpretation, and pedagogical coherence of the AI-generated feedback. These observations reinforce emerging research demonstrating that the educational effectiveness of generative AI systems depends not only on the underlying language model, but also on the pedagogical quality of prompt design and contextual framing [11,15,36].

At the same time, the study revealed several operational and conceptual limitations that remain important for interpreting the findings cautiously. The pilot relied on simulated learner profiles rather than authentic student assessment data, meaning that the observed patterns represent exploratory demonstrations rather than validated evidence of educational effectiveness. Consequently, the framework should currently be understood as a prototype-oriented and exploratory formative assessment system rather than a validated psychometric or adaptive learning model [43].

The simplified three-variable formulation also prioritizes interpretability and operational feasibility over psychometric sophistication. While the framework was conceptually inspired by multidimensional and confidence-informed assessment research [5,6,34], it does not implement formal Item Response Theory estimation, latent trait modeling, or adaptive testing methodologies. Instead, its primary contribution lies in demonstrating how metacognitive indicators can be integrated into AI-supported formative assessment workflows within existing LMS infrastructures.

Another important finding concerns institutional feasibility. The framework was intentionally designed around technologies already available within many higher education environments, including Canvas LMS, Google Workspace services, and cloud-based APIs. This design strategy reduced technical barriers and enabled rapid prototyping without requiring dedicated infrastructure or large-scale software development resources. These findings support broader educational technology perspectives emphasizing augmentation of existing institutional ecosystems rather than replacement-oriented innovation approaches [15,17,41].

Overall, the principal findings suggest that confidence-informed AI-enhanced assessment systems may provide a promising direction for extending conventional LMS environments toward more reflective, personalized, and context-aware formative learning support. By combining metacognitive indicators, AI-generated feedback, and interoperable LMS workflows, the CONF.i framework demonstrates the potential value of integrating cognitive and metacognitive dimensions into future digital learning analytics and formative assessment systems [7,13,35].

5.2. Theoretical Contributions

This study contributes theoretically to the emerging intersection of learning analytics, metacognitive assessment, and generative artificial intelligence in higher education by proposing and operationalizing a confidence-informed formative assessment framework integrated within a standard LMS environment. Although exploratory in scope, the CONF.i framework extends existing educational technology research by demonstrating how learner confidence indicators can be incorporated into AI-supported assessment workflows to support richer interpretation of learner behavior beyond correctness-based evaluation alone [7,9,35].

One of the principal theoretical contributions of this study lies in its integration of cognitive and metacognitive dimensions within a unified digital assessment workflow. Traditional LMS assessment systems typically emphasize observable performance outcomes such as grades, correctness percentages, or completion rates. While these metrics provide useful information regarding academic achievement, they often fail to capture learner certainty, perceived understanding, and self-assessment behaviors that are closely associated with self-regulated learning and metacognitive development [13,35]. By incorporating confidence reporting directly into assessment interaction, the CONF.i framework contributes to broader theoretical discussions concerning multidimensional interpretation of learner performance in digital learning environments [8,27].

The proposed three-variable model additionally contributes to the conceptualization of confidence-informed assessment as a lightweight and operationally accessible alternative to more computationally intensive multidimensional assessment systems. Although the framework was conceptually informed by multidimensional and confidence-aware assessment research [5,6,34], its contribution does not reside in formal psychometric modeling or latent trait estimation. Instead, the study advances a practical interpretation-oriented framework that prioritizes usability, interpretability, and LMS compatibility while preserving theoretically meaningful distinctions between correctness and learner certainty.

In this sense, the framework contributes to emerging discussions regarding “pedagogically informed simplification” within educational technology design. Rather than pursuing highly complex adaptive modeling systems requiring specialized psychometric infrastructure, the CONF.i approach demonstrates how simplified metacognitive indicators may still produce meaningful pedagogical insights when embedded within authentic educational workflows [41,42]. The study therefore contributes to theoretical conversations concerning the balance between psychometric sophistication and practical institutional implement ability in educational analytics systems.

A second important theoretical contribution concerns the integration of confidence-informed indicators into generative AI feedback processes. Existing AI-assisted educational feedback systems frequently rely primarily on correctness-based assessment inputs or conversational interaction patterns [11,15]. In contrast, the CONF.i framework demonstrates how metacognitive variables such as learner confidence and confidence-performance alignment can function as contextual inputs shaping the structure, tone, and pedagogical orientation of AI-generated formative feedback.

The findings suggest that incorporating confidence indicators into prompt-engineering workflows enables more differentiated and context-sensitive feedback generation. For example, the framework distinguished between:

• correct but uncertain learners requiring reinforcement and confidence-building support;
• incorrect yet highly confident learners potentially requiring conceptual recalibration and reflective intervention.

This distinction contributes theoretically to emerging perspectives in AI-enhanced formative assessment that emphasize the importance of integrating cognitive, affective, and metacognitive learner dimensions into educational AI systems [13,15,35].

The study also contributes to broader theoretical discussions surrounding self-regulated learning and metacognitive awareness within digital environments. Confidence reporting functioned not only as an analytical variable, but also as a reflective activity encouraging learners to evaluate their own certainty and understanding during assessment interaction. Participant observations suggested that the confidence-selection process itself promoted greater awareness of uncertainty, conceptual gaps, and reasoning confidence. These findings align with prior metacognitive research emphasizing the role of self-assessment and reflective monitoring in supporting learner regulation and conceptual awareness [7,13,35].

From a learning analytics perspective, the CONF.i framework additionally contributes to the theoretical expansion of LMS-based analytics beyond performance-only interpretation. Conventional LMS dashboards frequently focus on metrics such as:

• grades,
• activity frequency,
• attendance,
• assignment completion.

The confidence-informed analytics introduced in this study demonstrate how learner certainty and calibration patterns may provide an additional interpretive layer for understanding engagement, misconceptions, and instructional intervention needs [8,27]. The resulting framework therefore contributes to evolving theoretical perspectives advocating multidimensional and learner-centered educational analytics approaches.

Another theoretical contribution concerns the conceptual framing of AI as instructional augmentation rather than instructional replacement. The CONF.i framework was intentionally designed to position generative AI feedback as a supportive instructional mechanism operating under instructor oversight rather than as an autonomous grading or teaching system [15,17]. This augmentation-oriented perspective contributes to ongoing theoretical and ethical discussions regarding responsible AI integration in higher education, particularly concerning:

• transparency,
• instructor agency,
• pedagogical accountability,
• human-AI collaboration.

The study suggests that AI systems may be more educationally appropriate and institutionally acceptable when designed to complement existing instructional practices rather than displace educator expertise [14,17,38].

The framework also contributes theoretically to discussions regarding interoperability and sustainable educational innovation. By demonstrating how AI-enhanced formative assessment can be integrated through LTI standards and institutionally available cloud services, the study supports theoretical perspectives emphasizing incremental augmentation of existing educational ecosystems instead of disruptive infrastructure replacement [19,20,41]. This approach is particularly relevant within higher education contexts where scalability, cost, and technical compatibility strongly influence adoption of educational technologies.

Finally, the study contributes to the emerging body of scholarship exploring the pedagogical implications of generative AI in formative assessment contexts. Much of the current literature on generative AI in education remains focused on:

• content generation,
• tutoring systems,
• academic integrity concerns,
• conversational interaction.

The CONF.i framework extends this discussion by exploring how generative AI can be combined with metacognitive assessment indicators to support reflective, personalized, and context-aware formative feedback workflows within institutional LMS environments [11,15,36].

Although exploratory and limited by the use of simulated learner profiles, the study provides a conceptual foundation for future research examining how confidence-informed AI systems may influence:

• learner reflection,
• metacognitive calibration,
• formative assessment practices,
• instructional analytics,
• personalized educational support.

In this sense, the primary theoretical contribution of the study lies not in validating a psychometric assessment model, but rather in proposing a feasible conceptual pathway for integrating metacognitive awareness, AI-generated feedback, and interoperable LMS infrastructures within future digital learning ecosystems [7,13,35].

5.2. Practical Implications

The findings of this study suggest several practical implications for higher education institutions, instructional designers, educational technology developers, and instructors seeking to integrate formative AI-supported assessment practices within existing LMS ecosystems. Although exploratory in nature, the CONF.i framework demonstrates that confidence-informed assessment and generative AI feedback can be operationalized using institutionally available technologies without requiring large-scale infrastructure replacement or highly specialized technical resources [19,21,22].

One of the most significant practical implications concerns institutional feasibility and interoperability. The framework successfully integrated Canvas LMS, Google Workspace services, Google Apps Script, and Gemini AI APIs through Learning Tools Interoperability (LTI) standards. This interoperability-oriented design demonstrates that universities already employing cloud-based educational ecosystems may be able to implement similar AI-enhanced formative assessment solutions using existing infrastructure and authentication systems [41,42]. From an administrative perspective, this approach reduces deployment barriers, minimizes implementation costs, and preserves compatibility with institutional workflows and academic records systems.

The use of lightweight cloud-based technologies also has important implications for scalability and accessibility. Unlike many advanced adaptive learning systems that require dedicated servers, proprietary platforms, or extensive software development resources, the CONF.i framework was implemented using broadly available educational technologies and scripting environments. This design strategy may be particularly valuable for institutions with limited technical infrastructure or constrained educational technology budgets [14,18]. The findings therefore suggest that meaningful AI-enhanced assessment innovation does not necessarily require large-scale institutional investment if interoperability and modular design principles are prioritized.

From an instructional perspective, the study highlights the practical value of incorporating confidence reporting into routine assessment workflows. Traditional LMS assessment systems generally provide instructors with correctness-based performance data while offering limited visibility into learner certainty, perceived understanding, or confidence calibration [8,27]. The confidence-informed indicators generated by the CONF.i framework enabled instructors to identify patterns such as:

• hidden learner uncertainty;
• overconfidence associated with misconceptions;
• confidence misalignment across assessment topics;
• potential intervention needs.

These multidimensional analytics may support more targeted instructional decision-making and allow instructors to distinguish between learners requiring conceptual remediation and those primarily needing confidence reinforcement or metacognitive support [7,13,35].

The findings additionally suggest that confidence-aware AI-generated feedback may improve the perceived personalization and contextual relevance of formative assessment processes. Because the Gemini AI feedback system incorporated both correctness and confidence indicators, the responses generated not only to whether learners answered correctly, but also to how certain they felt regarding their understanding. This differentiation allowed the system to provide:

• reinforcement-oriented feedback for underconfident learners;
• reflective prompts for overconfident incorrect responses;
• scaffolded guidance for learners demonstrating conceptual difficulty.

Such adaptive formative feedback may support learner reflection and self-awareness more effectively than conventional automated grading comments focused exclusively on correctness outcomes [13,15,35].

Another practical implication concerns the role of prompt engineering in educational AI systems. The pilot implementation demonstrated that the pedagogical quality of AI-generated feedback depended heavily on the structure and specificity of instructional prompts. Early prototype iterations frequently produced overly generic responses or repetitive recommendations. However, iterative refinement of the prompt templates improved:

• contextual relevance;
• metacognitive interpretation;
• instructional coherence;
• perceived educational usefulness.

These findings reinforce the practical importance of pedagogically informed prompt engineering when implementing generative AI systems in educational settings [11,15,36]. Institutions adopting AI-supported educational tools may therefore need to consider prompt design as an instructional design activity rather than solely a technical configuration task.

The study also carries important implications regarding the role of instructors in AI-enhanced educational environments. The CONF.i framework intentionally positioned generative AI as a supportive instructional mechanism operating under instructor oversight rather than as an autonomous grading or tutoring system [15,17]. Instructor observations during the pilot indicated that maintaining compatibility with familiar Canvas workflows reduced adoption friction and increased perceived usability. At the same time, instructors retained authority over:

• grading decisions;
• interpretation of analytics;
• instructional intervention strategies;
• feedback contextualization.

This augmentation-oriented approach may be particularly important for maintaining instructor trust and pedagogical accountability during institutional AI adoption processes [14,17,38].

The framework additionally demonstrates practical opportunities for extending LMS-based learning analytics beyond performance-only reporting systems. Conventional LMS dashboards frequently emphasize grades, attendance, or activity metrics without integrating metacognitive indicators or reflective learner dimensions [8,27]. The confidence-informed dashboards implemented in this study illustrate how institutions might enrich learning analytics environments through relatively lightweight additions to existing assessment workflows. Such enhancements may support earlier identification of misconceptions, confidence miscalibration, or learner disengagement patterns.

From a learner perspective, the findings suggest that confidence-reporting activities themselves may encourage reflective learning behaviors. Several participant observations indicated that the requirement to report confidence prompted learners to consider their certainty, reasoning processes, and conceptual understanding more explicitly during assessment interaction. This finding aligns with prior research suggesting that metacognitive self-assessment activities can support learner awareness and self-regulated learning processes [7,13,35]. Consequently, confidence-informed assessment may serve both analytical and pedagogical functions within digital learning environments.

The study also highlights practical limitations that institutions should consider before implementing similar frameworks at larger scales. Although the prototype functioned successfully during pilot testing, several technical constraints were identified, including:

• API response variability;
• spreadsheet-based database latency;
• occasional asynchronous processing delays;
• dependence on external AI service availability.

These limitations suggest that larger-scale institutional deployment would likely require more robust backend architectures and database infrastructure than those employed during the exploration prototype phase [18,41].

In addition, the findings underscore the importance of responsible and transparent AI integration practices within educational environments. Because generative AI systems may occasionally produce inaccurate, incomplete, or overly generic responses, the framework emphasized formative support rather than high-stakes automated decision-making [14,17,38]. Institutions considering implementation of similar systems should therefore ensure:

• instructor oversight;
• transparent AI usage policies;
• clear learner communication;
• mechanisms for feedback review and correction.

Finally, the study suggests that confidence-informed AI-enhanced assessment may represent a practical pathway for extending existing LMS ecosystems toward more reflective, personalized, and learner-centered formative assessment practices. Rather than replacing institutional learning infrastructures, the CONF.i framework demonstrates how interoperable educational technologies, lightweight analytics models, and generative AI services can be combined incrementally to support richer instructional interpretation and metacognitive learning support within authentic higher education contexts [41,42].

5.3. Limitations

Several limitations should be considered when interpreting the findings of this exploratory study. Although the CONF.i framework demonstrated the technical feasibility and pedagogical potential of integrating confidence-informed assessment and generative AI feedback within a Canvas LMS environment, the study was intentionally designed as an early-stage prototype evaluation rather than as a large-scale validation of educational effectiveness. Consequently, the findings should be interpreted cautiously and viewed primarily as preliminary evidence intended to support future refinement, scalability analysis, and empirical investigation [43].

One of the most significant limitations concerns the use of simulated learner profiles during the pilot implementation. Although real participants interacted with the platform interface and provided usability feedback, the assessment response patterns and confidence configurations employed during testing were intentionally simulated by the research team. This design approach enabled controlled exploration of multiple confidence-performance alignment scenarios, including overconfidence, under confidence, aligned mastery, and aligned struggle. However, because these profiles were not derived from authentic classroom assessment data, the findings cannot be interpreted as evidence of actual learner behavior, metacognitive calibration, or instructional impact [35,43].

Relatedly, the study did not evaluate long-term learning outcomes, conceptual retention, self-regulated learning development, or measurable improvements in academic performance. The pilot primarily emphasized technical integration, usability exploration, and formative workflow feasibility. As a result, the framework’s potential influence on academic achievement, learner motivation, metacognitive growth, assessment performance, and instructional effectiveness remains uncertain. Future longitudinal studies involving authentic classroom implementations and real learner populations will therefore be necessary to determine the educational impact of confidence-informed AI-supported assessment systems [7,13,35].

Another important limitation concerns the exploratory nature of the proposed three-variable assessment model itself. Although conceptually informed by prior multidimensional and confidence-aware assessment research [5,6,34], the framework does not implement formal psychometric methodologies such as Item Response Theory (IRT), latent trait estimation, adaptive testing algorithms, or probabilistic confidence calibration models. Instead, the model prioritizes interpretability, operational simplicity, and compatibility with standard LMS environments. While this lightweight approach supports practical implementation and institutional accessibility, it also limits the analytical precision and theoretical sophistication of the resulting learner analytics. Consequently, the framework should not currently be interpreted as a validated psychometric assessment model, but rather as an exploratory formative analytics approach designed to support reflective instructional interpretation [43].

The limited sample size and pilot-scale implementation additionally constrain the generalizability of the findings. The study involved a relatively small group of participants interacting with a prototype system within a controlled institutional context. Institutional variables such as LMS configuration, technical infrastructure, instructor familiarity with digital technologies, AI literacy, and pedagogical culture may substantially influence implementation outcomes across other educational settings [19,20]. Therefore, the findings cannot be generalized broadly across institutions, academic disciplines, or learner populations without further empirical investigation.

Several technical limitations were also identified during pilot implementation. The prototype relied heavily on Google Apps Script and spreadsheet-based data management systems, which occasionally introduced operational constraints including API response latency, asynchronous synchronization delays, spreadsheet write bottlenecks, and intermittent interface refresh requirements. Although these issues did not prevent successful workflow execution during the pilot, they may become increasingly problematic under large-scale deployment conditions involving higher concurrency and more complex data processing demands [14,18]. Future implementations would therefore likely require more scalable backend architectures and dedicated database infrastructures to support broader institutional adoption.

The framework’s dependence on external generative AI services additionally introduces limitations related to service availability, response variability, latency fluctuations, and model updates outside institutional control. Because Gemini AI models operate as externally managed cloud-based systems, response quality and model behavior may change over time as providers update APIs and underlying language models. This dependency creates challenges related to reproducibility, consistency, and long-term instructional reliability [14,17,38]. Furthermore, occasional variability in AI-generated feedback quality was observed during pilot testing, particularly regarding repetitive phrasing, overly generic recommendations, and inconsistent conceptual specificity. These findings reflect broader concerns in educational AI research regarding the pedagogical reliability and interpretive consistency of large language model outputs [11,15,38].

Another limitation concerns the prompt-engineering process itself. Although iterative refinement substantially improved the pedagogical quality of the generated responses, prompt design remained highly dependent on researcher interpretation and instructional assumptions. Minor modifications in prompt structure, contextual framing, or instructional phrasing occasionally produced noticeably different AI-generated outputs. Consequently, the effectiveness and consistency of the feedback system may be sensitive to prompt configuration choices and implementation context [11,36]. Additional research is therefore needed to establish more systematic methodologies for educational prompt engineering, validation, and quality assurance.

The study also did not examine several learner-centered variables that may significantly influence confidence reporting and interpretation of AI-generated feedback. These variables include cultural background, language proficiency, assessment anxiety, prior experience with artificial intelligence systems, and self-efficacy beliefs. Confidence-reporting behaviors may vary considerably across learners and educational contexts, potentially affecting the interpretability and reliability of confidence-informed analytics [7,35]. Similarly, learner trust in AI-generated feedback may be shaped by prior attitudes toward artificial intelligence and digital learning technologies. These dimensions were outside the scope of the present exploratory study but represent important directions for future research.

Ethical and privacy considerations also represent important limitations and ongoing challenges. Although the prototype employed institutionally authenticated LMS workflows and avoided high-stakes automated decision-making, the integration of learner analytics and generative AI systems raises broader concerns regarding transparency, data governance, algorithmic bias, student privacy, and explainability of AI-generated feedback. The present study did not conduct a formal ethical impact assessment or systematic bias evaluation of the generated responses [14,17,38]. Future research should therefore investigate how confidence-informed AI systems can be implemented responsibly within institutional governance frameworks while preserving fairness, accountability, transparency, and learner trust.

Finally, the study focused primarily on formative assessment contexts within higher education mathematics-related activities. The applicability of the framework across other academic disciplines, different educational levels, large-scale online courses, and high-stakes assessment environments remain uncertain. Additional interdisciplinary implementations will therefore be necessary to evaluate the adaptability, scalability, and pedagogical relevance of confidence-informed AI-supported assessment frameworks across broader educational contexts [41,42].

Despite these limitations, the study provides a valuable exploratory contribution by demonstrating the feasibility of combining LMS interoperability, confidence-informed assessment, and generative AI feedback within a unified instructional workflow. The limitations identified throughout the pilot additionally provide important guidance for future research, system refinement, and responsible institutional implementation of AI-enhanced formative assessment systems [43].

5.4. Future Research Directions

The findings of this exploratory study suggest multiple directions for future research concerning confidence-informed assessment, generative AI feedback systems, and LMS-integrated learning analytics. Although the CONF.i framework demonstrated the technical feasibility and pedagogical potential of combining metacognitive indicators with AI-supported formative assessment workflows, substantial additional investigation is required to evaluate educational effectiveness, scalability, psychometric validity, and long-term instructional impact [43].

One of the most immediate priorities for future research involves the implementation of the framework using authentic learner populations and real classroom assessment data. The present study relied primarily on simulated learner profiles to evaluate technical integration and feedback generation workflows. Consequently, future studies should examine how real students interact with confidence-reporting mechanisms and AI-generated formative feedback across authentic instructional contexts. Such research could evaluate learner engagement, confidence calibration behaviors, conceptual understanding, self-regulated learning processes, and perceptions of AI-supported feedback. Longitudinal classroom studies would be particularly valuable for determining whether confidence-informed formative assessment influences learning outcomes, metacognitive development, or academic performance over time [7,13,35].

Future research should also investigate the psychometric properties and analytical validity of the proposed three-variable framework. Although the current model prioritized interpretability and operational simplicity, additional work is needed to examine the reliability of confidence reporting, the consistency of confidence-performance alignment patterns, the stability of composite performance indicators, and the predictive validity of confidence-informed analytics. More advanced statistical and psychometric methodologies could be incorporated to strengthen the analytical rigor of the framework, including Item Response Theory (IRT), Bayesian confidence modeling, latent trait estimation, and adaptive assessment algorithms [5,6,34]. Such investigations may help determine whether confidence-informed indicators can reliably support diagnostic assessment and personalized instructional intervention at larger scales.

Another important direction concerns the refinement and evaluation of AI-generated formative feedback processes. The findings demonstrated that prompt engineering substantially influenced the pedagogical quality and contextual relevance of Gemini-generated responses [11,15,36]. Future studies should therefore explore systematic methodologies for educational prompt optimization, feedback consistency evaluation, hallucination reduction, pedagogical quality assurance, and explainability of AI-generated instructional outputs. Comparative studies examining different prompting strategies, language models, and instructional scaffolding approaches may help identify best practices for AI-supported formative assessment environments.

Additional research is also needed regarding learner perceptions and trust in AI-generated feedback systems. Although participant reflections during the pilot suggested generally positive perceptions of confidence-aware feedback, important questions remain concerning learner trust, perceived credibility, emotional responses to AI feedback, dependence on automated guidance, and willingness to challenge AI-generated explanations. These dimensions may significantly influence how learners engage with AI-enhanced formative assessment systems and whether such systems effectively support reflective learning and metacognitive awareness [13,15,38].

Future investigations should further explore how confidence-informed AI systems influence self-regulated learning behaviors. The requirement for learners to report confidence during assessment interaction may itself function as a metacognitive intervention encouraging reflection, uncertainty monitoring, and self-evaluation [7,35]. Experimental and longitudinal studies could examine whether repeated exposure to confidence-informed assessment promotes improvements in confidence calibration, self-awareness, reflective judgment, strategic learning behaviors, and academic self-efficacy. These investigations may contribute to broader theoretical understanding of how AI-enhanced educational environments support metacognitive development.

Another promising research direction involves expanding the framework beyond mathematics-oriented formative assessment contexts. The present study focused primarily on mathematics-related assessment activities because they provide relatively structured correctness evaluation and clear confidence-performance interpretation. However, future studies should investigate the applicability of confidence-informed AI-supported assessment across humanities, social sciences, language learning, professional education, and interdisciplinary learning environments. Different disciplinary contexts may require alternative confidence structures, feedback strategies, and analytical interpretations [41,42].

Research examining scalability and institutional deployment is also necessary. While pilot implementation demonstrated operational feasibility within a small-scale prototype environment, future large-scale implementations may encounter additional challenges related to concurrent user processing, database scalability, API usage limitations, institutional security requirements, and infrastructure sustainability. Future system development could explore migration from spreadsheet-based architectures toward dedicated cloud databases, microservice architectures, scalable analytics pipelines, and enterprise LMS integration frameworks. Such work would help determine the operational requirements necessary for institutional adoption at scale [18,41].

Future research should additionally investigate ethical and governance considerations associated with confidence-informed AI assessment systems. Important unresolved questions remain regarding transparency of AI-generated feedback, explainability of learner analytics, algorithmic bias, privacy protection, responsible data governance, and institutional accountability. Formal ethical evaluation frameworks and bias audits should therefore accompany future implementations involving authentic learner data [14,17,38]. Particular attention should be given to how confidence-related analytics may influence learner labeling, instructional intervention decisions, or perceptions of academic ability.

Another significant research opportunity concerns adaptive and personalized learning pathways. The current framework generated differentiated formative feedback based on confidence-performance alignment patterns, but future versions could extend this capability toward adaptive resource recommendation, personalized learning sequences, targeted remediation pathways, and dynamic assessment difficulty adjustment. Integrating confidence-informed analytics with adaptive learning systems may enable more responsive educational environments capable of supporting individualized learning trajectories [8,27].

Cross-cultural and multilingual research also represents an important future direction. Confidence expression and self-assessment behaviors may vary substantially across cultural, linguistic, and educational contexts [7,35]. Future international comparative studies could examine how cultural norms influence confidence reporting, how learners interpret AI-generated feedback, how metacognitive behaviors vary across educational systems, and how confidence-informed analytics perform in multilingual environments. Such investigations would be especially valuable for understanding the broader applicability and fairness of confidence-informed AI-supported assessment systems.

Future research should additionally explore instructor-centered perspectives regarding AI-enhanced formative assessment. Although the present study included observational insights from instructors, more extensive investigation is needed concerning instructor trust in AI-generated analytics, perceived workload implications, pedagogical usability, decision-making support, and professional development needs. Understanding instructor experiences will be critical for supporting sustainable institutional implementation and effective human-AI collaboration in educational contexts [15,17].

Finally, future work should examine how confidence-informed assessment systems interact with broader learning analytics ecosystems and institutional data infrastructures. Integrating confidence-aware indicators with attendance analytics, engagement metrics, discussion participation, assignment progression, and behavioral learning traces may support richer multidimensional models of learner development and instructional support [8,27]. Such integration could contribute to the evolution of more holistic and learner-centered educational analytics environments.

Overall, the exploratory findings presented in this study suggest that confidence-informed AI-enhanced formative assessment represents a promising and underexplored area of educational technology research. By integrating metacognitive indicators, generative AI feedback, and interoperable LMS infrastructures, future research may help advance more reflective, personalized, and context-aware digital learning ecosystems capable of supporting both cognitive and metacognitive dimensions of learning [7,13,35].

Figure 7 shows the comprehensive dashboards displaying the three variables (Grade, Confidence, Performance) for each individual student’s profile. The visualizations illustrate how the assessment framework model provides richer diagnostic information than traditional single-score reporting.

6. Conclusion

This study presented the CONF.i framework, an exploratory confidence-informed formative assessment system integrating Canvas LMS interoperability, metacognitive assessment indicators, and generative AI feedback within a unified educational workflow. The framework was designed to investigate how learner confidence, correctness-based assessment, and AI-generated formative feedback could be combined to support richer instructional interpretation and more reflective learning experiences in higher education environments. The findings demonstrate that integrating confidence-informed assessment and generative AI services within existing LMS ecosystems is both technically feasible and pedagogically promising when implemented through interoperable and institutionally compatible technologies [19,21,22].

From a technical perspective, the study demonstrated successful integration between Canvas LMS, Google Apps Script services, Google Workspace infrastructure, and Gemini AI APIs using Learning Tools Interoperability (LTI) standards. The framework supported end-to-end assessment workflows including assignment deployment, confidence reporting, real-time data processing, AI-generated feedback delivery, analytics visualization, and grade synchronization with the institutional LMS environment. These results suggest that AI-enhanced formative assessment systems can be implemented without requiring extensive infrastructure replacement or proprietary adaptive learning platforms, thereby reducing institutional barriers to educational innovation [41,42].

The study also demonstrated the practical feasibility of operationalizing a confidence-informed three-variable assessment model within standard LMS workflows. By combining Grade, Confidence, and Performance indicators, the framework enabled multidimensional interpretation of learner behavior extending beyond traditional correctness-based evaluation. The resulting learner profiles revealed meaningful distinctions between:

• aligned mastery,
• underconfident competence,
• overconfident struggle,
• aligned struggle.

These distinctions suggest that learner confidence may provide important metacognitive information that conventional LMS grading systems frequently fail to capture [7,9,35].

The integration of confidence-informed indicators into the AI feedback generation process additionally contributed to more differentiated and context-sensitive formative responses. Rather than generating generic correctness-based comments, the Gemini AI services produced feedback that adapted according to learner confidence-performance alignment patterns. Correct but uncertain learner profiles received reinforcement-oriented support, whereas incorrect yet highly confident profiles generated responses emphasizing conceptual reflection and misconception review. These findings indicate that confidence-aware prompt engineering may enhance the pedagogical relevance and personalization of AI-generated formative feedback [13,15,36].

Participant observations and usability feedback further suggested that confidence reporting encouraged reflective thinking regarding certainty, understanding, and self-assessment during assessment interaction. The confidence-selection process appeared to promote greater metacognitive awareness by prompting learners to evaluate not only whether they believed answers were correct, but also how certain they felt regarding their reasoning processes. These observations align with prior research emphasizing the importance of metacognitive monitoring and reflective learning within formative assessment environments [7,13,35].

From an instructional analytics perspective, the framework demonstrated how confidence-informed indicators may enrich conventional LMS reporting systems. Instructor dashboards incorporating confidence-performance alignment patterns provided multidimensional views of learner behavior that extended beyond correctness-based summaries. These analytics enabled identification of possible hidden uncertainty, overconfidence patterns, and confidence miscalibration across assessment topics, potentially supporting more targeted instructional intervention and pedagogical decision-making [8,27].

The study additionally contributes to broader discussions concerning the role of generative AI in education. Rather than positioning AI as an autonomous instructional replacement system, the CONF.i framework adopted an augmentation-oriented approach in which AI-generated feedback functioned as a supportive instructional mechanism operating under instructor oversight [15,17]. This design perspective emphasizes the importance of preserving instructor agency, pedagogical accountability, and responsible AI integration within educational environments [14,17,38].

At the same time, several limitations constrain interpretation of the findings. The exploratory nature of the pilot, the use of simulated learner profiles, the limited participant sample, and the simplified assessment model prevent generalization regarding educational effectiveness or psychometric validity [43]. Additionally, technical limitations related to cloud-based scripting environments, API variability, and scalability remain important considerations for future institutional deployment. Consequently, the framework should currently be interpreted as an exploratory prototype demonstrating feasibility and conceptual potential rather than a validated large-scale assessment solution.

Despite these limitations, the study provides a meaningful contribution to emerging research exploring the intersection of metacognitive assessment, learning analytics, and generative AI in higher education. The findings suggest that integrating learner confidence into AI-supported formative assessment workflows may support richer pedagogical interpretation, more personalized feedback generation, and greater learner reflection than conventional correctness-only assessment systems [7,13,35].

Future research should therefore investigate the framework using authentic learner populations, longitudinal classroom implementations, and more sophisticated analytical methodologies. Additional studies examining psychometric validity, learner trust, adaptive feedback systems, scalability, ethical governance, and cross-disciplinary applicability will be necessary to determine the broader educational implications of confidence-informed AI-enhanced assessment systems [14,17,38,43].

Overall, the CONF.i framework demonstrates a promising pathway for extending LMS-based formative assessment toward more reflective, personalized, and context-aware educational ecosystems. By integrating confidence-informed analytics, interoperable LMS infrastructures, and generative AI feedback within a unified workflow, the study highlights the potential for future educational technologies to support not only cognitive performance measurement, but also metacognitive awareness, learner reflection, and adaptive formative learning support [7,13,35].