Conceptual Development of a Gamified Application for Interactive Learning with an Artificial Intelligence Mentor

1. Introduction

When it comes to learning, it is usually understood as the acquisition of knowledge and skills. When we talk about traditional learning, we can say that it involves a scenario where the student is a passive observer, responds to posed questions, while the teacher plays the role of an authoritative knowledge transmitter, and, in principle, the lecture is a monologue by the teacher without interaction with the students except when asking questions. In traditional learning, the teacher imparts all knowledge to the students, using the textbook and additional literature as additional sources of knowledge [1]. However, the use of new technologies for learning is becoming increasingly common in today's society. Various technologies such as computer programs, applications, and online platforms enable learning through computers, tablets, mobile phones, and other digital devices. For example, educational software or applications may use gamification to motivate students. The research thinks educational apps must have rich content, in the premise of scientific nature, appropriate use of multimedia, in order to create the user's perspective, to ensure that the level of educational apps. [2]

The aim of this paper is to identify effective learning approaches, so they can later be applied to modern learning methods, specifically a gamified learning application with an Artificial intelligence mentor that would prepare secondary school graduates for the state graduation exams. In the Republic of Croatia, by successfully passing the state graduation exams (hrv. Državna matura), high school students have the right to enrol in higher education institutions. This study constitutes a segment of the master's thesis titled "Conceptual Development of a Gamified Application for Interactive Learning with an AI Mentor," [3] which was awarded a gold medal at ARCA, the 21st edition of the International Exhibition of Inventions held in Zagreb.

In 2019, a study [4] was conducted to standardise the terminology used to define game elements as a means for designing and implementing gamification strategies in the educational domain. To assess the importance of various gamification elements, an online survey was conducted with experts, including game developers, game designers, researchers, and teachers. According to the results, the majority of respondents considered objectives, levels, and progression as key elements. Interestingly, in educational domains, it was found that a lack of objectives and a sense of progression reduces the motivation and engagement of students [5].

An AI mentor in this learning application would be based on the GPT-4 (Generative Pre-trained Transformer 4) model, a large-scale multimodal model capable of processing input images and text to generate textual outputs. While less capable than humans in many real-world scenarios, GPT-4 demonstrates human-level performance on various professional and academic benchmarks, including ranking in the top 10% in a simulated legal exam [6]. This form of artificial intelligence gained public attention with the release of ChatGPT, based on its predecessor, the GPT-3.5 model, in late 2022. Over a million users signed up to test the model within the first week [7]. Prominent examples of GPT-4 model utilisation in education include Duolingo (Max version) and Khanmigo. Duolingo Max offers two AI-powered options - Explain My Answer and Roleplay. Explain My Answer provides students with the opportunity to learn more about their response in a lesson, offering a simple explanation of why the answer was correct or incorrect and allowing them to request examples or additional clarifications. Roleplay enables students to practise real-world conversation skills. They can discuss plans for an upcoming vacation, order coffee in a Parisian café, go furniture shopping, or ask a friend to go hiking [8]. Khanmigo is an AI-driven instructor for students and an assistant for professors. It assists students, for example, in identifying errors in programming code, and aids professors in lesson planning [9].

Numerous developers crafting mobile education applications rely on their subjective ideas, neglecting to thoroughly address the learners' needs. This oversight often results in products that fail to gain popularity. Prioritising the user's viewpoint involves initiating the design process with the learners' needs in mind, conducting comprehensive studies on their characteristics and key requirements. This approach aids developers in refining their work design ideas. [2] To create a successful and user-centred learning experience in the application, graduating students completed a survey with the main objectives of defining issues in the current approach to learning in high school education and assessing the readiness of graduating students for the state graduation exams.

Objectives of the Study:

• Explore the quality of high school education in Croatia and investigate whether graduating students are "forced" to have their parents pay for exam preparations: The assumption is that most high school graduates plan to attend exam preparation courses. If they attend preparatory courses, parents of graduating students face a significant financial burden, even though education, as a rule, should be free. The minimum wage in Croatia in 2022 was 3750 kuna, or 497.71 euros, meaning that parents would pay approximately the entire amount or more than the minimum wage for preparatory courses for three obligatory exams. Using the exam preparation application would result in savings for parents. Also, since these are exams that determine the continuation of education and the life path of individuals, the issue of discrimination against students from vocational and artistic schools should be carefully considered. [10]
• Understand the ways they prepare for the graduation exam to tailor the application to their learning style: In the Facebook group "Graduates 2022./2023.," where the survey was conducted, scripts from external educational institutions (e.g. Trinom, Algebra, Maxtura) from various subjects are easily found. Trinom's scripts are most shared or requested.
• Identify their main problems and frustrations during exam preparation
• Find the golden mean of the best learning approach for acquiring knowledge and making learning interesting for graduating students and eliminating negative learning approaches in the app: According to Biggs [11], there are three common theories:
  • Learning is primarily a direct result of individual differences between students.
  • Learning is primarily the result of appropriate teaching.
  • Learning is the result of students’ learning-focused activities which are engaged by students as a result both of their own perceptions and inputs and of the total teaching context.
• Learn about their learning experiences when using educational apps

2. Materials and Methods

Secondary school graduates participated in an online survey consisting of four segments, each with its specific purpose. The first section of the survey is focused on gathering general information, the second on assessing their readiness for taking the graduation exams, the third on investigating current learning approaches within secondary education, while the fourth examines the use of digital technologies in education. The survey included a combination of open-ended questions to explore the learning experience in high school education in the third section, as well as closed-ended and scale-type questions in other sections. The Likert scale was used for data analysis (Table 1).

3. Results

In an online survey conducted in early April 2023, a total of 67 high school graduates participated, including 45 respondents from the grammar schools, 21 respondents from the vocational programs, and 1 respondent from the arts program.

3.1. General Information

To begin with, respondents were asked whether they believe that students from grammar school programs are in a better position for passing the graduation exam. The results indicated that graduates from the grammar schools and those from vocational/arts programs agree that students from grammar schools are in a better position than those from vocational and arts programs.

3.2. The Impact of High School Education on the National Exams

To assess the readiness of high school graduates for the graduation exam, a series of questions were posed to the respondents. With the statement that the exam could be passed solely based on high school knowledge, respondents neither agree nor disagree (Likert = 2.6). This indicates that high school education does not provide a sufficient level of knowledge for passing the graduation exams.

At the time of the survey, precisely two months before the graduation exams, 67% of the respondents began preparing for the exam. Among them, 31% study for the exams two to five hours per week, 27% study up to two hours per week, 27% study five to eight hours per week, 11% study more than ten hours per week, and 4% study between eight and ten hours per week.

Graduates neither agree nor disagree that they study on time for the exam (Likert = 2.78).

The readiness of graduates for the graduation exam and the impact of high school education on exam results can be observed from the expectations regarding grades on the exam and the grades they believe they would achieve if they were taking the exam at the time of the survey. For example, three mandatory subjects for the graduation exams were considered: Croatian language, Mathematics, and a Foreign language. The average grade that graduates aim for in the Croatian language exam is 4.06, in Mathematics 3.64, and in a Foreign language 4.06. The average grade they believe they would achieve at the time of the survey is 3.03 in Croatian language, 2.6 in Mathematics, and 3.5 in a Foreign language. It is noticeable that there is the smallest difference in grades in a Foreign language, while there is a significant difference in grades in Croatian language and Mathematics.

To a great extent, respondents will use scripts (such as Trinom) as study materials (Likert = 4.3), followed by older exams (Likert = 4.12), the Internet (Likert = 3.25), video materials/YouTube (Likert = 2.8), and to a lesser extent, high school textbooks (Likert = 2.01) for studying.

To assess the effectiveness of high school education, respondents were asked whether they attend preparatory courses for the graduation exam in educational institutions such as Trinom, Algebra, and others, or if they opt for private tutoring. 55% of respondents attend preparatory courses, 4% plan to attend them, and 40% neither attend nor plan to attend.

Although most respondents will attend preparatory courses, they neither agree nor disagree that these courses are necessary for successfully passing the graduation exam (Likert = 2.6).

Graduates are moderately motivated to study for the exam (Likert = 2.90), and taking the exam is a significant source of stress for them (Likert = 4.05).

3.3. Approaches to Learning in High School Education

The next part of the survey was dedicated to exploring the attitudes of high school graduates towards the learning methods in high school education, aiming to retain effective learning approaches and eliminate ineffective ones in the app. Respondents were required to textually explain how teaching is conducted in subjects they consider themselves to have the highest and lowest levels of knowledge, as well as in their favourite and least favourite subjects. The goal of the questions posed was to find a middle ground between the best learning methods for acquiring knowledge and learning that is interesting to students. They were also asked if they had ever failed a subject and, if so, the reasons they believed led to their failure. Only one respondent failed a grade due to "a lack of time for grade correction."

3.3.1. Best Level of Knowledge

Achieving the best level of knowledge in a subject can be categorised based on several factors.

The first factor for graduates is interest in the subject that does not depend on the teacher and learning approaches. The second factor is the teacher and their approach to students. Graduates also believe that solving tasks is crucial for acquiring knowledge. The last factor is the overall teaching method. From graduates’ responses it can also be observed that for successful knowledge acquisition, repetition, independent/group work, the quantity of material, discussion, and practical work are essential.

3.3.2. Favourite Subject

The favourite subject of high school graduates, as well as the subject in which they have the highest level of knowledge, can be categorised based on factors. Survey results have shown that it is not necessary for the favourite subject of graduates to be the same as the subject in which they have the highest level of knowledge. Four respondents stated that their favourite subject is the same as the subject in which they have the highest level of knowledge, while six respondents mentioned that they do not have a favourite subject.

Although some students are not satisfied with the teacher and their methods, they believe that their favourite subject is due to personal interests. For some students, their favourite subject is credited to the teacher who made it interesting. Some respondents will like a subject because they relate the material to real life and can apply the knowledge gained later on. The reason a subject is the favourite for some students is a combination of personal interests, interesting content, and/or the teacher's approach.

3.3.3. Lowest Level of Knowledge

In subjects where respondents believe they have the lowest level of knowledge, they almost exclusively attribute their insufficient knowledge to the teacher and their teaching approach. Only one graduate stated, "The teacher is excellent, but I'm not motivated for that subject." There are several reasons they believe teachers poorly convey knowledge.

One reason is the lack of motivation of teachers for teaching material and lack of explanation of the material or not covering material in class. Second reason is the lack of communication between teachers and students, the third is the lack of dynamism in teaching and conducting classes in an uninteresting way. The fourth reason is the incompetence of teachers, their lateness, and/or changes in the teaching staff.

3.3.4. Least Favourite Subject

Regarding the least favourite subject personally for the graduates, six of them claim that their the least favourite subject is also the one from which they have the least amount of knowledge, while three graduates either do not have or do not know which is their least favourite subject.

Respondents claim that they dislike a particular subject due to a poor learning approach or its absence, due to a lack of interest in them or the teacher's approach to teaching and students.

3.4. Approaches to Learning in High School Education

The last part of the survey covers the use of digital technology. Graduates agree that digital technology has a positive impact on the learning experience (Likert = 3.73).

Educational apps were used by 70% of graduates, 10% were unsure, and 19% of graduates did not use learning apps.

They rated the Duolingo, language learning app, with a score of 3.13, and they believe it was of little help (Likert = 2.21) in learning foreign languages.

The Photomath app received a rating of 4.46, and they believe it helped them to a great extent (Likert = 3.52) in solving mathematical problems.

They rated the web platform Ispravi.me, a popular Croatian spell-check platform, with a score of 3.38, and they consider it to be moderately helpful (Likert = 2.52) in learning spelling.

4. Discussion

The learning application should provide the same opportunities for high school students in grammar, vocational, and artistic programs to prepare for their graduation exams. Bezinović's work has been confirmed by a survey, showing that students in grammar schools are in a better position than those in other high school programs. The application should be accessible to all high school students so that they can equally prepare for their graduation exams. Although 67% of respondents started preparing for the graduation exams, the grades they would receive at the time of the survey and the grades they aim for differ significantly. High school education, therefore, has not provided knowledge that students will remember; instead, they "cram" for the exams, which is completely opposite to effective learning methods.

The application should provide access to learning materials that will stay in students' memory, motivate them to learn, and reduce the stress they feel about taking the state graduation exams. A significant problem is that even though graduating students do not consider attending preparatory courses necessary for successful exam completion, most of them will attend, and the materials provided by these institutions will be used as study materials for the exams. We can conclude that graduating students do not turn to high school education for learning or reviewing the material but rather to external educational institutions.

Furthermore, graduating students will predominantly use scripts from these institutions for studying, although not all will attend preparatory courses. Such scripts are intended for internal use only, i.e., for participants in the preparatory courses, but students share or sell them, leading to a serious violation of copyright. Therefore, the application must offer an alternative solution to replace the piracy of scripts.

According to the responses of graduating students regarding the teaching methods in high school education, the most significant factor in learning is the teacher and their approach to teaching, which confirmed Biggs’ theory about learning [11]. In effective teaching, a good teacher is motivated, passionate about their subject, enriches the learning experience with additional content, is ready to assist and support students, and presents the material in an interesting way, connecting it to real-life situations or applying it in practice with students. Since the goal of this paper is to create a concept for a learning application that is not based on the traditional form of teaching, i.e., a teacher delivering content to students, the application needs to provide a similar experience.

Additionally, the application must feature solving tasks (Survey question - best level of knowledge) and reviewing material. Options for group work and discussions within the application should also be considered. Learning the material should be interesting and enriched with additional content to achieve the dynamism found in good teaching examples. Students using the application should have the opportunity for additional explanations of the material and some form of communication. The content must be organised and comprehensive.

Graduating students agree that the use of digital technology positively impacts the learning experience, and most of them have used educational apps. It is interesting to note that apps like Photomath and Duolingo on the Play Store have over 100 million downloads and a rating of 4.6, but graduating students rated Photomath with a score of 4.46, close to the Play Store rating, while they rated Duolingo with 3.13. They also found Photomath more helpful in learning than Duolingo. The assumption is that Photomath was more helpful due to its use in school assignments for mathematics, while Duolingo is more commonly used for personal interests. However, these results should be further investigated, especially since gamification elements, such as those in Duolingo, are crucial features of this learning application. The web application Ispravi.me moderately helped with spelling, but it does not provide explanations for spelling mistakes, and the application should deepen the learning of spelling.

From the conducted research, we can conclude that an application strictly preparing graduating students for exams would be short-term productive but would not provide long-term knowledge. Graduating students, and high school students in general, need an option for engaging learning with understanding.

5. App Prototype

According to the conducted research, the features of the learning application have been defined, and a prototype of the application has been created. Before the final app design, black and white high-fidelity wireframes of the application were created in Figma and usability testing was conducted. Usability testing is a popular methodology for researching user experience. During usability testing, the researcher (facilitator or moderator) asks participants to perform tasks, usually using one or more specific user interfaces. While the participant completes each task, the researcher observes the tester's behavior and listens to feedback. [12] Usability testing helps identify potential issues that may arise during the user's experience with the product. Once issues are identified, designers can work on correcting inconsistencies. [13] Usability testing of this application was conducted with two participants, a high school graduate and a graduate from the previous year, showing that the application is intuitive for users, and minor adjustments were made during the user interface design phase. The final design of the application can be seen in the next chapter.

5.1. Onboarding

When users first access the application, they have the option to log in or register (Figure 1). During login, they enter their email address and password, while during registration, they are required to provide their name, email address, and password. Additionally, users are allowed to quickly log in/register via their Gmail or Facebook accounts.

After that, users choose the learning option in the application (Figure 2). Users can choose whether they want to learn in-depth with understanding, i.e., the secondary school learning option, a quick learning option for state graduation exams, or both. In the exam learning mode, users will be offered only the theoretical part for studying and quizzes to solve. This feature provides an alternative solution to the illegal use of study scripts and offers a similar approach to learning. If users choose learning with understanding, each chapter in the application has additional content. After choosing the learning option, users select the type of school they attend (three-year, four-year, or five-year), and specify their grade to input their grades without including the current grade they are attending (Figure 2). After registration, there will be a specific segment on the home screen for entering the average grades from the current grade. If the user has subjects in school that are available in the application, they have the option to choose them and simply input grades for those subjects or, alternatively, manually add additional subjects. After entering the grades, the application automatically calculates the average grade for that grade level.

If users choose secondary school learning, they are redirected to screens where they can manually enter the subjects they want to study and select their preferred daily study time (Figure 3).

If they are secondary school graduates, they have the option to choose ten faculties in order of priority to which they want to apply. Based on these inputs, the application generates a personalised learning plan for users - suggesting subjects for exams and recommending optimal times for daily study (Figure 4).

5.2. Learning

After users go through the registration process, they arrive at the main screen of the learning application (Figure 5 and Figure 6). The user's main learning screen welcomes them with a personalised greeting, aiming to provide an accessible and individualised approach to each student. The bottom part of the screen contains navigation, allowing access to leaderboards, learning, AI mentor chat, and settings. In the top part of the screen, users can see their overall number of accumulated points and the available number of hints they can use during quiz solving. They can view and/or change the learning mode in the application. Users can also track their progress through the percentage of completed material, displayed subjects chosen during registration, and insights into time spent learning during the day and the past week.

In the graduation exams learning option, users have sections for preparing faculty entrance exams (Figure 7). Users can take notes on preparation, add note titles, and log the time spent studying/exam preparation. This provides them with an organisational tool that facilitates tracking the time spent preparing, contributing to better learning organisation.

By selecting subjects (Figure 8), users have options for reviewing material and accessing lessons within the chosen chapter. Users can also see the percentage of completed material for a specific subject, with the option to quickly access the next lesson or start learning. In the exam learning option, users can solve sample state graduation exams.

The lessons within the application are divided into multiple screens (Figure 9). For secondary school learning mode, users are presented with additional resources and recommendations related to the covered material. Example of a such resource is the Star Walk 2 app for deepening understanding of space-related geography.

Each part of the lesson has a button that guides the user to a chat with the AI mentor (Figure 10), providing continuous support and explanations if something is unclear or if they want to learn more.

After finishing a lesson, users have the option to take a quiz covering various question types such as single and multiple-choice, short answer, numeric questions, and matching answers. When users enter their final answers, a quiz results screen is displayed, showing the quiz score and the number of points earned. The next screen displays user responses and explanations for each question (Figure 11).

On the initial screen for learning the Croatian language, options for learning essay and summary writing are available. When the user chooses the essay option (Figure 12), they can learn how to write essays or select essay writing exercises. Essay writing learning proceeds in the same way as regular lessons. For essay writing exercises, three text entry fields are available on the screen: introduction, discussion, and conclusion. This is implemented to provide clarity in essay writing and when displaying essay results.

After the user submits an essay, a results screen appears (Figure 13), showing the achieved results. In the next section, essay results are displayed. For essay tasks and summary writing, the AI mentor reviews the work. It first shows spelling mistakes (like the Ispravi.me platform), and when a misspelt word is selected, it explains why it is misspelt. In the second step, it points out the user's knowledge errors. In the third step, it suggests how the user could have better written some sentences. This solves the problem of the Ispravi.me platform and provides a new level of learning.

When users want to review material, with a simple click on the review button, they are redirected to a screen where they can choose whether to review the entire material they have already covered in the application or only specific chapters (Figure 14). After choosing the review mode, a quiz opens, designed in the same way as in learning material.

5.3. AI Mentor

When users click on the button in the bottom navigation, a screen for interaction with the AI mentor opens (Figure 15). On that screen, a selection of different conversation options is displayed. They can choose from thematic options such as getting advice on entering their desired faculty or seeking advice if they are uncertain about choosing a faculty and tracking progress in learning material. If their questions do not fall into the mentioned topics, there is also the option of using a traditional messaging system. Additionally, users can access previous conversations with the AI mentor.

5.4. Leaderboards

On the leaderboard screen, the user has the option to see the desired category of the leaderboard – by faculty, subject, or the overall number of points (Figure 16). Leaderboards for faculties and subjects display only users who have entered them into the application. When it comes to the faculty leaderboard, the number of points is calculated based on the points users earn in subjects required or elective for admission to a specific faculty.

6. User Opinion about the Application

During usability testing, testers were asked for their impressions of the application and whether they would use it if it were on the market. As it can be seen from Table 2, they generally had positive comments about the application and did not mention any criticisms.

7. Conclusion

The concept of the application has been developed based on the identified challenges faced by secondary school graduates. This application would provide access to the grammar curriculum for students from vocational and artistic high schools, thereby enhancing their chances of enrolling in their desired faculty. The issue of script piracy from external institutions would be resolved as students would have access to the entire high school graduation material. Additional value to the application would be achieved through the option of studying for high school, which students can use for better comprehension of the high school curriculum, facilitating a better understanding of the material and compensating for potentially inadequate teaching. By developing such an application, even without an AI mentor, students would not feel the need to pay for private exam preparations. Additionally, the application would offer quizzes, reviews, and additional content that students consider crucial for attaining a high level of knowledge, with motivation provided through progress tracking. However, it would be worth considering the addition of extra features such as discussions. Further research is essential to gather the opinions of students and secondary school graduates on the proposed learning approach. Conducting additional usability testing and comparing learning functionalities with traditional methods is crucial and special attention must be given to security aspects when using artificial intelligence within the application.