Beyond ChatGPT: A Hybrid Chatbot Model for Reliable Educational Administration

1. Introduction

Educational institutions rely heavily on clear and efficient communication between students, faculty, and administrators to ensure a smooth learning experience. Timely access to information and resources is essential for academic success. Traditionally, communication has depended on in-person interactions, emails, and phone calls, which can be time-consuming and resource-intensive [1]. The advent of chatbot systems offers a promising solution to streamline administrative tasks and enhance user experiences by providing prompt and accurate responses to a wide range of inquiries [2,3]. These intelligent conversational agents can reduce the administrative burden and improve accessibility for students, faculty, and staff by providing readily available information 24/7 [4].

Despite their potential, current chatbot technologies face significant limitations. Existing approaches, such as rule-based, retrieval-based, and generative-based models, struggle to meet the diverse administrative needs of educational institutions effectively [5]. For instance, generative models like ChatGPT can handle open-ended questions but may sometimes produce inaccurate or uncertain responses [6]. This issue is particularly critical in educational settings, where the accuracy and reliability of information are paramount.

To address these limitations, we propose a novel hybrid chatbot model specifically designed for reliable educational administration. This model integrates the strengths of three distinct approaches—rule-based, retrieval-based, and generative-based chatbots—using a smart classifier system for intelligent query routing. By combining these methods, our hybrid model aims to provide superior performance and a dependable source of information within educational environments. Our research evaluates the effectiveness of individual chatbot models in handling educational inquiries, assessing their strengths and weaknesses, user satisfaction, and overall usefulness. Based on these evaluations, we have developed a hybrid model with an intelligent classifier to route queries to the most suitable approach. Our objective is to demonstrate the superiority of this hybrid model in terms of accuracy and reliability compared to individual models. This work contributes to the development of adaptable chatbot systems capable of addressing a broader range of administrative inquiries, which is particularly beneficial in resource-constrained educational environments. It also aims to inform future research on hybrid chatbots for educational settings.

The structure of the remaining sections of this paper is as follows: Section 2 provides a review of relevant related works; Section 3 discusses the methodology employed; Section 4 outlines the results and discussion of the research; and Section 5 concludes the research.

2. Literature Review

The amount of research exploring the use of chatbots in the educational field is increasing [1], and the educational landscape is witnessing a surge in interest in chatbots as valuable tools for enhancing student learning experiences and streamlining administrative tasks. This growing adoption is driven by the potential of chatbots to provide 24/7 accessibility to information and resources [7], fostering increased student engagement [8], and reducing the workload on administrative staff [9]. Studies have shown that chatbots can effectively address various needs within educational settings, which can be broadly categorized into information provision and administrative support. Regarding information provision, chatbots can address FAQs and provide users with general information regarding various aspects of educational institutions, including activities [16,17,18,19], admissions [3,9,20,21], academics [8,22,23,24], libraries [25,26], and other topics like tuition fees and campus life [12,27] . The second type of chatbot is designed to assist with administrative tasks, aiming to improve efficiency and user experience through automation. This automation benefits students, educators, and administrative staff by saving time [28,29], reducing workload and labor costs [3,23,30,31,32], offering 24/7 availability [4,7,15], increasing user experience [33,34], engagement [4,8,35], and even automating the admission process [29].

Chatbot Architectures: Based on their underlying architecture or approach, chatbot models can be classified as rule-based, retrieval-based, and generative chatbots. Rule-based chatbots follow a predefined set of rules and are designed to respond to specific keywords or phrases. Retrieval-based chatbots use a combination of predefined responses and machine learning algorithms to select the best response to a user's input. They work by matching the user's input to a database of pre-existing responses. Generative chatbots, on the other hand, use deep learning techniques like neural networks to generate responses from scratch. When it comes to chatbot development for educational institutions, the literature reflects a variety of approaches. For rule-based chatbots, some studies utilized development tools like Artificial Intelligence Markup Language (AIML) [8,30,37]. Generative-based chatbots were created using techniques such as Recurrent Neural Network (RNN) [38,39] and Unsupervised Learning [33]. Retrieval-based chatbots found application in educational settings through platforms like DialogFlow, a natural language understanding platform [17,31,32,41,42,43], as well as the RASA framework [44,45,46], and Chatterbot platform [11,47].

Comparative Analysis: While individual studies have explored different chatbot approaches, there remains a gap in the literature concerning the development of a comprehensive and adaptable chatbot model capable of effectively handling various administrative tasks. While individual chatbot approaches have shown promise, there remains a limited understanding of their relative strengths and weaknesses in educational settings. Comparative analyses can address this gap by directly evaluating different models. One study [42] compared rule-based (AIML) and generative (Sequence-to-Sequence) approaches, revealing a trade-off between user satisfaction and task completion for rule-based models, and superior information retrieval for generative models. Building upon the idea of comparative analysis, another study [1] developed five chatbot models using various techniques and found limitations in pattern-matching approaches due to overfitting.

Generative models: The emergence of large language models like ChatGPT, have garnered significant interest for their ability to handle open-ended questions and generate creative text formats [43]. In the context of educational administration, this translates to the potential for providing students with explanations, summaries, and even creative writing prompts. In their work, the authors of [44] identified several key advantages of using ChatGPT in education. These include the ability to tailor learning to individual students, develop innovative assessment methods, and enhance students' critical thinking abilities. They suggest that by carefully considering both the potential benefits and drawbacks of ChatGPT, it can be used to improve the overall learning experience for university students.

Research Gap: The studies reviewed in this section showcase the potential of various chatbot approaches in educational settings. However, some limitations exist. Rule-based models can be inflexible for open-ended questions, while retrieval-based models may struggle with complex inquiries. Generative models, like ChatGPT, due to their reliance on statistical patterns in vast datasets can lead to inaccuracies, particularly in factual inquiries [45]. Additionally, generative models may struggle to understand the nuances of human language and context, potentially leading to misleading or irrelevant responses. Our research proposes a hybrid chatbot model that leverages the strengths of these approaches to overcome these limitations. By combining rule-based, retrieval-based, and generative-based approaches, we aim to develop a more robust and adaptable model for handling diverse administrative tasks in educational institutions.

3. Materials and Methods

This section details the methods employed in the comparative analysis of rule-based, retrieval-based, and generative-based chatbots designed to address administrative inquiries within educational institutions, it also proposes the development and evaluation of a hybrid model that leverages the strengths of these individual approaches. The methodology, further illustrated in Figure 1, encompasses five key steps: data collection, model development, model evaluation and comparison, classifier implementation, and model integration.

3.1. Data Collection and Dataset Creation

A dataset is crucial for training any model. Due to the lack of datasets specifically tailored to Sakarya University, we constructed a new dataset from scratch. This dataset comprises a collection of questions and corresponding answers stored in a text file format. To ensure clarity and organization, a consistent notation system has been adopted. Question sentences are marked with a "Q:" prefix, while answer sentences are denoted with an "A:". The data has been sourced from various channels: Sakarya University's FAQ pages, and social media groups where students posed questions about the university and questions were generated based on information available on the university's official websites. This comprehensive approach ensured the dataset captured a wide range of inquiries typically encountered by students. Additionally, incorporating valuable feedback from students further enriched the dataset's relevance. data pre-processing steps have been performed (removing punctuation and duplicate questions, segmenting long sentences into smaller units) to clean and organize the collected information. The resulting dataset encompasses 2253 lines of conversation, categorized into various domains such as Admissions and Registrations, Academic Affairs, Financial Matters, and Student Services. This structure ensures the dataset's broad coverage and addresses the needs of prospective, current, and foreign students at Sakarya University.

3.2. Models Creation

The core of this research hinges on the development and application of diverse models. This subsection, 3.2 Models Creation, delves into the specific steps taken to construct these models. We will detail the implementation of all three models: a rule-based approach, a retrieval-based approach, and a generative-based approach.

3.2.1. Rule-Based Chatbot

To develop a rule-based chatbot, Artificial Intelligence Markup Language (AIML) was selected. AIML is a widely used programming language for creating chatbots and virtual assistants. It is an XML-based language that uses pattern matching and natural language processing techniques to simulate conversation with human users [46]. It has been opted for due to its simplicity, flexibility, and user-friendly syntax. AIML's built-in features enable efficient handling of diverse user queries [46]. The Pandorabots platform has been utilized. Pandorabots is a platform that utilizes AIML to build and deploy chatbots which provides a user-friendly interface for defining AIML rules, managing the knowledge base, and integrating the chatbot [47]. 637 categories were established from the dataset described in section 3.1. The development process can be summarized as follows:

• AIML File Generation: The Pandorabots playground was used to generate AIML files from the dataset, creating the initial chatbot prototype.
• Prototype Integration: The prototype was then integrated into the Pandorabots framework, enabling user interaction.
• User Testing: A group of 113 students (undergraduate and postgraduate) interacted with the chatbot.
• Evaluation and Improvement: User interactions were logged and analyzed (elaborate on the specific aspects evaluated, e.g., response accuracy, user satisfaction). This analysis led to a review and enhancement of the chatbot's functionalities.
• Redeployment: The improved version of the chatbot was redeployed for further user interaction.

Figure 2 shows the pseudocode for the built rule-based chatbot model, and Figure 3 represents its graphical interface.

3.2.2. Retrieval-Based Chatbot

The retrieval-based chatbot leverages a JSON file to store information relevant to user inquiries. This file maintains data regarding Sakarya University, categorized into the same main categories as the dataset (section 3.1). The JSON structure includes:

• intents: An array of intent objects, each representing a specific category such as ‘weather’, ‘courses’, and ‘tuition fees’.
• Questions: An array of user queries or inputs associated with the intent.
• Responses: An array of possible responses or answers the chatbot can provide related to the given intent.

Data pre-processing is crucial for the retrieval-based model. This involves:

• Tokenization: Breaking down text into individual words or units.
• Stopword removal: Eliminating common words that hold little meaning such as ‘the’, and ‘a’.
• Stemming: Reducing words to their base form such as ‘running’ -> ‘run’. Porter Stemmer has been employed for this purpose.
• Processed dataset creation: Combining the preprocessed questions (patterns) and corresponding preprocessed answers (randomly chosen from the response options).

TF-IDF (Term Frequency-Inverse Document Frequency) is a core technique used in this model. It is a numerical statistic used to evaluate the importance of a word to a document in a collection or corpus [48]. A TF-IDF vectorizer is initialized to calculate the similarity matrix using cosine similarity. This step aims to determine the closest match between the user's input and the preprocessed questions within the dataset.

Entity extraction involves identifying specific elements within the user's input that are relevant to the query. This is achieved through a function named extract entities. It performs the following actions:

• Tokenization: Breaking down the user input into individual words.
• Pattern matching: Checking the user input against predefined patterns associated with the intent.
• Entity extraction: Utilizing regular expressions to extract specific entities from the matched patterns.

Finally, the generated response function takes the user input and generates a response. This function:

• Preprocesses the user input.
• Calculate the similarity between the preprocessed input and the preprocessed dataset questions using cosine similarity.
• Identifies the most similar question (intent) based on the calculated similarity scores.
• Extract entities relevant to the identified intent using the extract_entities function.
• Returns the corresponding response associated with the most similar question and extracted entities.

Figure 4 illustrates the pseudocode for the constructed retrieval-based chatbot model, while Figure 5 displays its graphical interface.

3.2.3. Generative-Based Chatbot

This section clarifies the fine-tuning process employed to convert a pre-trained language model into a functional chatbot using the Hugging Face transformers library. The objective is to fine-tune the model on a custom Q&A dataset (section 3.1) and leverage it to generate responses to user queries. The generative-based chatbot implementation can be summarized as follows:

• Library Imports: Necessary libraries are imported, including torch for PyTorch functionalities and AutoTokenizer and AutoModelForSeq2SeqLM from the transformers library for handling the pre-trained model and tokenizer.
• GPU Availability Check: The code checks for a CUDA-enabled GPU to accelerate computations. If available, the GPU is used; otherwise, the CPU takes over.
• Loading the Dataset: The dataset created in section 3.1 is loaded line by line, separating each line into individual questions and answers.
• Separating Questions and Answers: The entire dataset is traversed to split questions and answers into distinct lists.
• Loading Pre-trained Model and Tokenizer: The pre-trained model ("tuner007/pegasus_paraphrase") is specified, and the corresponding tokenizer and model are loaded using AutoTokenizer and AutoModelForSeq2SeqLM. The loaded model is then transferred to the chosen device (GPU or CPU).
• Tokenization and Encoding: Both questions and answers are tokenized and encoded using the loaded tokenizer. Padding and truncation are applied to ensure sequences adhere to specific length requirements. These encoded sequences are then converted into PyTorch tensors.
• Fine-tuning the Model: Fine-tuning involves the following sub-steps:

• Optimizer Selection: An Adam optimizer is chosen and initialized with a learning rate of 5e-5. The fine-tuning process iterates through 40 epochs.
• Encoding Input and Output Sequences: For each epoch, the model encodes the input and output sequences for each Q&A pair.
• Generating Output and Calculating Loss: The model generates an output sequence based on the input sequence and calculates the loss against the provided target output sequence.
• Backpropagation and Updating Model Parameters: The calculated loss is accumulated and used to update the model's parameters through backpropagation.
• Saving Model State: After each epoch, the model's state (including both model weights and tokenizer) is saved whenever a decrease in the average loss is observed.

8.

Model Evaluation: The fine-tuned model's performance is assessed using the training set.

9.

User Interaction with Fine-tuned Model: The final step involves user interaction with the chatbot:

• Optimizer Selection: An Adam optimizer is chosen and initialized with a learning rate of 5e-5. The fine-tuning process iterates through 40 epochs.
• Encoding Input and Output Sequences: For each epoch, the model encodes the input and output sequences for each Q&A pair.
• Generating Output and Calculating Loss: The model generates an output sequence based on the input sequence and calculates the loss against the provided target output sequence.
• Backpropagation and Updating Model Parameters: The calculated loss is accumulated and used to update the model's parameters through backpropagation.
• Saving Model State: After each epoch, the model's state (including both model weights and tokenizer) is saved whenever a decrease in the average loss is observed.

Figure 6. Generative-based model algorithm.

Figure 6. Generative-based model algorithm.

Figure 7. Generative-based chatbot GUI.

Figure 7. Generative-based chatbot GUI.

3.3. Models’ Evaluation and Comparison

This section presents a comparative analysis of three prominent chatbot models utilized in educational institutions: rule-based, retrieval-based, and generative-based models. Through a rigorous evaluation process to:

• Unveil the strengths, weaknesses, and unique performance characteristics of each approach.
• Lay the groundwork for identifying advancements and exploring the feasibility of a hybrid model that leverages the most advantageous features of all three approaches.

Figure 8 visually depicts the lifecycle of the comparison process, providing a roadmap for the subsequent sections.

3.3.1. Define Evaluation Metrics

The initial step involves determining relevant metrics to assess the chatbot models' performance, the core metrics include:

• Accuracy: Percentage of inquiries answered correctly and relevantly.
• Precision: Proportion of responses that are topical and pertinent to the user's query.
• Recall: Proportion of relevant questions accurately answered.
• F1 Score: Harmonic means of precision and recall, offering a balanced view.

In addition to the core metrics, additional metrics have been considered such as:

• Response Time: Average time taken to generate a response.
• User Satisfaction and Engagement: Subjective metrics assessed through surveys to gauge user perception of helpfulness, ease of use, and overall effectiveness.
• Flexibility: Ability to adapt to new information or situations such as ease of adding new knowledge, handling unexpected queries.

3.3.2. Prepare Evaluation Data

To ensure an accurate evaluation, a dedicated dataset was compiled using real-world data, the factors have been considered are:

• Data Source: Extracted from the last updated FAQ page on Sakarya University's main website.
• Selection Criteria: 300 representative queries and corresponding answers, representing approximately 20% of the total training dataset size, were chosen to capture the diverse range of user inquiries typically encountered in the university setting.
• Rationale: Incorporating these authentic and up-to-date queries aims to assess the models' performance in accurately understanding and responding to inquiries specific to Sakarya University.

3.3.3. Setup Evaluation Scenarios

Three distinct scenarios or use cases are defined to represent the digital assistant's functionalities:

• Scenario 1 (Prospective Students): Admissions, academics, and general inquiries.
• Scenario 2 (Enrolled Students): Registrations, tuition fees, courses, and campus facilities.
• Scenario 3 (Foreign Students): Admission and registration, tuition fees, weather, language proficiency requirements, and accommodation.

These scenarios encompass various domains and topics relevant to Sakarya University.

3.3.4. Evaluation Process

In this step, the specific evaluation approaches for each model have been outlined:

• Rule-based Chatbot:

• Focus: Ability to match user queries to predefined rules and generate appropriate responses.
• Metrics: Measure the accuracy of matching and rule coverage.

2.

Retrieval-based Chatbot:

• Focus: Selecting the most relevant response from a predefined set.
• Metrics: Use accuracy, precision, recall, and F1 score to evaluate response selection accuracy.

3.

Generative-based Chatbot:

• Focus: Assess the quality of generated responses.
• Factors: Consider response coherence, grammar, and relevance to the user query.

3.3.5. Conduct User Surveys

User surveys were conducted to gather feedback on:

• Satisfaction Level: How satisfied users were with the interaction.
• Perceived Usefulness: How helpful users found the chatbot.
• Overall User Experience: Ease of use and overall impression.

A total of 267 undergraduate students participated by responding to the 10-question survey (Table 1). All user interactions and feedback were recorded for further analysis.

3.3.6. Compare and Analyze Results

This final step involves:

• Comparing performance: Analyze the performance of each model across the different evaluation metrics and user feedback.
• Identifying strengths and weaknesses: Examine each model's effectiveness in terms of accuracy, response quality, user satisfaction, and other relevant factors.

The purpose of evaluating the models was to:

• Review and improve: The evaluation aims to identify areas for improvement in both the dataset and the design of the models.
• Hybrid model development: By comparing the models' strengths, the analysis seeks to explore the possibility of integrating them into a single, enhanced hybrid model.

3.4. Classifier Implementation

The evaluation process revealed the distinct strengths of each chatbot model in handling administrative tasks across various university-related categories. To efficiently route user queries to the most suitable model, a classifier is implemented. This section details the chosen approach and its implementation steps. Figure 9 presents the pseudocode for the Multinomial Naive Bayes Classifier, while Figure 10 displays a sample of the classifier dataset.

3.4.1. Classifier Selection:

A Multinomial Naive Bayes classifier is employed for category classification. A Multinomial Naive Bayes Classifier is a probabilistic machine learning algorithm used for text classification [49]. It assumes that the occurrence of a word in a document is independent of the occurrence of other words. due to its:

• Simplicity: Easy to understand and implement.
• Efficiency: Performs well with large datasets and requires minimal training data compared to some complex models.
• Effectiveness: Often yields competitive results for text classification tasks.

3.4.2. Data Preparation:

A dedicated dataset (Figure 10) was created for classifier training, containing:

• Question Column: Represents the input user query as text.
• Category Column: Represents the corresponding category or class such as admissions, registration, and fees.

3.4.3. Implementation Steps:

• Data Loading: The Question and Category columns are loaded into separate lists, typically named corpus and labels, respectively, for easier processing.
• Text Classification Pipeline:

• The Pipeline class from scikit-learn is utilized to create a text classification pipeline.
• The pipeline consists of two crucial steps:

•    Text Vectorization: The CountVectorizer transforms textual data into numerical features, enabling the classifier to process the text.
•    Classification: The MultinomialNB algorithm implements the Naive Bayes classification model to categorize the text data based on the learned patterns.

3.

Data Splitting:

• The train_test_split function from scikit-learn is used to divide the dataset into training and testing sets.
• A common practice is to allocate 40% of the data for testing (test_size=0.4) to assess the classifier's performance on unseen data.
• Setting the random_state parameter to a fixed value (e.g., 42) ensures reproducibility, meaning the same data split will occur when the code is run multiple times.

4.

Classifier Training:

• The training set is used to train the classifier using the fit method of the pipeline.
• During this process:

•        The text data from the training set is vectorized using CountVectorizer.
•        The transformed numerical features are then used to train the Naive Bayes model, enabling it to learn the relationships between text patterns and their corresponding categories.

5.

Model Saving:

• The trained classifier is saved to a file (classifier.pkl) using the joblib.dump function.
• This step allows for reusing the trained model for future predictions without the need for retraining, saving computational resources.

6.

Evaluation:

• The performance of the trained classifier is assessed using the testing set (X_test).
• The predict method is employed to predict category labels for the unseen test data.
• The accuracy_score function is used to evaluate the model's accuracy by comparing the predicted labels with the actual labels (y_test).
• Accuracy represents the percentage of correctly classified instances.

Figure 9. Multinomial Naive Bayes Classifier algorithm.

Figure 9. Multinomial Naive Bayes Classifier algorithm.

Figure 10. Sample of Classifier Dataset.

Figure 10. Sample of Classifier Dataset.

3.5. Models Integration (Hybrid Model)

Building upon the individual strengths of the rule-based, retrieval-based, and generative-based models, this section describes their integration into a cohesive framework, forming a hybrid chatbot system.

The integration process revolves around two key components:

• Classifier: The trained Multinomial Naive Bayes classifier, as explained in Section 3.4, plays a crucial role in efficiently routing user queries.
• Main Program Loop: This core loop manages the user interaction flow within the system.

Figure 11 Represents the pseudocode for the proposed hybrid model, and Figure 12 Describes it’s process flow which can be summarized as:

• User Query Input: The user submits their query through the system's interface.
• Query Processing Function (process_user_query(query))

a. Category Prediction: The function leverages the trained classifier to predict the category (e.g., admissions, registration, fees) that the query most likely belongs to.

b. Model Selection: Based on the predicted category, the function directs the query to the most suitable model:

•   Rule-based model: Well-suited for well-defined queries with a clear answer in the knowledge base.
•   Retrieval-based model: Effective for retrieving relevant responses from a predefined set when the answer might involve variations in phrasing.
•   Generative-based model: Ideal for handling open-ended questions, complex inquiries, or situations where a creative response is desired.

3.

Model Response Generation: The selected model processes the query and generates a response tailored to the user's needs.

4.

Response Delivery: The generated response is returned from the process_user_query(query) function to the main program loop.

5.

User Interface Interaction: The main program loop displays the response to the user through the system's interface, completing the interaction cycle.

Figure 11. Hybrid model algorithm.

Figure 11. Hybrid model algorithm.

Figure 12. Hybrid Model Flow Diagram.

Figure 12. Hybrid Model Flow Diagram.

4. Results And Discussion

For randomly selected 300 different queries, the performance of all three chatbot approaches was evaluated. The rule-based chatbot achieved a matching accuracy of 73.33% for user queries to predefined rules, effectively generating appropriate responses. It covered 63.33% of the rules defined within AIML categories (Table 2).

On the other hand, the retrieval-based chatbot demonstrated a higher performance with a score of 83.33% in selecting the most relevant response from a predefined set of responses. Additionally, it provided a wide range of response diversity, enhancing the overall user experience (Table 3).

To evaluate the generative-based chatbot, the chatbot's responses for the same set of queries were assessed by an English specialist annotator. The manual evaluation using the Likert scale (strongly agree(5), agree(4) , neutral(3), disagree(2), strongly disagree(1)) revealed that the model achieved scores of 72.12%, 75.42%, 87.23% for response quality, coherence, grammar, and language fluency, respectively (Table 4).

The evaluation results highlighted the need for improvements in all three chatbot models and the dataset as well. For the rule-based model, an expansion of the rules defined within AIML categories was necessary to cover a wider range of user queries. As a result, an additional 112 categories were added to enhance the model's coverage. In the case of the retrieval-based chatbot, the random() function was activated to introduce more response diversity, especially for queries that could have multiple valid answers. This adjustment aimed to enhance the user experience and provide varied responses. Furthermore, to improve the performance of the generative-based model, the dataset was modified to include more examples that consider comprehension and context, thereby ensuring the generation of more contextually appropriate and coherent responses. Table 5 summarizes the results of the questionnaire administered to users who interacted with all three models. The scores were calculated for each question and for all three models using Equation 1. User satisfaction rates were determined based on questions 2, 6, and 7. The retrieval-based model received the highest score in user satisfaction. Model usefulness was assessed through questions 3, 8, and 9, and the rule-based model achieved the highest score in this category. User engagement was evaluated using questions 5 and 10, and the rule-based model also scored highest in this aspect. Question 4 focused on measuring human-like conversation, and the generative-based model received the highest score in this area.

$$\mathrm{Score}={\displaystyle \frac{\left(summationofall67students\right)}{10\ast 67}}\ast 2$$

(1)

The evaluation results and user feedback revealed distinct advantages for each chatbot model. The rule-based model demonstrated superior performance in terms of model usefulness and user engagement. This was primarily attributed to its ability to provide accurate answers in specific categories such as tuition fees and course schedules. Additionally, the rule-based model offered an attractive user interface through the utilization of rich media tags provided by the AIML playground (Pandorabots). The retrieval-based model outperformed the others in terms of response selection, leading to higher user satisfaction. The generative-based model excelled in delivering a more human-like conversation experience, owing to its implementation of a generative pre-trained model with a transformer architecture. To assess the strengths of each model in handling various administrative tasks within Sakarya University and to propose the hybrid model, a detailed analysis was conducted on all the conversations. The task completion rates for each model are summarized in Table 6. The results indicate that the rule-based model performed well in the categories of Admissions and Registrations (87.33%) and Financial Matters (89.56%). The retrieval-based model achieved the best results in the categories of Academic Affairs (84.67%), Student Services (88.94%), and Campus Life (91.44%). On the other hand, the generative-based model excelled in the categories of General Information (100%) and Cross-domain Queries (73.12%).

This categorization resulted from the analysis of the study, where the specific chatbot implementation, quality of data, and training provided to the chatbot models contribute to the different strengths exhibited by each chatbot model in specific task categories. As part of the evaluation process, the individual responses of the chatbots have been analyzed. Table 7 presents the various response categories.

Using the information provided in Table 8, Accuracy, Precision, Recall, and F-Measure were computed utilizing the following formulas [50]:

$$Accuracy={\displaystyle \frac{\mathrm{True}\mathrm{Positives}+\mathrm{True}\mathrm{Negative}}{\mathrm{True}\mathrm{Positives}+\mathrm{True}\mathrm{Negative}+FalsePositives+FalseNegative}}$$

(2)

$$Precision={\displaystyle \frac{\mathrm{True}\mathrm{Positives}}{TruePositives+FalsePositives}}$$

(3)

$$Recall={\displaystyle \frac{\mathrm{True}\mathrm{Positives}}{TruePositives+FalseNegative}}$$

(4)

$$F-score=2\ast {\displaystyle \frac{\mathrm{Precision}\ast \mathrm{Recall}}{Precision+Recall}}$$

(5)

Accuracy measures the overall correctness of the model's predictions. the retrieval-based model still has the highest accuracy among the three models. The accuracy alone is not sufficient to evaluate a model's performance comprehensively. Precision and recall provide additional insights into the model's ability to correctly identify positive instances and minimize false positives or false negatives, respectively. Precision is a measure of how many of the responses predicted as positive are actually correct, the retrieval-based model still has the highest precision, meaning that when it predicts a response as positive, it is correct 78% of the time. Recall measures the ability of the model to correctly identify positive responses, the retrieval-based model still has the highest recall, indicating that it can correctly identify positive responses 88% of the time. F1-Measure is the harmonic mean of precision and recall and provides a single metric to evaluate the overall performance, The retrieval-based model still has the highest F1-Measure, indicating a balanced performance in terms of precision and recall.

The purpose of evaluating the models was to review and make the necessary modifications to both the dataset and the design of the models. Additionally, the purpose of comparing the models was to identify their respective strengths to integrate them into a single hybrid model. The models' evaluation has been completed and required modifications have been made to both the dataset and the design of the models. Based on the models' comparison, the polynomial Naïve Bayes classifier has been implemented to forward specific types of queries to different models. For instance, queries related to 'Admissions and Registrations' and 'Financial Matters' are directed to the rule-based model. 'Academic Affairs', 'Student Services', and 'Campus Life' queries are handled by the retrieval-based chatbot, while 'General Information' and 'Cross-domain' queries are addressed by the generative-based model. Table 9 demonstrates the advancements and accuracy achieved by the hybrid model. The evaluation of the model utilized the updated dataset, with an equal number of queries employed for all three models. The results showcased a noteworthy improvement in comparison to the rule-based, retrieval-based, and generative-based models. This progress serves as evidence that the hybrid model effectively utilizes the strengths of all three models. This achievement stands as the primary contribution of this work.

According to [2] assessing the practicality of Chatbot systems requires employing a suitable method to evaluate the efficiency of the software engineering product, along with a broader and statistically meaningful sample of users.

A detailed comparison in Table 10 reveals the proposed hybrid model surpasses all existing state-of-the-art administrative educational chatbots. This superiority is achieved through combining the strengths of each model (rule-based, retrieval-based, and generative-based models). For example, the hybrid model leverages the rule-based system's efficiency for handling common tasks, the retrieval-based system's ability to access and deliver relevant information, and the generative-based system's capacity for natural language understanding and response generation.

5. CONCLUSIONs

This study investigated the potential of hybrid chatbot models for enhancing educational administration. By comparing rule-based, retrieval-based, and generative-based approaches, we identified distinct strengths and weaknesses in handling student inquiries. Our findings revealed that rule-based models excel in domains such as Admissions and Registrations, and Financial Matters, while retrieval-based models demonstrate superior performance in Academic Affairs, Student Services, and Campus Life. Generative models proved effective in handling Greeting and General Information inquiries as well as cross-domain queries. Our proposed hybrid model, which combines these approaches and employs a query routing classifier, demonstrated an accuracy improvement over individual models. This enhancement underscores the model's potential to optimize efficiency and user experience within educational settings. Benchmarking against state-of-the-art solutions further solidified the model's competitive performance. Future research should focus on expanding the dataset for more robust model training, exploring advanced classification algorithms such as deep learning-based techniques that could further enhance query routing and overall system performance, and conducting comprehensive user studies to assess the model's impact on student satisfaction and administrative efficiency.