An Intelligent Conversational Agent for Automated Training and Assessment in the Hospitality Industry

1. Introduction

The hospitality industry operates in an increasingly competitive global market where service quality directly impacts customer satisfaction, brand reputation, and operational success [1]. Hotels must ensure their staff members are thoroughly trained in complex operational procedures spanning housekeeping, reception, maintenance, food service, and safety protocols. However, traditional training methods present significant challenges: they are resource-intensive, requiring dedicated trainers and substantial time investments; they lack scalability, making it difficult to train multiple employees simultaneously across different locations; they offer limited personalization, failing to adapt to individual learning paces and knowledge gaps; and they provide delayed feedback, preventing immediate correction of misunderstandings [2,3].

These limitations are particularly acute in sectors with high employee turnover rates, such as hospitality, where rapid onboarding of new personnel is essential while maintaining consistent service standards [4]. The COVID-19 pandemic further exacerbated these challenges, highlighting the urgent need for remote, scalable, and effective training solutions that minimize in-person contact while maintaining educational quality [5].

The impact of these shortcomings becomes especially evident in high-pressure operational scenarios that are common in the hospitality industry. For instance, a hotel receptionist with only a few weeks of experience may suddenly face the arrival of a delayed international tour group late at night, requiring immediate coordination, multilingual communication, and efficient problem solving under stress. In such situations, inexperienced staff members often struggle with uncertainty regarding language choice, guest interaction protocols, or exceptional service requests, which can negatively affect both employee confidence and guest satisfaction.

The system proposed in this work aims to mitigate these challenges by acting as an intelligent real-time assistant for hospitality personnel. Rather than relying solely on prior training or supervisor availability, employees can receive contextual guidance and operational support during critical interactions, enabling them to act with confidence from their very first days on the job. Furthermore, as the system evolves, it is envisioned not only as a support tool for staff members but also as a direct assistance platform for guests themselves, capable of autonomously handling common requests, multilingual communication, and service-related inquiries in real time.

Recent advances in Artificial Intelligence (AI), particularly in Large Language Models (LLMs) such as GPT-4, Claude, and Gemini, have opened transformative opportunities for education and training [6,7]. These models demonstrate remarkable capabilities in natural language understanding, content generation, contextual reasoning, and personalized interaction [8]. Unlike traditional rule-based chatbots, LLM-powered conversational agents can engage in sophisticated dialogues, generate diverse educational content dynamically, provide nuanced feedback, and adapt to individual learner needs [9].

Despite growing research on AI in education, there remains a significant gap in the literature regarding end-to-end, LLM-powered systems specifically designed for comprehensive corporate training in vocational domains. Most existing studies focus on either content generation or assessment in academic settings, rather than integrated solutions for industry-specific professional development [10].

This paper addresses this gap by presenting the design, implementation, and validation of an intelligent conversational agent tailored for automated training and assessment in hotel operations. Our contribution is threefold:

1.

We develop a complete LLM-based training system that integrates content delivery, practice exercises, formal assessment, and automated reporting within a single conversational interface.

2.

We demonstrate the system’s effectiveness through rigorous Technology Readiness Level 4 (TRL-4) validation in a controlled laboratory environment, achieving 100% success across five comprehensive functional test cases.

3.

We provide a replicable framework that can be adapted to other service industries requiring standardized yet personalized training at scale.

The system leverages Google’s Gemini 2.0 Flash model to power natural language interactions across four key modalities: answering general hotel inquiries, delivering structured training content, providing practice drills, and conducting formal assessments. The agent dynamically generates diverse question types (True/False, Multiple Choice, Open-ended, and Scenario-based) from internal hotel documentation, automatically evaluates responses, provides personalized constructive feedback, and generates comprehensive performance reports with data visualizations. The remainder of this paper is organized as follows: Section 2 reviews related work on intelligent tutoring systems, LLMs in education, and technology adoption in hospitality; Section 3 describes our system architecture, design decisions, and validation methodology; Section 4 presents the results of our TRL-4 evaluation; Section 5 discusses the implications, limitations, and comparison with existing approaches; and Section 6 concludes with future research directions.

2. Materials and Methods

2.1. System Architecture and Design

The proposed intelligent conversational agent follows a modular three-layer architecture designed to ensure scalability, maintainability, and separation of concerns. Figure 1 illustrates the system’s high-level architecture comprising: (1) the presentation layer (frontend), (2) the business logic layer (backend), and (3) the cognitive layer (LLM integration). This modular design allows for independent development, testing, and future enhancement of each component while maintaining system cohesion.

2.1.1. Presentation Layer: Conversational Interface

The frontend is implemented using Chainlit [12], a Python framework specifically designed for building production-ready conversational AI applications. Chainlit provides native support for real-time messaging, persistent session management, file handling and display, tool execution callbacks, and multi-turn conversation context. The interface components are built with React and TypeScript, ensuring type safety and reducing runtime errors. Tailwind CSS provides utility-first styling, enabling rapid development of a modern, responsive user interface that adapts seamlessly across desktop and mobile devices.

The conversational interface guides users through four primary interaction modes, presented as clearly labeled selectable options upon system initialization:

1.

General hotel inquiries – Users can ask natural language questions about hotel policies, procedures, facilities, operational guidelines, and general information. The agent retrieves relevant information from the processed internal knowledge base and provides contextually appropriate responses with source attribution.

2.

Structured training – Users select specific training objectives from a predefined list aligned with hotel operational areas (e.g., housekeeping procedures, guest relations protocols, safety regulations, food service standards). The agent presents comprehensive instructional content extracted from official hotel documentation, organized in manageable segments. Users can request clarification or additional details through follow-up questions.

3.

Practice drills – Users engage in self-paced practice by answering dynamically generated questions focused on specific objectives or spanning general content. This mode emphasizes skill reinforcement and knowledge retention without formal scoring. The agent provides immediate correctness feedback and explanatory guidance to support learning, but results are not permanently recorded.

4.

Formal assessment – Users complete comprehensive evaluations covering all training content or specific objective domains. The system generates diverse question types, records all responses with timestamps, calculates objective scores, provides detailed constructive feedback, generates performance analytics, and produces exportable PDF reports. Assessment results are permanently stored for progress tracking and competency verification.

The interface employs conversational design principles to minimize cognitive load: users interact primarily through natural language input with supplementary button-based actions for structured responses (e.g., True/False, Multiple Choice selections). The system provides clear progress indicators during assessments, confirmation dialogues for critical actions, and contextual help throughout the interaction flow.

2.1.2. Backend Processing System

The backend is developed in Python 3.11.9 using FastAPI [13], a modern, high-performance web framework built on Starlette for asynchronous request handling and Pydantic for data validation. FastAPI’s automatic generation of OpenAPI documentation facilitates API testing and future integration with external systems. The framework’s native support for asynchronous operations enables efficient handling of concurrent user sessions and non-blocking LLM API calls.

Key backend modules and their functionalities include:

1.

Document processing module: This module utilizes PyMuPDF (fitz) [14] and PyPDF2 [15] libraries to extract textual content and embedded images from PDF documents containing hotel protocols and training materials. PyMuPDF handles complex PDF parsing, including multi-column layouts, tables, and graphics, while PyPDF2 provides supplementary extraction capabilities. The extraction process preserves document structure, including hierarchical headings (Objective 1, 1.1, 1.1.1), paragraphs, ordered and unordered lists, and embedded images with positional context. Pillow (PIL) [16] processes extracted images, enabling format conversion, resizing, and potential optical character recognition (OCR) for image-embedded text. The processed content is indexed by training objectives, creating a structured knowledge base optimized for semantic retrieval.

2.

Question generation module: This module interfaces with the LLM through carefully engineered prompts to generate questions based on specified training objectives, difficulty levels, and content scope. The module implements four distinct question generation pipelines:

• True/false questions: Binary choice items designed to test factual knowledge, procedural understanding, and identification of common misconceptions. The prompt instructs the model to generate unambiguous statements that are clearly true or false based on source documentation.
• Multiple choice questions: Four-option questions with one correct answer and three plausible distractors. The prompt specifies that distractors should reflect common errors or partial understanding, not arbitrary incorrect options. The model is instructed to avoid "all of the above" or "none of the above" options to ensure clarity.
• Open-ended questions: Free-response items requiring detailed explanations, procedure descriptions, or problem analysis. The prompt guides the model to formulate questions that assess deeper comprehension, critical thinking, and ability to articulate knowledge. Expected answer length and key points are specified in the generation prompt to facilitate subsequent evaluation.
• Scenario-based questions: Situational problems simulating realistic hotel scenarios (e.g., guest complaints, emergency situations, operational conflicts). The prompt instructs the model to create authentic scenarios with sufficient context, followed by questions evaluating decision-making, problem-solving, and application of training content. Scenarios include relevant details (time of day, guest type, available resources) to enable nuanced responses.

For each question type, the generation prompt includes example questions, quality criteria (e.g., clarity, relevance, and appropriate difficulty), and formatting specifications. The system currently constrains question generation to four predefined formats: true/false, multiple choice, open-ended, and scenario-based questions. This limitation acts as a proactive guardrail mechanism for the language model, reducing output variability and minimizing the risk of generating structurally inconsistent or pedagogically unsuitable content.

The first two formats, namely true/false and multiple choice questions, provide highly controlled answer spaces and therefore constitute the most restrictive and reliable assessment types. In contrast, open-ended and scenario-based questions offer greater flexibility and realism, enabling the evaluation of contextual reasoning and decision-making skills that are particularly relevant in hospitality environments. However, these more generic formats also introduce a higher likelihood of hallucinations, ambiguities, or factual imprecisions in the generated content.

For this reason, the constrained design of the question generation module is complemented by the subsequent assessment module, which performs a reactive validation and review process to detect potential inconsistencies, inaccuracies, or low-quality outputs before the content is ultimately delivered to users.

3.

Assessment and evaluation module: This module implements a hybrid evaluation approach tailored to question type characteristics. For closed-ended questions (True/False and Multiple Choice), the system performs deterministic evaluation by exact string matching between user responses and predefined correct answers, ensuring 100% reliability and immediate feedback. For open-ended and scenario-based questions, the module constructs specialized evaluation prompts that instruct the LLM to assess response quality against multiple criteria: factual accuracy (correctness of stated information), completeness (coverage of required key points), coherence (logical organization and clarity), and practical applicability (demonstration of understanding how to apply knowledge). The evaluation prompt includes the question text, expected answer components, user’s response, and assessment rubric. The model returns structured feedback containing: (a) overall quality rating, (b) identification of strengths (specific correct elements), (c) identification of weaknesses or omissions, (d) constructive suggestions for improvement, and (e) an estimated score or proficiency level. Among these dimensions, the overall quality rating is considered the most critical indicator for ensuring reliable system behavior, as it provides a concise and robust summary of the model’s evaluation capabilities. The identification of strengths and weaknesses is also highly relevant, since these elements offer grounded and interpretable feedback directly tied to the user’s response.

In contrast, the final two components—constructive suggestions and estimated proficiency levels—while still useful from a pedagogical perspective, are inherently more speculative. These outputs require the model to extrapolate beyond direct factual assessment, increasing the likelihood of subjective interpretations or less reliable recommendations. Consequently, the system places greater emphasis on the first three dimensions when evaluating assessment quality and consistency.

To ensure evaluation consistency, the module implements validation checks on LLM-generated assessments, including completeness verification (all rubric criteria addressed) and tone analysis (ensuring constructive, non-judgmental feedback). Ambiguous or incomplete evaluations trigger regeneration with modified prompts.

4.

Reporting and visualization module: Upon assessment completion, this module aggregates user performance data and generates comprehensive PDF reports using ReportLab [17]. Reports include multiple components:

• Executive summary: Overview of overall performance, final score, completion time, and proficiency level.
• Performance breakdown: Detailed analysis by question type (True/False, Multiple Choice, Open-ended, Scenario-based), showing correct answers, incorrect answers, and score percentages for each category.
• Visual analytics: Charts and graphs created with Matplotlib [18], including bar charts showing performance by objective area, line graphs displaying progress over multiple assessments (if historical data exists), and radar charts illustrating competency profiles across different skill domains.
• Response history: Complete record of all questions presented, user responses, correct answers, and feedback provided. Open-ended responses are included with evaluator comments and improvement suggestions.
• Personalized recommendations: LLM-generated summary identifying overall strengths, priority areas for improvement, and suggested next learning activities based on performance patterns.

The module formats reports professionally with hotel branding elements (if provided), clear section headings, appropriate pagination, and table of contents for lengthy reports. Reports are generated asynchronously to avoid blocking the user interface during PDF compilation.

5.

Session management module: This module maintains comprehensive session state for each authenticated user, stored in structured JSON format. Session data includes: user identification and metadata, complete interaction history (all messages, timestamps, and context), training progress (objectives covered, materials viewed, time spent), assessment records (questions answered, scores achieved, feedback received), and system configuration (selected language, interface preferences). Session persistence enables features such as resuming interrupted assessments, tracking longitudinal progress across multiple training sessions, generating historical performance analytics, and providing personalized content recommendations based on past interactions.

2.1.3. Cognitive Layer: LLM Integration

The system leverages Google’s Gemini 2.0 Flash model [11] as its core cognitive engine. Gemini 2.0 represents Google’s latest generation of large language models, featuring enhanced instruction following, longer context windows (up to 1 million tokens in extended versions), improved factual accuracy through retrieval-augmented generation capabilities, and multimodal understanding supporting text, images, and structured data. Main advantages:

• Performance: Gemini 2.0 demonstrates state-of-the-art capabilities in instruction following, context retention across long conversations, and response coherence, as validated by independent benchmarks [19].
• Contextual understanding: The model excels at maintaining context across multi-turn dialogues, essential for educational conversations where users build upon previous exchanges.
• Multimodal capabilities: While our current implementation focuses on text processing, Gemini’s native multimodal understanding provides extensibility for future inclusion of image-based training content, video analysis, and diagram interpretation without architectural changes.
• API accessibility: Google provides robust, production-ready API access via the google.generativeai Python library with comprehensive documentation, rate limiting, error handling, and monitoring capabilities.
• Cost-effectiveness: The Flash variant offers an optimal balance between capability and computational cost, making it suitable for prototype validation and potential production deployment at scale.

Integration is achieved through LangChain [20], a framework specifically designed for building LLM-powered applications. LangChain provides essential abstractions including prompt templates for consistent formatting, conversational memory for context management, tool integration for extending LLM capabilities with custom functions, output parsing for structured data extraction, and chain composition for complex multi-step workflows. These abstractions significantly reduce development complexity and improve code maintainability.

Prompt engineering constitutes a critical component of system effectiveness. We developed specialized prompt templates for each major function:

• Information retrieval prompts: Instruct the model to extract relevant information from indexed source documents, present it clearly and concisely, cite specific document sections when appropriate, and offer to clarify or elaborate based on user needs.
• Question generation prompts: Specify question format (True/False, Multiple Choice, Open-ended, Scenario-based), difficulty level (introductory, intermediate, advanced), content scope (specific objective or general knowledge), quality criteria including clarity, relevance, alignment with learning objectives, and pedagogical soundness. Prompts include few-shot examples demonstrating desired question quality.
• Evaluation prompts: Define assessment rubrics explicitly, emphasizing objectivity, fairness, constructive feedback orientation, and identification of specific strengths and weaknesses. For open-ended questions, prompts specify required feedback components (accuracy assessment, completeness analysis, suggestions for improvement) and appropriate feedback tone (encouraging yet honest, specific rather than generic).
• Feedback generation prompts: Guide the model to produce motivational, instructive, or formal feedback depending on interaction context (practice vs. formal assessment), user performance level, and identified learning needs. Prompts emphasize growth mindset principles, particularly within the Practice Drills and Formal Assessment modes, where constructive feedback focuses on effort, improvement, and skill development rather than fixed ability judgments. This approach is especially beneficial in assessment-oriented interactions, as it promotes continuous learning, reduces discouragement, and supports ethical, learner-centered evaluation practices.

All prompts undergo iterative refinement through testing with diverse inputs, with adjustments made to improve output quality, consistency, and alignment with educational best practices.

2.1.4. Infrastructure and Deployment

The TRL-4 prototype is deployed using a hybrid cloud infrastructure designed to balance functionality, security, and cost-effectiveness:

• Google cloud platform: Hosts the Gemini 2.0 Flash model, accessed via RESTful API calls. Google’s infrastructure ensures high availability, low latency, and automatic scaling to handle variable request loads. The model was configured using a temperature of 0.7 and a top_p of 0.95. The temperature parameter controls the randomness of the generated responses, balancing creativity and coherence, while top_p applies nucleus sampling to limit token selection to the most probable candidates, improving response quality and consistency.
• Amazon Web Services (AWS): Hosts the FastAPI backend application, Chainlit interface server, session storage database, and generated content (PDF reports, cached LLM responses). We utilize AWS EC2 instances for compute, S3 for file storage, and RDS for structured session data.
• Development workstation: Provides secure access to the testing interface and development environment for system refinement and debugging during validation.

2.2. Functional Requirements

The system was designed to meet eight core functional requirements (FR1–FR8) derived from comprehensive user needs analysis and domain expert consultation with hotel management and human resources professionals:

FR1: Document processing – The system must accurately extract textual content, images, and structural elements (headings, lists, tables) from PDF documents while preserving logical hierarchy and relationships. Extraction accuracy must exceed 95% for text content and correctly identify 90% of embedded images.

FR2: Function-oriented conversational interface – Users must be able to select from four distinct interaction modes (general inquiries, structured training, practice drills, formal assessment) with clear guidance on each mode’s purpose and functionality. Mode transitions should be seamless, allowing users to switch contexts without losing progress.

FR3: Natural language understanding – The system must interpret user inputs expressed in natural language, recognizing intents, extracting relevant entities, and maintaining conversational context across multiple turns. The system should handle varied phrasings, colloquial expressions, and minor grammatical errors without failure.

FR4: Personalized question generation – The agent must dynamically generate questions adapted to user needs, covering four distinct formats (True/False, Multiple Choice, Open-ended, Scenario-based) and aligning with specified training objectives or general content. Generated questions must demonstrate pedagogical soundness, appropriate difficulty, and clear formulation.

FR5: Automated response evaluation – The system must assess user responses across all question types, providing accurate scoring for closed-ended questions (100% accuracy) and consistent, criterion-based evaluation for open-ended responses. Evaluation turnaround time must not exceed 10 seconds per response to maintain interaction fluidity.

FR6: Constructive feedback generation – The system must generate personalized feedback adapted to interaction context, user performance, and learning needs. Feedback must be specific (referencing particular response elements), constructive (identifying both strengths and areas for improvement), and motivating (encouraging continued learning effort).

FR7: Performance reporting – Upon assessment completion, the system must generate structured PDF reports including executive summary, detailed performance breakdown, visual analytics, complete response history, and personalized recommendations. Reports must be generated within 30 seconds and formatted professionally.

FR8: Session management and data persistence – The system must maintain comprehensive session state, enabling progress tracking, historical performance analysis, and personalized content delivery. All user interactions, assessment results, and generated content must be securely stored with appropriate access controls and data retention policies.

2.3. Validation Methodology: TRL-4 Testing

Technology Readiness Levels (TRLs) provide a standardized framework for assessing technology maturity from basic research (TRL-1) through proven operations (TRL-9) [21]. This study targets TRL-4: "component and/or breadboard validation in laboratory environment." At TRL-4, individual technology components are integrated and tested in a controlled laboratory setting to validate basic functionality and demonstrate technical feasibility, but the system is not yet validated with real end-users in operational environments.

2.3.1. Test Environment

We established a controlled laboratory test environment to ensure reproducibility and systematic evaluation. The environment comprised:

• Hardware: Google Cloud server hosting Gemini 2.0 Flash model API, AWS EC2 instance hosting the pipeline conected with the GitHub repository (automating the project updates), Development workstation for test execution and result documentation (AMD Ryzen 7 6800H (3.20 GHz), 16GB RAM, Windows 11).
• Software: Python 3.11 runtime environment, All required libraries (FastAPI, Chainlit, PyMuPDF, PyPDF2, Pillow, ReportLab, Matplotlib, LangChain, google.generativeai) installed via pip with version pinning for reproducibility, VS Code IDE for development and debugging.
• Data: A comprehensive PDF document containing complete hotel operational protocols, organized by objectives: Presentation and Personal Hygiene, Work Organization and Team Structure, Safety and Prevention, Guest Relations, Room Types and Cleaning Procedures, etc. This document served as the single source of truth for all training content and question generation.

All tests were conducted in a network-isolated environment (no external users) with controlled conditions to ensure reproducibility.

2.3.2. Test Case Design

We developed nine comprehensive functional test cases (TC-1 through TC-9) to systematically validate the core system capabilities identified in the functional requirements:

• TC-1 Protocol-aligned responses: Evaluate the quality of the bot’s responses based on the hotel’s operational protocol (used to get information about the hotel and its operations).
• TC2 Handling of ambiguous questions: Assess how the agent responds to unclear or poorly defined queries.
• TC-3 Response time evaluation: Measure the bot’s efficiency when handling frequent queries.
• TC-4 Guided training on a hotel-related topic: Validate the training module using internal documentation as a reference (used to evaluate the training module).
• TC-5 Employee self-assessment: Verify that the agent can generate different types of assessment questions.
• TC-6 Bot-led evaluation: Ensure the agent can assess employee responses and provide a final evaluation.
• TC-7 Response consistency across different formats: Validate that the agent provides consistent answers when questions are phrased differently.
• TC-8 Management of limitations: Confirm that the agent clearly communicates its lack of access to external system integrations and directs users to contact the service center when necessary.
• TC-9 Evaluation of medium-complexity interactions: Measure the agent’s effectiveness in resolving moderately complex queries.

2.3.3. Evaluation Methodology

Test execution followed a systematic manual evaluation approach: for each test case, a researcher interacted with the system following predefined test procedures, documented all system outputs (responses, generated questions, evaluations, reports), and assessed outputs against expected results using objective criteria. Pass/fail determination was made by consensus of two independent evaluators to reduce subjective bias. For quantitative measures (e.g., question generation success rate, evaluation accuracy), we required 90% or higher success across multiple trials (minimum 10 trials per test case). For qualitative measures (e.g., feedback quality, report coherence), we applied predefined rubrics with clear rating scales.

All test results, including screenshots, generated artifacts, and evaluator notes, were systematically documented in a test report for traceability and future reference.

2.3.4. Expected Results and Success Criteria

Now that the test cases have been designed, we need to determine the expected capabilities of the assistant and the measurable indicators used to evaluate whether the system achieves its intended operational and educational objectives.

• The agent is able to respond in a natural and polite manner, strictly adhering to the documentation, and successfully answering at least 90% of user queries (TC-1).
• The agent requests clarification or additional guidance from the user when queries are ambiguous, asking for more context instead of fabricating answers (TC-2).
• All responses are delivered in under 3 seconds, with at least 95% of responses meeting this response time (TC-3).
• The agent responds in accordance with the documentation and is able to provide educational explanations as well as practical examples. It is expected that employees resolve their doubts in at least 90% of cases (TC-4).
• The agent generates a questionnaire with correct and incorrect answers, producing a complete and coherent evaluation in more than 90% of cases (TC-5).
• The agent evaluates user responses in line with the documentation, providing an assessment that accurately reflects the answers given in more than 90% of cases (TC-6).
• The agent responds in a consistent and coherent manner, with responses including the same key points in more than 80% of cases, even when questions are phrased differently (TC-7).
• The agent clearly states that it does not have access to external integrations and offers an appropriate alternative, responding clearly and without hallucinations in more than 80% of cases (TC-8).
• A maximum of three interactions is required to reach a resolution, achieving a Success@3 rate above 95% (TC-9).

3. Results

This section presents the results of the validation tests conducted on the hotel intelligent assistant. The evaluation covered conversational quality, pedagogical capabilities, system robustness, and operational efficiency. While several test cases reached full compliance with the defined success criteria, specific limitations were identified in response time performance and ambiguity management.

Figure 2. Comparison between the expected success thresholds defined for each test case and the performance obtained during the POC validation.

Figure 2. Comparison between the expected success thresholds defined for each test case and the performance obtained during the POC validation.

3.1. Overall Test Performance

The agent demonstrated strong performance in protocol adherence, guided training, self-assessment generation, and interaction efficiency. Most test cases met or exceeded the expected success thresholds. However, TC-2 (Ambiguity Management) and TC-3 (Response Time Evaluation) revealed notable performance gaps that require further optimization. The following subsections describe the results by functional category.

3.2. User Interaction and Efficiency

This section evaluates the agent’s ability to respond to user queries effectively and efficiently.

3.2.1. Protocol-Aligned Responses and Interaction Efficiency (TC-1 & TC-9)

TC-1 achieved a 100% success rate. The agent consistently provided polite, natural, and human-like responses, strictly adhering to the hotel documentation. The system demonstrated a strong understanding of user preferences and operational context, successfully resolving all evaluated queries without deviations from the defined protocol.

TC-9 also reached a 100% success rate. The agent resolved user queries in fewer than three interactions in all evaluated cases. The results confirm a high level of conversational efficiency, with user intents correctly identified and satisfied without unnecessary follow-up questions.

3.3. Pedagogical and Assessment Capabilities

This section evaluates the agent’s ability to support employee training and evaluation processes.

3.3.1. Guided Training Based on Documentation (TC-4)

TC-4 obtained a 100% success rate. The agent demonstrated a strong ability to understand employee doubts and provide clear, structured explanations aligned with internal documentation. Responses were educational in nature and enabled users to clarify their questions effectively, validating the agent’s role as a guided training tool.

3.3.2. Self-Assessment Generation and Evaluation (TC-5 & TC-6)

TC-5 achieved a 100% success rate. The agent successfully generated self-assessment questionnaires covering the requested topics. The generated evaluations were coherent, well-structured, and aligned with the training material, confirming the system’s ability to autonomously support employee self-evaluation processes.

TC-6 also reached a 100% success rate. The agent accurately evaluated employee responses and produced a final score consistent with the answers provided. The resulting evaluations were meaningful and aligned with the expected outcomes defined in the documentation.

3.4. System Robustness and Consistency

The following subsections evaluate the system’s robustness in maintaining consistent responses across varying input formulations and in appropriately handling operational limitations.

3.4.1. Response Consistency Across Different Formats (TC-7)

In TC-7, the agent achieved a 100% success rate. The system consistently provided the same key informational points when questions were formulated in different ways. Although minor stylistic variations were observed, the core content and procedural accuracy remained unchanged across all tested formulations.

3.4.2. Limitation Management (TC-8)

TC-8 achieved a 80% success rate. In most cases, the agent correctly communicated its lack of access to external systems and avoided hallucinated capabilities. However, in 20% of the evaluated cases, the agent produced responses that suggested unsupported actions, indicating the need for stricter limitation enforcement.

3.5. Operational Constraints and Areas for Improvement

This section highlights key operational challenges identified during testing, particularly regarding ambiguity handling and the system’s response time performance under continuous usage.

3.5.1. Ambiguity Management (TC-2)

TC-2 reached a 90% success rate. The agent generally avoided fabricating answers and instead guided the user or requested additional context when faced with ambiguous queries. Nevertheless, isolated failures were observed where clarification was not explicitly requested, highlighting an opportunity to strengthen disambiguation mechanisms.

3.5.2. Response Time Performance (TC-3)

TC-3 achieved a 50% success rate. While the agent consistently provided correct answers, performance degraded after multiple consecutive queries. From approximately the fifth interaction onward, response times frequently exceeded the defined threshold, revealing a significant limitation in sustained real-time performance under continuous usage.

4. Discussion

The results presented in the previous section indicate that the hotel intelligent assistant demonstrates a high level of technical and functional maturity. These findings confirm the system’s viability as a conversational training and operational support tool within hospitality environments. This discussion interprets the results in light of workforce development needs, operational constraints in hotel settings, practical adoption considerations, and directions for further optimization.

4.1. Interpretation of Core Findings

The validation results reveal strong performance in core conversational, pedagogical, and consistency-related capabilities. Notably, the system achieved 100% success rates in protocol adherence (TC-1), interaction efficiency (TC-9), guided training (TC-4), self-assessment generation (TC-5), automated evaluation (TC-6), and response consistency (TC-7).

These outcomes suggest that current large language model architectures, when grounded in structured organizational documentation, are sufficiently mature to support practical hospitality applications. The agent’s ability to consistently align responses with internal hotel protocols demonstrates reliable retrieval-augmented reasoning and controlled generation, which are critical for minimizing misinformation in operational contexts.

Particularly significant is the system’s pedagogical functionality. The capacity to provide structured explanations (TC-4), generate coherent self-assessments (TC-5), and evaluate employee responses with full alignment to documented criteria (TC-6) reflects a high degree of instructional coherence. Unlike traditional static training materials, the conversational format allows dynamic clarification and contextual adaptation, enhancing the effectiveness of knowledge transfer.

Moreover, achieving 100% consistency across differently formulated queries (TC-7) confirms the robustness of the knowledge grounding mechanism. Minor stylistic variations did not affect informational integrity, indicating stable semantic representation of procedural knowledge.

4.2. Conversational Quality and Efficiency

The system’s performance in conversational interaction is particularly relevant for hospitality environments, where clarity, tone, and efficiency are essential. The 100% success rate in TC-1 demonstrates that the assistant consistently maintained polite, human-like communication aligned with hotel service standards. This is crucial not only for employee-facing applications but also for potential future guest-facing deployments.

Similarly, the 100% success rate in TC-9 highlights strong intent recognition and efficient dialogue management. Resolving queries in fewer than three interactions minimizes cognitive load and supports seamless integration into daily workflows. In operational hotel contexts—where employees often interact with systems during active service—brevity and accuracy are essential.

The high efficiency observed also suggests that the assistant’s conversational architecture successfully balances context retention with targeted response generation, avoiding unnecessary follow-up exchanges.

4.3. Pedagogical Implications for Workforce Development

The assistant’s validated training capabilities address persistent challenges in hotel workforce development, including high employee turnover, limited training time, and variability in instructional quality.

First, automated guided explanations reduce dependence on human trainers for routine clarification tasks. The 100% success in TC-4 confirms that the system can function as an on-demand knowledge mentor, accessible at any time without scheduling constraints.

Second, the self-assessment generation and automated grading capabilities (TC-5 and TC-6) enable continuous competency evaluation without manual supervision. This reduces training overhead while ensuring standardized evaluation criteria. Unlike human evaluators, the system applies identical scoring logic across all employees, eliminating inter-rater variability.

Third, the integration of conversational clarification with structured evaluation supports a more interactive learning cycle. Employees can review explanations immediately after evaluation, reinforcing knowledge retention and encouraging self-directed learning.

Collectively, these capabilities position the assistant as both a knowledge repository and an adaptive training facilitator.

4.4. Analysis of Identified Limitations

Despite strong overall performance, two primary limitations were identified: ambiguity management (TC-2) and response time degradation under sustained interaction (TC-3).

4.4.1. Ambiguity Management (90% Success Rate)

While the agent generally requested clarification when encountering ambiguous queries, isolated failures indicate that disambiguation protocols require reinforcement. In hospitality contexts, ambiguous queries are common, particularly when employees reference informal terminology or partial procedural information.

The observed 10% failure rate suggests that the model occasionally inferred intent prematurely rather than explicitly requesting clarification. Although no critical misinformation was generated, strengthening clarification prompts and implementing stricter ambiguity detection rules would enhance reliability.

4.4.2. Response Time Performance (50% Success Rate)

The most significant limitation concerns sustained response time under consecutive interactions. Although responses remained accurate, performance degradation after approximately five consecutive queries reveals constraints in real-time operational scalability.

In active hotel environments—particularly during peak hours—employees may require rapid sequential queries. Delays beyond acceptable thresholds could reduce usability and adoption willingness. This issue likely reflects computational load, session memory accumulation, or backend processing constraints rather than conceptual model limitations.

Addressing this bottleneck is essential before large-scale deployment.

4.4.3. Limitation Enforcement (80% Success Rate in TC-8)

In 20% of cases, the agent implied capabilities beyond its defined scope, suggesting potential unsupported actions. While not catastrophic, such behavior risks undermining user trust. Implementing stricter capability-boundary reinforcement and system-level guardrails would mitigate hallucinated affordances.

4.5. Operational and Commercial Implications

From an operational perspective, the validated capabilities demonstrate immediate applicability in internal hotel training contexts.

Cost efficiency: Automated training delivery and evaluation reduce dependency on in-person instruction, lowering trainer workload and associated costs. Once deployed, marginal training costs approach zero.

Scalability: A digital assistant can simultaneously support multiple employees across shifts or properties, facilitating standardized training across hotel chains. Nevertheless, for practical large-scale deployment, further optimization would be required to improve response times and maintain performance under high concurrent usage, as identified in TC-3.

Consistency and quality assurance: Uniform application of documented procedures enhances compliance and service quality consistency. The 100% consistency performance (TC-7) is particularly valuable in brand-standard enforcement.

Data-driven insights: Automated evaluations generate structured performance data that can inform managerial decisions, identify knowledge gaps, and optimize training materials.

Compliance and data protection: The proposed architecture also presents favorable conditions regarding regulatory compliance and information security. Since the system is based on controlled internal documentation and operates through managed cloud services (Google Gemini) and LangChain orchestration, the risk of unauthorized data exposure is reduced when appropriate deployment configurations are applied. In addition, the framework can be aligned with GDPR principles by limiting the processing of personal data, controlling data retention policies, and ensuring secure communication through encrypted API interactions. These considerations are particularly important in hospitality environments, where employee records, operational procedures, and internal evaluations may involve sensitive organizational information.

However, response time optimization must be addressed to ensure smooth integration into fast-paced service environments.

4.6. Study Limitations

Several limitations of the current validation must be acknowledged:

• Controlled testing environment: Tests were conducted in structured validation scenarios rather than live operational contexts with real hotel employees.
• Single organizational context: The knowledge base was limited to one hotel documentation set. Generalizability to different hotel categories (luxury, boutique, resort, business) remains to be validated.
• Short-term evaluation: Long-term learning retention, employee satisfaction, and impact on service quality metrics were not assessed.
• Performance Stress Testing Scope: Although response time degradation was identified, broader load-testing under simultaneous multi-user conditions was not conducted.

4.7. Future Research and Development Directions

Several development pathways emerge from the findings:

1.

Performance optimization: Improve backend infrastructure and memory management to maintain response speed under extended usage.

2.

Enhanced ambiguity detection: Implement rule-based clarification triggers or confidence-threshold mechanisms to reduce premature assumptions.

3.

Stricter capability guardrails: Strengthen system-level constraints to prevent unsupported action suggestions.

4.

Operational field validation: Deploy the assistant in a live hotel environment to assess real-world usability, employee acceptance, and sustained performance.

5.

Multimodal expansion: Integrate voice interaction or visual aids for procedures requiring demonstration.

6.

Adaptive learning models: Develop personalized training pathways that adjust difficulty based on employee performance patterns.

7.

Agent-based architecture migration: Replace the current LangChain-based pipeline with an autonomous agent architecture integrating specialized tools for retrieval, reasoning, evaluation, and task execution. This approach would improve modularity, contextual decision-making, scalability, and the ability to dynamically select appropriate tools depending on the user’s intent and interaction type.

4.8. Broader Implications for Hospitality Digital Transformation

The successful validation of this intelligent assistant underscores the growing maturity of AI-driven conversational systems in service industries. Achieving high reliability in protocol adherence, pedagogical guidance, and automated evaluation suggests that AI can move beyond experimental pilots toward structured operational support tools.

In the broader context of hospitality digital transformation, such systems could extend beyond employee training to guest assistance, operational guidance, and decision-support functions. However, sustained performance reliability and strict limitation enforcement remain prerequisites for full-scale adoption.

Overall, the results indicate that conversational AI has reached a level of stability and instructional coherence suitable for structured hospitality training applications, with clearly identifiable technical improvements required to ensure scalable, real-time deployment in dynamic service environments.

5. Conclusions

This study confirms the technical and functional viability of deploying a Large Language Model-powered intelligent assistant to support hotel employee training and operational guidance. The system demonstrated high reliability across conversational quality, pedagogical support, knowledge consistency, and automated evaluation capabilities. The validation results indicate that current AI architectures, when grounded in structured hotel documentation, are sufficiently mature to sustain controlled training environments and structured knowledge assistance tasks.

The assistant achieved 100% success rates in protocol adherence (TC-1), interaction efficiency (TC-9), guided training (TC-4), self-assessment generation (TC-5), automated evaluation (TC-6), and response consistency across different query formulations (TC-7). These findings establish that conversational AI can reliably deliver standardized procedural knowledge, generate coherent assessment instruments, and apply consistent evaluation criteria without human intervention. The system’s ability to maintain semantic stability across differently phrased questions further reinforces its suitability for environments where procedural accuracy and brand-standard compliance are critical.

From an operational perspective, the results demonstrate strong potential for reducing the resource intensity of traditional training models. Automated explanation delivery, self-assessment generation, and grading processes can significantly decrease dependency on dedicated trainers while ensuring objective and uniform evaluation standards. Additionally, the assistant’s conversational efficiency—resolving all validated interactions within three turns—supports seamless integration into daily hotel workflows without disrupting service operations.

However, the validation also identified specific limitations requiring further optimization. Ambiguity management (90% success rate) revealed occasional instances where clarification mechanisms could be strengthened to avoid premature intent inference. Limitation enforcement (80% success rate) indicated the need for stricter guardrails to prevent suggestions of unsupported capabilities. Most notably, sustained response time performance (50% success rate) exposed degradation after multiple consecutive interactions, highlighting scalability constraints that must be addressed prior to full operational deployment.

Importantly, these limitations do not undermine the system’s effectiveness for structured procedural training, which constitutes the primary use case. Instead, they define clear technical targets for improvement, including enhanced ambiguity detection, stronger capability-boundary controls, and backend performance optimization to ensure stable real-time responsiveness under extended usage conditions.

Looking forward, the technology is positioned for advancement toward real-world operational validation. Future deployments should assess employee acceptance, long-term knowledge retention, and measurable impacts on service quality and performance metrics. Broader stress testing under multi-user conditions will also be necessary to confirm scalability across hotel properties and shifts. The current hybrid cloud architecture, which distributes functionalities across Google Cloud Platform and Amazon Web Services, provides sufficient modularity and deployment flexibility to be adapted to alternative infrastructures such as Microsoft Azure, Google Cloud, or fully AWS-based environments. Similarly, the framework is not restricted to Gemini models and could integrate other high-performing LLMs through standardized API-based orchestration, improving portability and reducing vendor dependency.

Overall, this proof-of-concept provides foundational evidence that conversational AI can function as a reliable, scalable, and pedagogically coherent training support system within hospitality contexts. As optimization efforts address the identified performance constraints, intelligent assistants are likely to become integral components of hotel workforce development strategies. The findings suggest that organizations strategically investing in AI-driven training infrastructures may achieve enhanced operational consistency, improved employee preparedness, and sustainable competitive advantages in an increasingly digitalized service landscape.