Chatbot Design and Implementation: Towards an Operational Model for Chatbots

1. Introduction

The chatbot is often referred to as a “conversational interface” or “conversational user interface”, indicating a technology that enables communication between people and information systems (IS), using a human language [1]. While chatbots are usually associated with a text-based communication in a messenger or chat window [2], conversational interfaces can also include voice assistants integrated in smart or wearable devices, smartphones, social robots, autonomous vehicles, and other devices. The Cambridge Dictionary [3] defines a chatbot as a “computer program designed to have a conversation with a human being, especially over the internet”. In practice, modern chatbot platforms vary in behaviour and complexity, and can be delivered as standalone applications or as a service based upon another technology platform.

Over the past decade, the growth in the successful use of chatbots has often been linked to advances in the development of AI technology. Virtual assistants have been developed for a range of different functions, but all use AI technologies and natural language processing (NLP) algorithms [4]. For the past decade, new approaches based on non-linear neural networks have become increasingly evident, and the use of recurrent neural networks (RNNs) has demonstrated significant success in solving typical NLP problems [5]. Bernardini et al. [6] have shown how competitive pressures in global markets and rapid advances in hardware performance - making new technologies increasingly accessible - have led to a significant increase in interest in the potential of AI. In 2018 Devlin et al. [7] introduced a BERT (Bidirectional Encoder Representations from Transformers). This model significantly improved conversational interface performance, and for chatbot developers, BERT provided new opportunities to solve applied NLP tasks typical of most chatbots: user intent recognition, question answering, classification of what? [8].

The subsequent improvement of Large Language Models (LLMs) and related text generative models have had a significant impact on the development of chatbots. The latest and most significant development in the field of NLP is the creation of the GPT (Generative Pre-trained Transformer) model by OpenAI in 2018. It shows excellent results in tasks involving the processing of human language text. The GPT model is trained on a wide range of open text data, including Wikipedia, and is able to generate coherent text of almost indistinguishable quality from human-written text [9]. GPT-4, released in March 2023, is the latest and most advanced version of the GPT family of models to date. It is a multimodal model that works not only with textual information, but also with images, and has pioneering abilities to handle a wide range of real-world tasks performed by humans. Benchmarks show that the model scores higher than the average human on many popular standardised exams in maths, physics, chemistry, statistics, biology and other fields [10]. Today, chatbots and generative AI are widely used by large enterprises as well as small and medium-sized enterprises (SMEs) in many industries. They commonly automate processes in marketing and sales, customer support, human resources, and other areas [11].

Chatbots are attractive to businesses as they facilitate customer service through a conversational style that is natural and intuitive for humans [12]. Despite the rapid development of technology in this area, the increasing capabilities of AI algorithms and the growing number of successful real-world applications, chatbot implementation projects face a number of difficulties. The expectation of quick and immediate success from chatbot implementation that many companies had in the middle of the last decade has been replaced by a more sceptical attitude towards the potential of virtual assistants [13]. Practical experience has shown that chatbots cannot be used effectively in all situations, whilst creating a high-quality service requires significant investment. The technological complexity and unpredictability of project outcomes present additional difficulties for implementation teams, who often lack practical experience and theoretical knowledge [14,15,16].

This article explores the range of issues that confront those involved in chatbot implementation projects. Building upon earlier research by the authors [17], a new model is formulated and presented, which can support IT managers (from large and medium-sized companies), IT integrators and other industry practitioners and specialists involved in the creation and deployment of chatbots. The research identifies common patterns and rules that can be useful for a wide range of organisations whose activities are related to customer service, which can be transformed into a client-chatbot interaction, benefiting both the company and the client. More specifically, this article addresses the following research questions (RQs):

RQ1. What are the main challenges which companies face when implementing chatbot projects?

RQ2. What are the critical success factors (CSFs) that contribute to the successful implementation of chatbots in companies?

RQ3. What operational model can support the chatbot implementation process and help responsible managers to deliver expected chatbot project outcomes?

Following this introduction, the article comprises five further sections. In section 2, the different aspects of the research method are discussed. This is followed by a review of relevant literature and development of the provisional conceptual framework for the study in Section 3. Section 4 then sets out and discusses the research results and addresses the research questions. Finally, the conclusion section considers the limitations of the study, and points up possible future avenues of research in this field.

2. Research Method

2.1. The Case Study Approach and Research Philosophy

The research centres on a case study of a chatbot implementation in one of the world's leading steel producers (presented here as an alias - ESteelCo) that took place in the two-year period March 2021 to March 2023. The ESteelCo case study provided an in-depth inquiry into the complexity of chatbot implementation and was the basis for generation of new knowledge in this subject area. The company’s global strategy has a strong client focus and the company and its clients employ popular instant messengers (WhatsApp, Telegram) as the preferred means of communication. In addition, the company launched a chatbot service, which offers a new modern way to solve clients’ problems. ESteelCo offers its products to a variety of different customer groups, and the launch of the chatbot service provided an immediate response to client enquiries.

The overall research aim was to develop a model based on findings obtained from analysing the chatbot implementation process. The authors adopted an interpretivist philosophy to observe the chatbot integration process within the selected organisation. New perceptions of the key issues were obtained through the recording and analysis of the unique experiences of the people involved in the project. The selected paradigm aligns with the application of qualitative methods of analysis to answer the RQs. The experience of practitioners, personal views, criticisms, and beliefs of individuals involved in the study represent essential data that was recorded and analysed to explore the phenomenon and develop rules for the new chatbot implementation model. An interpretivist philosophy is also appropriate for an in-depth case study of the social phenomenon in an existing company environment [18]. The research process comprised four main phases as depicted in Figure 1. The key aspects of data collection and results validation, particularly important in a qualitative interpretivist case study, are now discussed in more detail below.

2.2. Data Collection

The study adopted a multi-method approach to data collection, employing both survey and observation to collect various sets of qualitative data in the case study company. Yin [18] (p. 99) argues that use of various data sources within a case study can significantly improve the quality of the research data, and here the personal experience of specialists participating in the chatbot implementation project made a significant contribution in terms of variety, completeness, and quality of data. This was complemented by observations by one of the authors who played a role within the project team and had the opportunity to observe the implementation of the chatbot project.

Semi-structured interviews were used, and the list of topics and structure of questions were designed to address RQ1 and RQ2 and validate and develop the provisional conceptual framework, discussed below. This was an exploratory study in which chatbot experts and practitioners responded to a pre-defined list of questions, the answers to which formed the basis for formulating a new model for implementing chatbots. At the same time, respondents had the opportunity in the interviews for wider discussion, to recall their recent experiences, think about why certain events took place, and suggest what could have been done better to integrate the chatbot more effectively with other company systems.

There were 15 respondents/interviewees who took part in the study. Eight participated directly in the chatbot implementation project in ESteelCo, and the other seven were external experts from various banking, IT, retail and pharmaceutical companies in Russia, China, and Germany (Table 1). The interviewees were selected “because they have particular features or characteristics which will enable detailed exploration and understanding of the central themes and puzzles which the researcher wishes to study” [19] (p. 78). They were a combination of experienced internal and external specialists with various tech-nical, industry and cultural backgrounds, which would support the generalisability of the research results and the formulation of a model for managing chatbots projects in other industry contexts. Guest et al. [20] (p. 78) suggested that “a sample of six interviews may have been sufficient to enable development of meaningful themes and useful interpretations”, and Kuzel [21] similarly recommended that, for qualitative research, six to eight interviews is enough for a homogenous sample, but 12 to 20 would be preferable when “trying to achieve maximum variation” (p.41).

In terms of the observations made during the study, a narrative approach was adopted for the recording and presentation of research records, which is characteristic of a qualitative study [22]. One of the authors was involved as a project team member, and had the opportunity to observe the issues relating to chatbot integration and operation through the various project stages, from initial design to product launch and service support. Significant daily events were recorded in the form of short notes, which were subsequently expanded and presented in a narrative form. Although this approach proved effective in providing relevant information and insights, bias in the perspectives taken is inevitable. For this reason, observation is used as a secondary data collection technique, which supplements the findings obtained from the semi-structured interviews.

2.3. Results Validation

Validity and reliability of data and results are important criteria that directly impact the quality of the study. Trochim et al. [23] assume that participants are the ones who can best review research findings and evaluate how much they accurately match their opinions. The level of trust in the results heavily depends on how correctly researchers record and document the views of the people who take part in a study. Participant validation is thus a technique that allows researchers to achieve credibility of research data by involving participants in the process of reviewing results [24]. Here, considerable attention was paid to establishing procedures for verifying the data obtained with the direct participation of the interviewees. Interviews were conducted remotely via videoconferencing using screen sharing functionality, and interviews were recorded. During the interview conversations, the key thoughts and statements of the interviewee were noted down by the interviewer, and in case of doubt, clarification was requested. In addition, the transcript was emailed to the interviewee for final review and corrections. Only after confirming that all opinions were recorded and formulated correctly, was the data used for processing and analysis, thereby minimising possible bias in the interpretation of the findings.

Guba et al. [25] identify dependability and confirmability as two important criteria which measure research reliability. They argue that to some extent generalisation can be achieved by a careful and exhaustive description of all stages of a study (dependability). Description of the research methodology, justification of the chosen methods, a detailed description of the ESteelCo case study, and the process of collecting and analysing data are all integral parts of this research work. All these activities aim to establish the quality of the research, and advance the possibility of reproduction by others. The second criterion, confirmability, is associated with the possibility that research findings and conclusions stated by a researcher can be confirmed by others. To this end five experts were contacted to check the accuracy and validity of the research findings. Of these five, two were from the pool of interviewed participants, whilst the other three were experts in this field of research, but not previously involved in the case study (Table 2).

Triangulation is a well-proven approach which helps to improve the validity and credibility of a study. One of the methods proposed by Patton [26] is associated with multi-method studies and the use of two or more data collection techniques. Use of different independent data sources and application of various methods to analyse them and find the answer to the same research question help to verify and validate qualitative analysis. In this context, the results from the interviews, which directly addressed the RQs, are supplemented by observation. Using two different data collection techniques to answer the same questions helps to examine the problem from different perspectives.

3. Relevant Literature and Provisional Conceptual Framework

This section reports on the findings from an initial scoping literature review that Bell et al. [27] have observed can provide “a means of gaining an initial impression” of relevant themes and that “the narrative review may be more suitable for qualitative or inductive researchers, whose research strategies are based on an interpretative epistemology” (p. 97). The scoping literature review aimed at identifying the key themes from the literature that could provide the basis for the development of a provisional conceptual framework for the research, and identify some of the key literature that would support the generation of responses to the RQs. The scan of relevant literature allowed the discovery of “topical relationships, research trends, and complementary capabilities” [28] (p.351), and the following three sub-sections report on this literature as it relates to the three RQs. Finally, sub-section 3.4 sets out the provisional conceptual framework derived from this literature review.

3.1. Chatbot Implementation Challenges

Despite the potential benefits of chatbots, many companies still find the prospect of chatbot implementation challenging. According to CB Insights [13], high user expectations are not always justified as the technologies and methods for creating chatbots are far from perfect. Srinivasan et al. [29] surveyed more than 300 business executives and IT directors in 12 countries and identified the most prevalent challenges companies face when implementing chatbots: scepticism from users in accepting a new way of communicating; the limited capabilities of chatbots and their inability to provide an acceptably personalised service; and the high number of errors in handling user requests. In addition, the complexity of the technical component of chatbots, and organisational barriers are other hurdles that companies must overcome. Srinivasan et al. [29] also point to the difficulty of addressing data security issues and compliance with local data protection laws, the lack of skilled labour, high integration costs, and the difficulty of platform selection due to high vendor fragmentation. This is confirmed by Kaushal et al. [30], who argue that companies have difficulties with platform selection and integration when implementing chatbots. The majority of chatbot platforms do not provide the necessary functionality for testing and customisation, and the resulting behaviour of the virtual assistant may fail to meet user expectations regarding quality, responsiveness and ability to correctly process a request formulated in human language.

Despite justified criticism from businesses regarding the conversational capabilities of modern chatbots, the results of research groups and demonstrations of the capabilities of chatbots from major technology companies show that the quality of a virtual assistants’ dialogue is almost indistinguishable from that of a human. In practice, however, companies are severely limited in what they can do because many NLP related tasks are challenging [31]. The creation of a perfect interactive assistant requires significant computing and financial resources to collect and process large amounts of data to train the language models, as well as the availability of appropriate specialists [32].

Challenges for chatbot developers are not always directly related to technology, organisational issues, or costs. Even a perfectly implemented chatbot that exploits all the technologies currently available may not be popular with users, some of which may be conservative in attitude and prefer the current status quo. Previous negative experiences with virtual assistants may also discourage potential new users [33].

Fach et al. [14], based on feedback from more than 90 leaders from different organisations worldwide, categorised the main challenges as follows:

• Setup challenges. Activities related to obtaining, preparing and processing of big data, bot training, implementation and launch, support and maintenance.
• User/customer acceptance. Not all users are ready to interact with virtual agents. This situation is expected to gradually improve, so in the long term, such difficulties should not prevent the introduction of new technology.
• Language challenges. Effective communicating with users can be hampered by technical limitations, difficulties in understanding various accents, as well as the ability of developers to deal with natural language processing (NLP) tasks.
• Regulatory restrictions and data security. Both internal company guidelines and national or international regulations (for example, GDPR) can impact system architecture and methods of receiving and storing customer data. This complicates the process of developing virtual assistants in practice and increases the cost of implementation.
• Technology-related challenges need to be addressed. It is advisable to implement technologies from strong vendors that have mature solutions, innovative features, and flexible APIs for integration with other company systems.
• This categorisation overlaps considerably with the perspectives given by Srinivasan et al. [29] and other authors.

3.2. Critical Success Factors for Chatbot Projects

The project management triangle, sometimes called the “iron triangle”, has been used for over 50 years to estimate project success. Time, quality and budget are the three parameters, which determine project objectives and allow the definition of measurable goals [34]. However, the above review of the challenges faced by companies implementing chatbots showed that most of the difficulties and failures are related not only to aspects of project management, but to a wide variety of other factors, these being external or internal to the organisation, and having a technological, social, or other nature. This suggests that the traditional approach to determining the success of a project, exemplified in the parameters of the “iron triangle”, could usefully be complemented by the identification of critical success factors (CSFs) for chatbot projects.

CSFs are well established as an approach to project management. Rockart [35] used CSFs as a method for planning information systems in organisations, and Bullen et al. [36] characterised CSFs as project related areas in which sufficient results will guarantee successful achievement of the project goals. Existing literature, documented case studies and IT integrator guidance available on the internet were analysed to find appropriate CSFs for chatbot projects. These CSFs and their sources are presented in Table 3 and the variety of factors suggests that a range of different subject areas impact the outcome of chatbot projects. These include new technology adoption, resource availability, vendor and platform selection, software licensing and implementation costs, as well as organisational and project management aspects.

3.3. Relevant Models, Methods and Frameworks

A number of technology adoption models are of relevance to chatbot projects. DePietro et al. [52] developed the Technology Organization Environment (TOE) framework, which has been adapted and adopted in a range of technology studies [53,54,55,56,57]. The framework suggests that three main factors influence technology innovation in the corporate environment (Figure 2). First, the technology factor concerns the IT infrastructure, hardware, network communication, software packages and services, available to the company, and the processes that facilitate its deployment. Second, organisational culture and structure, core business processes, and resource availability are seen as key elements in successful transformation. Third, the environmental factor refers to the external market within which the business operates, and the activities of other organisations – competitors, service providers, and government regulators.

Another model from the same era, but equally still of relevance today, is that developed by Earl [58], who set out a multiple methodology for IS strategy formulation. In fact, Earl envisaged three interlocking strategies covering IT, IS and IM (Information Management), but IS is considered here. Earl identified three approaches to developing strategy – top-down, bottom-up and inside-out. The top-down method is an analytical deductive approach that identifies business plans and IS needs. The analysis of CSFs is used via interviews, workshops, or document analysis to establish a connection between business goals and IS requirements. The bottom-up approach assumes that some existing IS have strategic potential and their functionality can support other business activities that provide a competitive advantage. Thus, an audit of current systems is required to recognize which ones need to be updated, what kind of add-ons or extension modules exist, what is the potential for enhancement and further development. Inside-out identifies innovation opportunities and discovers the new strategic potential for the company: investments in R&D, human talent and development of technological culture across the organisation will foster innovation and speed up new technology adoption.

There are a number of stage models used by IS researchers or IT practitioners to aid the planning and analysis of projects. These include the approach developed by the Project Management Institute (PMI), which distinguishes between initiation, planning, execution, and closeout phases [59], and the waterfall model [60] in which each phase is validated by stakeholders before moving on to the next phase. It is irreversible and does not allow the previous phases to be reviewed once they have been completed. The advantage of the waterfall model is its comprehensive approach, which helps to develop a well-integrated and documented system. The involvement of stakeholders at each stage ensures their participation and helps to minimise further complaints. The model is often criticised for its long system development process and the risk that requirements may change during development. This can lead to an implemented system that is out of date by the time it is completed [61].

Nicholas et al. [62] propose a stage model for IS implementation that illustrates, in a simplified form, typical project phases that are also found in other stage models. The four stages are: 1. Conception, in which the drivers of change are identified, feasibility is assessed, and external providers are short-listed; 2. Definition, which includes detailed analysis of the concept, system and functional requirements specification and detailed project planning; 3. Execution, when the IS is implemented, deployed and necessary training and process adjustment is undertaken; and 4. Operation, when system handover to the end-users is finalised, and systems maintenance procedures and service levels are put in place.

Agile methodologies for software development are also of relevance to chatbot projects. This way of developing new software is based on self-organised teams that can swiftly develop and deliver new software features, sometimes termed “product increments” [63]. There are a number of methodologies that provide guidelines for undertaking such development, including Scrum, Kanban, lean software development, and feature-driven development (FDD) [64,65]. The MVP (Minimum Viable Product) approach is based on agile methodology. It can be defined as a “version of a new product and allows teams to collect the maximum amount of validated learning about customers with the least effort” [66]. The main idea of this approach is to build the core functionality of the product, deploy it and get it used by early adopters as soon as possible. Given the complexity involved in AI related projects, agile methodologies are of particular value in that they are based on adaptive processes and strong communication amongst project participants [67].

Chatbot projects are characterised by a high degree of uncertainty in the early stages, unpredictability of the outcome, and the need to study user feedback in order to define new requirements. Many IT integrators support the use of the MVP method to address this uncertainty in their projects. Bansal et al. [68] argue that companies should grow their service from the basic form continuously improving its functionality by adding new features, integrations and skills moving towards the advanced level. This approach is supported by others who argue that the MVP method should encourage teams to design, develop and launch a quick pilot version of a chatbot to understand and assess customer reactions [37,41,52,69]. Once the first version with the core functionality is in operation, continuous improvement and optimisation activities can commence. They usually include customer feedback and behaviour research, AI models training and improvement, planning and adding new features incrementally, and additional user training.

3.5. Provisional Conceptual Framework

The provisional conceptual framework (PCF) for the research draws upon aspects derived from the literature discussed above. The CSF approach is adopted, but the list of factors presented in Table 3 is not seen as a definitive set of tools for the research. Further organization and prioritisation are required, and to this end the TOE framework is used, albeit in a modified manner. The Technology and Organisation categories of the framework are retained, but the Environmental element is excluded from the PCF, as there is little evidence of such factors in the CSFs listed in Table 3, and the likelihood of such aspects influencing chatbot projects is relatively low. However, “User Needs” is seen as a logical third category, which aligns with Brandtzaeg et al.’s [45] assertion that productivity is the most significant chatbot criterion and that this relates to a positive user experience.

Chatbot projects, like many other new IS implementations, undergo certain similar phases. Every project stage can be characterised by a certain set of the most important events and key actions that the team takes. Thus, for each stage there are corresponding CSFs, that positively influence the success of a particular stage, as well as create the prerequisites for subsequently successfully solving these problems. An analysis of the preliminary list of CSFs (Table 3) suggests that many of them are relevant to a particular project stage of a project. Classifying the CSFs according to a particular project stage brings a project-oriented perspective to the PCF, and to this end Nicholas et al.’s [62] four-step project life cycle model is utilised to provide further structure to the PCF.

In summary, the PCF comprises the following elements in an X-Y axis used to categorise CSFs: a modified TOE framework which classifies the various sources that influence chatbot projects using three change categories (Technology, Organisation and User Needs); and project stages, based on Nicholas et al.’s model [62]. The CSFs listed in Table 3 are provisionally allocated to the appropriate cell in the x-y axis as depicted in Table 4. This represents the researchers’ assessment of these CSFs based on existing literature sources. The resultant framework indicates that the CSFs are distributed more or less evenly over the cells of the matrix. This suggests that each of the change categories (Technology, Organisation, User Needs) has an impact on the project at each project stage, and that the identification of new CSFs and the refinement of existing ones can be effectively represented within this framework, which provides the basis for the primary case study research, through which its structure, elements and listed success factors are analysed, refined, expanded and validated.

4. Results and Discussion

This section directly addresses the three RQs drawing on the literature review presented in section 3 and on the primary interview material, using interviewee codes noted in Table 1. Documented notes from researcher observations are also cited, as appropriate, as a further data source.

4.1. RQ1. What Are the Main Challenges Which Companies Face When Implementing Chatbot Projects?

The three data sources noted above were analysed in conjunction to ascertain the key challenges. The challenges derived from an analysis of the literature were categorised and allocated to the appropriate cell in the PCF matrix. Then, the interview findings were analysed to produce a summary list of challenges raised in the interviews, and the individual statements of each interviewee were grouped into more general and universal formulations. In addition, the notes from the observations were similarly scanned and challenges were identified in the narrative text. The challenges identified from these three sources were then combined, via a process of aggregation and re-formulation. The results contain items from three sources, grouped by challenge “category”, as defined in the PCF, and ranked by a “score” value (Table 5).

Kaushal et al. [30], who explore different platforms for creating chatbots, point to the lack of out-of-the-box integration with popular software products, as well as the amount of work required to customise platforms and integrate with enterprise IS. This view was confirmed by many interviewees. Interviewee 1, for example, states that "the amount of work involved in integrating a chatbot with data sources is often underestimated during planning". Interviewee 4 adds that "integrating a chatbot with other business systems is particularly difficult and unpredictable". These statements are also supported by an analysis of meeting minutes and lessons learned during the implementation of the chatbot at ESteelCo, which indicates that the team underestimated the amount of work required to integrate the chatbot with the CRM system, resulting in a significant shift in the project start date. Dealing with complexity and amount of system integration work is thus one of the key challenges (Table 5).

The Score parameter reflects the result of a quantitative assessment of the results obtained. It is calculated as follows: each interviewee is assigned an experience level from 1 to 3, measured in points. Junior employees with little chatbot experience are assigned 1 point, senior specialists or project managers with more chatbot experience get 2 points, and chatbot experts with solid experience in integrating chatbots receive 3 points. The score is calculated as the sum of the points of the interviewee who mentioned the challenge. This provides a way of ranking the challenges so that chatbot implementation teams can better assess project risks and plan appropriate activities.

4.2. RQ2. What Are the Critical Success Factors (CSFs) That Contribute to the Successful Implementation of Chatbots in Companies?

The process of identifying CSFs was similarly based on the principle of triangulation, using three sources of data: the extant literature, the interview data and the observation notes. The critical success factors identified in the literature review were noted in section 3.2 and presented in Table 3.

During the interview sessions interviewees were asked to identify critical success factors for successful chatbot implementation. The interview script structure was entirely consistent with the PCF depicted in Table 4. The respondents named success factors concerning the four project stages (Conception, Definition, Execution, and Operation) and three change categories (Technology, Organisation, and User Needs) and discussed respective actions which help companies to meet their objectives. The process of merging CSFs from the three sources was carried out by finding identical or similar formulations, and by aggregating similar or complementary ideas into more general and universal statements to ensure that one CSF is not repeated in similar formats. This process is shown schematically in Figure 3.

As an illustrative example of this process, a number of literature sources maintained that the use of an agile, iterative approach, and the MVP method, are recommended when implementing chatbots [38,40,41]. This was discussed more broadly by several interviewees. Interviewee 12, for example, recommends minimising the size of the first MVP by "implementing 2-3 of the simplest but most popular scenarios in the first iteration". Interviewee 13 supports this, explaining that "launching a minimum viable product version quickly allows testing of the hypothesis and adjustment of the project roadmap". This view is confirmed by the observation findings and analysis of the Lessons Learned meeting minutes, which took place after the launch of the chatbot. Specifically, the team concluded that "an iterative approach and MVP method can be an effective way to manage chatbot development projects, especially when the expected results and outcomes are not clearly defined". In this way, after processing the information obtained from the analysis of publications, interview transcripts and observations, more general formulations of the CSFs were developed and located in the appropriate cell of the PCF.

This process of combining statements and calculating the Score value uses a similar approach to that described in sub-section 4.1. Here, items that received a score of 3 and below are excluded from the result, as in most cases they represent individual judgements of single respondents that have not been confirmed by other sources. The results are presented in Table 6, using the structure of the PCF. The identified CSFs allow implementation teams to strategically plan and execute chatbot projects and increase the chances of a successful chatbot launch.

4.3. RQ3. What Operational Model can Support the Chatbot Implementation Process and Help Responsible Managers to Deliver Expected Chatbot Project Outcomes?

The PCF played a key role during the practical and analytical phases of the study, helping to shape the data collection and processing activities. The CSFs provide a foundation for building the broader operational model for chatbot project development and implementation. The PCF structure facilitates the allocation of the CSFs into the appropriate cell in the matrix of change categories (Technology, Organisation and User Needs) and project phases (Conception, Definition, Execution, Operation), as depicted in Table 6. This represented an initial operational model, but further reflection and adjustment produced an enhanced, more organised and dynamic operating model with a more compact presentation.

First, a critical evaluation of the initial model suggested that such a large amount of information may prove difficult to work with in practice. The total number of CSFs was 59, far exceeding the typical number of 5-8 items suggested by Rockart [35]. Researchers have used different approaches to organise CSFs into different hierarchical structures and groups, and here grouping and aggregation were used to identify seven main CSFs, while maintaining a longer list of secondary factors.

Second, a number of CSFs that apply across several of the categories and stages of the model were identified, and these were taken out of the tabular structure of the matrix to sit above the others as more generally applicable. This also reflected the view of some interviewees that relating certain CSFs to specific cells in the PCF was problematic, suggesting more general factors to be followed across the stages of the project.

For example, the CSF - 'Focus on chatbot productivity' is listed in the definition phase of the User Needs category, although this factor is more global in nature. Interviewees 2, 3, 4, 6, 7, 8, 9 mentioned this factor as important but were unsure of the exact location of this factor in the PCF. Interviewee 8 explained that “he would like to attribute this CSF to the definition stage because it is important to think about chatbot productivity already when you plan your project and create specification. However, this is not really limited by the Definition stage, but rather a general rule which should be considered by the teams at each stage”. Focusing on chatbot productivity can be justified in almost any action to solve a variety of problems, such as choosing a technology and platform, setting goals, formulating technical requirements, defining functionality, planning the development backlog, testing, etc. So this CSF was removed from within the PCF matrix and relocated outside of it (Figure 4 - transition A).

Similarly, some CSFs were deemed relevant to a particular category, but across all project stages. However, at each stage, this CSF is characterised by specific actions that are relevant to that particular stage. For example, "Use the MVP method and iterative approach" is assigned to the Definition phase, but there are other entries associated with it in all the other project stages (Figure 4 – transition B). The use of MVP and agile methodologies is a recommendation that is relevant throughout the project. As the CSF factor is formulated in a more general way, it no longer corresponds to a specific cell in the two-dimensional matrix and can be taken out of it. At the same time, at each stage in the Organisation line, following this recommendation can lead to specific actions that correspond to the current stage of the project. In this way, these entities can be linked hierarchically - the CSF, which exists in the context of the whole project, and the associated activities, which are derivatives of this CSF and are recommended for execution according to the project stage. This dependency is shown schematically in Figure 5.

This process of aggregating model elements resulted in a reduced list of more general CSFs that fall outside the scope of the tabular structure of the PCF. The remaining CSFs are left within the cell matrix but have been reformulated as local and specific tasks or “Key Actions”. These can be seen as derivatives of the CSFs and respond to the key issues that teams need to address within the specified project stages. They are of value in moving the model from a mainly strategic level to facilitate project execution. Seven CSFs are placed outside the matrix, organised horizontally, and are more general entities which provide a broad conceptual and operational platform upon which the Key Actions can be grounded.

Third, existing literature suggests CSFs can be used to link strategic objectives to operational activities. In the context of IS strategy, Earl [58] put forward a three-level structure in which the CSFs are the element that links the company's strategic goals to tactical planning activities. A similar construct is presented by Bullen et al. [36], in which CSFs link an objective to key activities.

The incorporation of a strategy level based on objectives enhances the operational model, making it more applicable to project planning and control activities in chatbot integration projects. This can be achieved by moving the CSF - "Focus on one business objective" - to a higher level and adjusting its wording. Trying to implement a chatbot in several different business processes at the same time can lead to a lack of focus and insufficient investment in the implementation of each of the planned functions, which significantly reduces the quality of the final result and most likely leads to the failure of the chatbot. Solving a single but important task will significantly improve the likelihood of a successful project launch. In project management, the key metrics that measure the success of the project are usually associated with the project goal or objective. This is reflected in the model, which assumes that the activities to define the project aim, the success indicators and the corresponding metrics are carried out in the Definition phase and fall under the Organisation category. The updated construct resulting from the creation of the new Objectives layer is shown in Table 7.

The new model has thus been developed from the PCF to comprise three conceptual and operational layers: the Objectives layer, the CSFs, and Recommended Key Actions. The top Objectives layer concerns information about strategic objectives of the project and specifies appropriate metrics for assessing achievement of these objectives. The model calls for limiting the functionality of the first version of the chatbot, strictly focusing on one business objective and choosing one or more metrics that match that objective.

The central part of the model is the CSFs layer which contains 6 CSFs that are applicable to different projects and should be discussed by the team in strategic meetings. The Recommended Key Actions (the bottom layer) are derived from the CSFs and should be viewed as a flexible set of tasks that can be modified, removed or added to, depending on the organisational context and specific project requirements. They are allocated to the three change categories that affect chatbot projects: those of a technological nature, organisational aspects, and user needs; and to the four phases, which are typical for most projects: Conception, Definition, Execution, and Operation.

The application of the model facilitates a holistic and comprehensive approach to planning and managing virtual chatbot implementation projects. By linking the strategic level of business objectives, critical success factors and key actions, the new model creates the foundation for successfully building and launching virtual assistants. This was confirmed by experts during the validation process, who confirmed their view that this model can make chatbot projects more manageable and increase the chances of success.

The validation process involving five experts consisted of three steps. First, the experts were provided with a new model for chatbot implementation, along with a detailed description. Next, a discussion of the model took place between the researcher and the experts, focusing on the model's structure, content, and application recommendations. Finally, the experts independently completed a questionnaire evaluating four aspects of the model: evaluation of the model structure, evaluation of the CSFs, evaluation of the Key Actions, and evaluation of the new model in general. Overall, the experts were positive in their support of the model, albeit with some suggested minor modifications which have been incorporated in Table 7.

4.4. Model Application and Guidance

The model is best seen as a flexible tool that can be used alongside established planning methods and techniques. It is most effectively deployed by adopting a top-down approach. The preliminary stage requires the development of the concept and formulation of the project idea. Then the project team are best served by focusing on one business objective, after which time the model can be utilised. In addition to agreeing the key business objective, one or more metrics that directly measure the achievement of the objective should be specified. If this proves difficult to achieve at the outset of the project, metrics can be added or refined later in the Conception stage.

After defining objectives and selecting success metrics, the CSFs and Key Actions layers of the model should be explored sequentially. The six CSFs are general formulations that can be applied in different phases across the duration of projects, so teams are encouraged to discuss them during planning and review sessions to establish a link with the real context of the organisation. By exploring the relevance of each CSF to a particular project, project stakeholders gradually prioritise their requirements and gain a comprehensive understanding of the challenges, issues, risks and opportunities associated with the upcoming chatbot implementation.

The CSFs are reflected and interpreted in the Key Actions layer in the form of recommended tasks and actions. The Key Actions layer is the most detailed yet flexible part of the model, and teams may find it necessary to adapt them to particular project circumstances. For example, the task "Find an expert to join your team" may not be relevant for all teams. At the same time, the specifics of the organisation's business, IT environment, internal rules and procedures may require important specific tasks to be solved that are not listed in the operational model. For example, there may be a number of important tasks related to integration with specific IS, creating new micro-services to implement specific planned bot functions, and preparing necessary data.

From a project lifecycle point of view, it is advisable to use the operating model as early as possible - as soon as the idea of potentially using a chatbot in a company has been raised. The model is designed to help teams with strategic planning and to shape the project's action plan, so its role is most important at the very beginning. However, the project manager and other team members can refer to the model periodically throughout the course of a project. This can help to check how well the current tasks are aligned with the CSFs. If there are significant gaps or problems, or significant changes to the project plan are needed, the operating model can be a useful tool for re-planning. Repeating the previous planning activities, from validating the key objective and reviewing the CSFs to defining the Key Actions, will facilitate the review and update of the project plan.

The model can also be used post-implementation as the final stage in the model (Operation) includes the list of actions and factors that are relevant to project servicing and maintenance. An iteratively evolving chatbot service will return periodically to the Execution phase to develop and release new versions with improved capabilities and features. Using the parts of the model associated with these stages, teams can be given guidance on how to adjust their work and improve their approaches. At the same time, the model can support the on-going management of chatbots as a check-list against current management approaches and project outcomes can be assessed and adjusted as appropriate. A review of the overall objective, metrics, CSFs and Key Actions can facilitate adjustments that improve project organisation and outcomes. The model can help identify and address gaps, solve known problems or improve the performance of an already successful project.

5. Conclusions

5.1. Contribution to the Theory and Practice

The main contribution of this research is the new model for successful chatbot implementation, which is designed to support the development process and deliver the expected outcomes of such projects. This will be of interest to researchers and practitioners involved in the implementation of virtual assistants and other projects related to new technology adoption. It provides answers to common questions about chatbot integration and offers a holistic, universal approach that links strategic business objectives to the practical steps needed to achieve successful chatbot projects.

The design of the model builds upon several widely used models and frameworks, the elements of which are combined into a new structure. This research confirms the universality and wide applicability of the TOE framework [52], as well as the possibility and suitability of making modifications to its structure depending on the context of application. The stage model by Nicholas et al. [62] became a part of the design of the new model, adding a project management perspective to the implementation of chatbots, and illustrating how such models can be used effectively in combination with other models, complementing the design with a process-oriented approach. The top-down approach proposed by Earl [58] has proven its flexibility and applicability in the context of addressing contemporary project planning issues. Finally, the popular CSF method [36] was used as a tool to investigate the factors that determine the success of chatbot implementation projects. This study thus extends the existing knowledge of the models and frameworks mentioned above, and demonstrates the possibilities of their application in a new context, as well as in combination with each other.

Flexibility, versatility and its generic nature are the characteristics of this model, which is designed to support teams from different countries and regions in chatbot development projects. It is relevant to projects across a range of industries, especially those aimed at automating communication between companies and their customers. It is designed for large and medium-sized companies that want to automate routine processes, although the model can also be used in small company projects, in which case managers applying the model are advised to omit redundant elements. The model does not specifically address voice assistants, but teams building voice bots may still find it valuable in considering the specific aspects of voice development.

Although the model may be of most use to project managers responsible for implementing chatbots in organisations, it can also be useful for other team members and stakeholders, including the project sponsor, IT directors and representatives from the contractor companies. IT integrator teams that provide chatbot implementation services can also benefit from its use by improving their processes, and understanding the needs of client organisations and meeting their expectations.

6.2. Limitations and Future Research

Despite the contribution of this research outlined above, it clearly has its limitations. From a methodological viewpoint, the selection of interviewees was limited by the availability of appropriate personnel. However, fifteen respondents took part in the study, representing six industries, three countries and speaking two languages. This diversity allowed the identification of characteristics of chatbot projects that are applicable cross-industry and internationally. Nevertheless, additional interviews could enhance and improve the possibilities for generalisation and wider applicability of the model.

Although the research focused on just one case study, the use of a multi-method and two data collection techniques - interviews and observation - maximised the value of the available data. Further research could include parallel cases studies in different companies in various industry sectors that would provide a richer and more diverse set of research data. The model is generic enough to be configurable and customizable. The Key Actions layer, in particular, should be considered as a preliminary list of broad recommendations that can be significantly expanded and adapted to the particular circumstances of each project. Practical testing of the model in real project implementations can help evolve the model, and the use of an action research strategy would provide an opportunity to study successive cycles of chatbot development, thereby enabling iterative improvement and optimisation. A focus on specific issues such as approaches and methodologies for platform selection and the impact on project outcomes would also be of value.

Finally, it is essential that the framework is regularly reviewed and updated in line with the latest technological advances in the industry to ensure the relevance of the model and its potential application in practice. In recent years, there has been a rapid development of LLMs and their application to solving practical problems, including the creation of chatbots. OpenAI's ChatGPT, for example has shown impressive results since the end of 2022, and many experts predict that this technology will transform entire industries in the near future. The rapid evolution of this technology will require assessment and incorporation of updates and revisions to the model put forward here. ChatGPT can be considered as one of the most advanced NLP tools, but nevertheless, the issues addressed in this research, as well as the new model to aid teams in solving chatbot problems in the context of an organisation, remain relevant today.