Extended Reality and Multimodal Artificial Intelligence for Human Performance: A Review of Current Status and Future Outlook

1. Introduction

Advanced technologies have had a transformative impact on education. With the rapid development of artificial intelligence (AI) and extended reality, fields like learning, training, and human performance sciences are embracing innovative possibilities. When being used for human performance, extended reality–––a collective term for immersive learning technologies such as virtual reality, augmented reality, mixed reality, and the metaverse–––offers simulated, experiential, and in-situ learning experiences for knowledge construction and skills application [1,2,3].

In addition to extended reality, since the introduction of large language models (LLMs; e.g., [14]), a type of AI technologies, researchers’ interests of AI technologies have surged [10,40,41]. The promising use of AI technologies gained wide attention in many different fields, such as health, medicine, and pharmacy [18], business [19], engineering [20], and learning technologies [10,40]. In human performance, studies focusing on the use of applied AI technologies such as web-based chatbots have burgeoned. For example, Mendoza et al. [11] proposed a model for developing chatbots specifically designed for educational purposes. Their model facilitates the creation of chatbots that address the needs of teachers, students, and administrative staff. They highlighted the utility of chatbots in various roles, such as providing information, handling frequently asked questions, offering feedback, and assisting with administrative tasks such as course registration and admission processes. Further, Ma et al. [12] improved chatbot technology by designing and developing ExploreLLM, which employs a schema-based structure rather than the traditional task decomposition method. This alternative approach enabled LLM technology to provide an additional dimension of capability, enhancing its effectiveness in facilitating reasoning and addressing complex problem-solving challenges encountered by users.

Although chatbots are useful and have a low barrier to application, the affordances of AI technologies and extended reality encompass well beyond chatbots alone. These technologies offer a broad range of possibilities, including immersive learning experiences, enhanced data visualization, and more sophisticated forms of human-computer interaction. Specifically, the potential of AI technologies can be significantly expanded through multimodal interactions that incorporate various sensory inputs and outputs, such as audio, images, and video. As vision-language models (VLMs) become more advanced and largely applicable [e.g., [12,13]], these multimodal capabilities enable richer, more interactive experiences, allowing users to engage with content in ways that closely mimic real-world interactions. This evolution in AI and extended reality could not only possibly enhance user experience but also set forth new avenues for innovation in learning technologies. In the design and implementation of learning technologies, multimodal interactions have been sought to support authentic experience [15], embodiment for deep learning [16], and learning analytics for deep insights about learning processes [17].

Research into the use of multimodal AI technologies in extended reality for enhancing learning is still in its infancy. Despite growing interest, the lack of understanding in this area may hinder researchers from designing and implementing effective multimodal AI technologies for learning in extended reality (Figure 1). Several previous reviews have focused on specific, fragmented aspects of the field. For example, Reiners et al. [4] reviewed literature in the intersection of AI and extended reality. The study indicated the applications and integrations of AI and extended reality, but not in multimodality. Rakkolainen et al. [5] reviewed the scope of multimodal interaction technologies in extended reality, but less on education and the design implications. Numerous other reviews on extended reality or the metaverse in education lacked a specific focus on multimodality [6,7] or AI [8,9].

The purpose of this study is to provide insights into the use of multimodal AI technologies in extended reality for learning, training, and performance. We fulfill the purpose through a systematic literature review to explore the current status of the field and offer a future outlook for research. Based on the backgrounds, we asked the following research questions (RQs) in this study to guide our exploration:

• What are the trends of multimodality in AI-supported extended reality for learning, training, and performance?
• What are the goals of multimodality in AI-supported extended reality for learning, training, and performance?
• What are the techniques and approaches to designing multimodality in AI-supported extended reality for learning, training, and performance?
• What are the future directions of multimodality in AI-supported extended reality for learning, training, and performance?

In the following sections, we first provide the definitions of terms, and then present the method used, followed by the results, discussion, and conclusion.

1.1. Backgrounds and Definitions of Terms

Our study inquired into the integration of multimodality within AI-supported extended reality environments, specifically aimed at enhancing human learning, training, and performance outcomes. In this section, we focused on defining the key terms used throughout the paper and providing a comprehensive overview of the core concepts that underpin our study.

1.1.1. Extended Reality

As an umbrella term for various immersive technologies, extended reality blends the physical and virtual worlds, allowing people to live, work, learn, play, and engage in activities within these interconnected environments with varying degrees of immersion [44,45,46,47,48]. Driven by human curiosity, imagination, and the need for immersive technologies, the concept and development of extended reality have evolved through successive generations of progress. The concept and imagination of simulated reality, initially emerging as fiction [22], evolved into the early development of virtual reality technologies [23] and, later, the metaverse [24]. With these different forms of extended reality, Milgram [25] developed a taxonomy of the virtuality continuum for mixed reality, which spans from real environments to fully virtual environments. Specifically, extended reality offers simulated environments that meet the needs of safe training, such as complex or dangerous tasks like flight simulations for pilots, military exercises, and medical procedures. Additionally, it provides exciting entertainment and creative experiences for the gaming industry.

Aside from the virtuality continuum and technological representations of extended reality [25], interactions within the simulation environment are also crucial [3]. These include not only the user’s engagement with the environment and its elements but also social interactions between users facilitated by network technologies.

A comprehensive view of extended reality is embodied in the metaverse. Key aspects of the metaverse, such as user avatars, social interactions, persistence, and decentralization are crucial [26,27,28,61]. Weinberger’s definition of the metaverse offers a visionary conceptualization of the future world with extended reality. Weinberger [27] (p. 13) maintained: “The Metaverse is an interconnected web of ubiquitous virtual worlds partly over- lapping with and enhancing the physical world. These virtual worlds enable users who are represented by avatars to connect and interact with each other, and to experience and consume user-generated content in an immersive, scalable, synchronous, and persistent environment. An economic system provides incentives for contributing to the Metaverse.”

1.1.2. Multimodality

Multimodality, in a broader sense, involves integrating multiple sensory modes of communication and interaction –– such as visual, auditory, and kinesthetic [21,29] –– and has become increasingly relevant in technologies used in everyday situations. By leveraging these diverse sensory modes, multimodality can enhance human reasoning and make transitions between different forms of representation more concrete and meaningful [32].

When being discussed with extended reality, multimodality leverages multiple sensory inputs to create a more immersive and interactive experience. By engaging various senses simultaneously, extended reality environments may provide realistic simulations and interactions. This approach is particularly valuable in educational settings, training programs, and performance enhancement strategies [30,31]. Through the engagement of multiple senses and methods of interaction, multimodal learning environments can potentially cater to diverse learning needs, enhance engagement, and improve the retention of information. For example, a combination of visual aids, interactive simulations, and auditory instructions may create a richer learning experience, leading to more effective knowledge construction and skills development [32,44].

1.1.3. AI in Human Performance

AI in education encompasses a broad spectrum, including its use as a technology to enhance learning, its integration into curricula, and the study of AI literacy to develop the AI competencies of students and stakeholders. More specifically, AI for human performance focused on leveraging AI technologies to augment human capabilities or enhance human performance. The conceptualization of AI has evolved since 1950s, when the term was first coined [33]. It has developed into a variety of applications, including complex decision-making systems [e.g., [34]], data analysis [e.g., [35]] and pattern recognition [36]. When AI is used to enhance or augment human performance, it can take various forms, such as learning analytics that personalize learning [e.g., [37]], diagnostic systems that assist in medical decision-making [e.g., [38]], virtual companions or agents that provide support [e.g., [42]], and autonomous systems that complement human tasks across different domains [e.g., [43]]. These AI technologies can be integrated into the design and development of both extended reality and multimodality to achieve the objectives of these advanced systems. This integration could enhance user experiences and fosters more intuitive interactions across various platforms, allowing designers and developers to create experiential environments that respond dynamically to user inputs and maximize the effectiveness of extended reality applications. [3]. However, the landscape and specific applications of AI in extended reality with multimodality remain ambiguous, and a scoping synthesis is lacking.

Given the emerging nature of research and development in the areas combining AI, extended reality, and multimodality for learning, training, and performance, this paper adopts a broad definition of AI. This broad definition is intended to encompass a wider range of relevant literature, as the field is still developing. Based on the literature [40,41], we defined AI as “any system or algorithm designed to execute tasks that normally necessitate human intelligence, such as learning, reasoning, problem-solving, perception, and language processing.” Moreover, we also embrace a broad definition of human performance in alignment with our scoping purposes. In this paper, human performance includes a wide range of learning, training, human skills/competence improvements (including human behaviors, cognition, psychological performance, emotion, rehabilitation, and collaborative workplace).

2. Method

2.1. Systematic Scoping Review and Machine Learning-based Semi-automatic Approach

Given our purpose of this study, to gain comprehensive insights into the emerging field of multimodal AI technologies in extended reality for human learning, training, and performance, we conducted a systematic scoping review [50]. According to Arksey and O’Malley, a scoping review is particularly valuable in emerging areas of research, as it helps to evaluate the current state of knowledge on the topic, map the existing literature, and understand the development of the field. Additionally, it can be used to identify critical gaps in research, providing a foundation for future studies and guiding the direction of more detailed investigations. We followed standard systematic review procedures outlined in the preferred reporting items for systematic reviews and meta-analysis (PRISMA, [51]) and adapted our approach with semi-automatic techniques using machine learning [52] to gather insights from the searched articles. Specifically, we used text-mining and topic modeling for our purposes. Researchers have argued that integrating text mining and topic modeling [53] in systematic reviews helps to gain preliminary insights and enhance the quality of mapping and synthesis [54].

2.2. Literature Search

We searched relevant databases available and relevant to the field, including PsycINFO (n = 255), Web of Science (n = 787), and EBSCOhost (n = 284). We didn’t restrict our search to specific years or publication date as our goal was to capture the scope of the existing literature. We used the following search terms and keywords in each database, we present the rationale of the search terms and keywords in each category in Table 1:

2.3. Procedures

We first collected the lists of the literature documents. With these documents, we identified duplications in the lists, and we removed them. The resulted comprehensive list of literature was then reviewed by humans following the inclusion and exclusion criteria (Figure 2 showed the procedures). We also conducted the first preliminary text mining and topic modeling with the base of the collected literature to capture the general ideas of the documents based on the keywords and the abstracts and to support preliminary analyses [53,55,56]. We reported the details of the machine learning-based semi-automatic approach for text mining and topic modeling in the following sections. With the pre-screening dataset, we applied inclusion and exclusion criteria (see Table 2 for details). The two major categories are the topic relevancy and article characteristics. Specifically, for topic relevancy, we also excluded instrument validation studies, studies with a sole outcome focus of technology acceptance; because these studies did not fit our purposes of understanding how the technologies are being used for learning, training, and performance. In addition, we excluded serious games, videogames (but not “virtual reality gameplay”) in the current study because we would like to put an emphasis on “extended reality,” in alignment with our discussions in Section 1.1.1. Further, we excluded studies that simply described extended reality as a multimodal experience without specifying the technologies or applications involved (i.e., only described the “extended reality” used as “multimodal”).

2.3.1. Machine Learning-Based Semi-Automatic Approach

The text mining and topic modeling processes were being executed in Python using several packages, including NLTK (Natural Language Toolkit), scikit-learn, and Gensim. We followed standard text-mining and topic modeling procedures to process the documents. First, we removed the special characters and extra spaces. We then converted the keywords in the documents to lowercase to ensure that the variations (e.g., “Metaverse,” “metaverse,” or “METAVERSE”) are treated as the same word, avoiding unnecessary duplicates and noise, as well as improving consistency and tokenization. We then processed “lemmatization,” of which the words were reduced to their base or root form (e.g., “sensing” became “sense”). Next, a Term Frequency-Inverse Document Frequency (TF-IDF) vectorizer was applied, with the number of features capped in 2000 [cf. [59]]. This step transforms the text into numerical values, allowing for the identification of important terms across the corpus. This limit on features ensured the most significant terms, based on their frequency in documents and across the dataset, are included, while reducing dimensionality and improving computational efficiency. Common stop words were excluded during preprocessing to enhance the relevance of selected features. Following the vectorization of the text using TF-IDF, we also created a corpus by converting the documents into a bag-of-words (BoW) representation. With the BoW, a Latent Dirichlet Allocation (LDA) algorithm was applied to uncover latent topics within the document corpus.

To determine the optimal number of topics, we used the coherence measure in “Gensim” (Generalized Similarity Measure) library to calculate the coherence value. The coherence measure evaluates how semantically consistent and meaningful the words within a topic are. A higher coherence value indicates that the words in the topic frequently appear together in the same documents and are more likely to form a coherent topic. The coherence measure of Cv (Coherence value), based on the normalized pointwise mutual information (NPMI) and the cosine similarity, was used by following the pipelines introduced in Röder et al. [57]. We chose Cv measure over other coherence measures such as UMass-coherence or UCI-coherence due to its robust normalization, which provides a consistent and interpretable score across different corpora. In addition, normalization helped mitigate the impact of corpus size and word frequency variations, aligning well with our goal of using it in a semi-automated procedure that incorporates human-in-the-loop analysis. our purpose to use it as semi-automated procedure with humanin-the-loop analysis. The formula for calculating Cv is as follows see [57,58], where $\left(\begin{array}{c}\kappa \\ 2\end{array}\right)$ is the number of unique word pairs in the topic (which is ${\displaystyle \frac{\kappa \left(\kappa -1\right)}{2}}$), $P\left({\omega }_i,{\omega }_j\right)$ is the joint probability of the words ${\omega }_i$ and ${\omega }_j$ co-occurring, $P\left({\omega }_i\right)$ and P(${\omega }_j$) are the individual probabilities of ${\omega }_i$ and ${\omega }_j$, respectively, and ε is used as a smoothing parameter to avoid zero probabilities.

$$Cv={\displaystyle \frac{1}{\left(\begin{array}{c}\kappa \\ 2\end{array}\right)}}{\displaystyle \sum }_{i<j}{\displaystyle \frac{\log \frac{\mathsf{{\rm P}}\left({\omega }_i,{\omega }_j\right)+ϵ}{\mathsf{{\rm P}}\left({\omega }_i\right)·P\left({\omega }_j\right)}}{-\log \mathsf{{\rm P}}\left({\omega }_i,{\omega }_j\right)+ϵ}}$$

(1)

This approach was conducted twice, first with the preliminary, pre-screening dataset (n = 1259), and then with the final included dataset after screening (n = 440). With the Cv coherence score, the optimal number of topics for the pre-screening dataset would be 14 (Coherence = 0. 449), while eight topics and 24 topics yielded the same coherence score of 0.448, we presented the information in Figure 3. For the distribution of dominant topic in the 14-topic solution, please see Figure 5.

Figure 3. Optimal number of topics for the pre-screening dataset.

Figure 3. Optimal number of topics for the pre-screening dataset.

Figure 4. Optimal number of topics for the dataset of final included studies.

Figure 4. Optimal number of topics for the dataset of final included studies.

Figure 5. Distribution of dominant topics across documents from the pre-screening dataset.

Figure 5. Distribution of dominant topics across documents from the pre-screening dataset.

Figure 6. Distribution of dominant topics across documents from the dataset of final included studies.

Figure 6. Distribution of dominant topics across documents from the dataset of final included studies.

2.3.2. Analysis Approach for the Pattern Review

To complement the findings from text mining and topic modeling, we conducted comprehensive human reviews aimed at uncovering nuanced insights and implications. Our pattern review was grounded in the results derived from these computational methods, ensuring a robust analytical framework. We maintained a detailed code sheet that cataloged essential information, including article titles, abstracts, publication years, authors, keywords, and links to the full-text articles. Additionally, we organized the machine learning-classified documents according to the dominant topics identified, specifically adopting the twenty-six-topic solution, the final included studies (see Figure 6).

Throughout our open coding process, patterns related to topical focus emerged and were further refined through iterative rounds of review. This process involved constant comparison [49] between the outputs of the machine learning analyses and the insights gained from human reviews. By integrating these two approaches, we aimed to achieve a richer understanding of the data, ultimately allowing for a more comprehensive exploration of the themes and implications present in the literature.

3. Results

3.1. Insights from Text Mining and Topic Modeling

To determine the optimal number of topics, we used the Cv coherence score described in Section 2.3.1. The optimal number is calculated to be 26 topics (Coherence = 0. 412), closely followed by 22 topics (Coherence = 0. 408) (see Figure 4). With the 26 topics solution, we examined the dominant topics among these 26 topics. Figure 6 showed the distribution of dominant topics across documents. Top ten dominant topics are Topic One, Topic Eleven, Topic Thirty-Six, Topic Sixteen, Topic Six, Topic Twenty-One (they are tied in the number of documents/papers, n_{Topic Six and Twenty-One} = 22), Topic Ten, Topic Seven, Topic Eight (they are tied in the number of documents/papers, n_{Topic Seven and Eight} = 13), and Topic Twenty. In comparison to the pre-screening dataset (Figure 5), the topic modeling in the final included study dataset (Figure 6) resulted in a more distinguished distribution of dominant topics.

We summarized the key words in the top ten topics in Table 3. The keywords of the most dominant topic (i.e., Topic 1) indicated that, in terms of extended reality technologies, most studies focused on “virtual reality,” followed by “augmented reality” (e.g., Topics 11, 26) and “mixed reality” (e.g., Topic 32). While the general term “multimodal” (e.g., Topics 6, 13, 16) was listed, text mining also yielded some keywords more granular in specificity. That is, several keywords provided insights on how “multimodality” was enacted in the included studies, e.g., “haptic,” “visual,” “time,” “assembly,” “sensing,” “eeg,” “emotion,” “brain,” “gaze,” and “eye tracking.” The keywords indicated that the studies utilized various technologies to detect different external signs of human experiences, such as emotions, eye movements, physical interactions, and brain activity. The AI technologies employed were aligned with the use of these multimodal technologies, with related keywords including “vision,” “recognition,” and “robot.” It also revealed that AI technologies were being adopted as a machine learning approach to validate the “accuracy” of evaluation and training results (Topic 21).

To gain further insights into the top five topics, we conducted a correlation analysis between them (see Figure 7). The results indicated that the strongest correlation was between Topics One and Topic Eleven (r = 0.83), followed by Topics Twenty-One and Twenty-Six (r = 0.81). By examining the keywords and content of these topics, we can reason an emphasis on “interaction” and the use of extended reality—particularly virtual reality and augmented reality—to enhance learning through usability study techniques. Additionally, we can observe that the technologies used to capture these interactions varied, including eye tracking, brain tracking, hand tracking with haptic interfaces, and emotion detection.

The number of publications each year reflected the trends of the topic over time. Research on extended reality with multimodal AI technologies for human performance began to emerge around 2002, fluctuated for more than a decade, and experienced a notable upswing in the 2020s (see Figure 8). This increase was likely driven by the rise of LLMs, facilitated by advances in big data availability and computing power, and aligns with recent trends in AI technology development and adoption. It is worth noting that the literature search for this review was conducted in mid-2024, and by that time, 80 papers had already been published that met the inclusion and exclusion criteria, approaching the total of 94 papers published in all of 2023.

Furthermore, we selected the top five topics and illustrated the trends of these dominant topics over the years (see Figure 9). The trends align with the overall trajectory, with Topic One showing a drastic increase starting in 2018. Moreover, initially, Topic Sixteen ranked lower than Topics Twenty-One and Twenty-Six. However, in 2022, Topic Sixteen crossed over to surpass both Topics Twenty-One and Twenty-Six, which experienced a decline. This emerging trend is not yet very pronounced and requires further observation over additional years for validation. It is essential to recognize that Topics Twenty-One and Twenty-Six contains technologies that are more resource-intensive.

3.2. Pattern Review

In addition to the text mining and topic modeling results, we also analyzed the pattern and information from the included studies, with subtle, nuanced, and informative observations from the literature to supplement the findings from text mining and topic modeling. In the following sections, we presented the “Goals and Outcomes,” “ It is interesting to note that several included studies exhibit crosscutting characteristics. For example, the multimodal techniques used are also associated with specific types of AI technologies. Although we discussed

3.2.1. Goals and Outcomes of AI-Supported Multimodal Extended Reality for Human Performance

To understand the goals of multimodality in AI-supported extended reality, we examine the subjects and content areas of the included studies. Specifically, rehabilitation [67,76], medical training [65,78], workforce training (including manufacturing performance) [68,69], language learning [70,77], history (cultural heritage) [71,73] and STEM (including chemistry learning, molecular structures) [74,75] are the subjects identified to be relevant in the areas of multimodality, AI, and extended reality for human performance. This identification highlighted the specific outcomes and areas where AI-supported extended reality can be productively implemented with multimodality.

After mapping these areas, it became evident that the key affordances of this intersection of technologies may lie in enhancing experiential learning, fostering embodiment, and leveraging smart adaptive technology for personalized learning. By integrating these elements, it is possible to create more immersive and tailored learning experiences that respond dynamically to individual needs. In a particular example, “rehabilitation” can greatly benefit from multimodal feedback associated with movement execution. For example, Ferreira dos Santos [76] reviewed the movement visualization techniques in virtual rehabilitation environments for individuals’ movements and motor learning outcomes. The review offers compelling suggestions for using multimodal extended reality in rehabilitation.

3.2.2. Disentangling the Dynamics of User Interactions in Virtual Environments with Multimodal Strategies

There is still a varierty of facets that researchers used to frame “mulitmodality” in VR with AI approaches, in practice, sometimes, multimodality could indeed inlcude “bimodality,” e.g., text and image, or text and video, it is rare for an application to contain more than two modalities simultaneously. An incredible change inlcudes the applications of context-aware, environment-sensitive spatial-sound computing used in extended reality. For example, Rubin and colleauges [65] discussed the use of extended reality, with either typical human anatomy models or tailored patient-specific models for preprocedural planning and education in anaesthesiology. They suggested that “multiuser shared spaces” is an innovative affordances of extended reality, allowing users to engage with holographic avatars in shared virtual environments. This capability, when paired with tools for preprocedural planning, could facilitate collaboration among distant colleagues in ways that traditional video conferencing cannot achieve, calling for the advancement in spatial computing. In a similar vein regarding spatial computing, Kim et al. [66] investigated robot-assisted surgery using a virtual vision platform (VVP) to enhance surgical performance. They found no significant difference in overall performance between the VVP and the conventional stereo viewer (SV) vision system. However, participants demonstrated better performance on one specific metric—the craniovertebral angle—when using the VVP. Additionally, participants expressed a positive view and preference regarding their experience with the VVP.

Indeed, one characteristic of the “metaverse” is the social interactions and collaborations enabled by the technology [26,27,28,61]. Interestingly, although “metaverse” was not on the keyword list revealed by text mining and topic modeling, several studies aim to foster collaborations and social interactions in extended reality through multimodality, using intelligent technologies to enhance human performance. To enhance remote collaborative work, Wang and colleagues [60] designed a virtual reality (VR)-Spatial Augmented Reality (SAR) system to investigate how haptic feedback can be used to enhance collaboration on physical tasks among users in different locations. Another study [62] also explored remote collaboration in mixed reality using multimodal sensing technologies for physical interaction (i.e., gesture and head pointing). Both studies find common ground in the multimodal approaches used in extended reality [60,62]. Specifically, they discussed the established concept of “annotation” in remote multimodal task collaboration, where 2D markers can be transmitted remotely in an extended reality interface. Furthermore, both studies [60,62] built on existing research focused on using “annotation” by suggesting that innovative tangible haptic feedback [60] and gesture- and head-based interactions [62] may better enhance outcomes such as enjoyment, confidence, and user attention in these extended reality environments than the “annotation” approach. [60,62].

In addition to the multimodal interactions enabled by embodied experiences in extended reality, the included studies also utilized multimodal inputs using mouse and keyboard, speech, and image data points [e.g., [63,64]]. Plunk et al. [63] explored the use of semi-supervised machine learning to study interpersonal social behaviors for individuals with autism spectrum disorder in virtual teamwork-based collaborative undertakings. Another multimodal approach involves using audio-tactile cueing to assist patients with spatial attention deficits [78]. This technique is specifically designed to enhance attention in such contexts [78,79]. The combination of audio and tactile cues is provided via a head-mounted display equipped with a cushion featuring six-coin vibrators evenly placed on the forehead, along with over-ear headphones.

3.2.3. Synergistic Multimodality with Emerging AI Technologies Using Machine Learning, LLMs, and VLMs

In addition to the focus on multimodality for the pattern review, the various AI technologies being used in the included studies also emerged as an interesting area of focus. In the current list of studies, AI technologies have been used with machine learning techniques, LLM and VLM architectures. We recognized that VLM represents an important and promising future direction, as techniques from this area can enhance the functionality of AI-supported multimodal extended reality [81,82]. However, its application in the included studies remains limited and is still in the early stages of development.

Machine learning and deep learning are classic and widely observed techniques of AI employed in the included studies. The machine learning techniques utilized in these studies are diverse in their applications, ranging from biodata classification and detection [84,85] to enhancing multimodal embodied interactions and movements [83]. This variety highlighted the versatility of machine learning, demonstrating its capability to address different challenges within the area of AI-supported extended reality. By employing these techniques, researchers developed more sophisticated systems that better understand and respond to human behaviors and interactions in complex environments. More specifically, Rahman [85] used multimodal physiological data (e.g., electroencephalogram and heart rate variability) to train machine learning models for designing virtual reality exposure therapy. This approach allows users to practice responding to anxiety-evoking stimuli in a virtual environment, gradually helping them cope with stress and anxiety. Specifically, the machine learning model can detect anxiety-induced arousal in users, triggering calming activities in the virtual reality setting to assist them in managing and alleviating their distress.

In another study, Alcañiz Raya [83] designed a virtual reality system with machine learning capabilities and a depth sensor camera to capture the body movements of children with autism spectrum disorder. Their results revealed that the machine learning classification accuracy for head, trunk, and feet movements was above eighty-two percent, higher than arms and legs (between sixty and seventy percent). Additionally, the system incorporated visual, auditive, and olfactory stimuli. The visual condition had the highest accuracy rate, followed by the visual-auditive stimuli and the visual-auditive-olfactory stimuli. A unique modality incorporated in this study [83] was olfactory stimuli. Using a wireless freshener with a programmable fan, the system can deliver twelve scents with adjustable duration and intensity. One example of a delivered scent is butter, which represents the smell of a muffin. To realize a fully immersive system using extended reality, the incorporation of olfactory stimuli is a critical area of research and development.

Machine learning and deep learning have laid the groundwork for the emergence of LLMs. For language-related tasks and text-based modality, transformer-based models enhance human performance. In workforce enhancement, Izquierdo-Domenech et al. [80] embedded speech recognition and LLM in augmented reality to process information in the environment (e.g., in a shop) so that expert knowledge can be used to support shop floor operators. This technique is an innovative way for technical documentation in the workforce. Noting that one essential component in these LLM-enabled architectures and applications is the expert knowledge base [64,80].

Another novel study combined multimodal, transformer-based models, and 6G communication technology to improve human experience [86]. Chen et al. [86] created a cross-modal intelligent system in metaverse using technologies such as transformers and graphic neural networks (GNNs). Their design addressed issues including semantic misinterpretation, transmission noise, and illusion creation. Their intelligent cross-modal graph semantic communication approach integrated modalities of text, audio, image, and haptics. These studies [e.g., [83,85,86]] addressed current issues of inauthenticity and aim to enhance human experience in extended reality through multimodal approaches and AI technologies.

Building on the discussion of GNNs, VLMs, though still emerging and relatively rare in our included studies, represent a valuable AI technology that can enhance interactivity and improve the modality experience in extended reality. For the purpose of improving metal additive manufacturing (AM) training in augmented reality, Fan and colleagues [82] integrated different tools to design a VLM-centered digital twin framework. The VLM leveraged was Generative Pretrained Transformer 4 Vision (GPT-4V) [87]. Similar to the training of LLMs, Fan et al. [82] implemented a two-stage training process for their VLM system. They began with pre-training for feature alignment, followed by fine-tuning for “instruction following and knowledge injection” (p. 261) (see Figure 10). In Figure 2, b, CLIP was developed by Radford et al. [88] served as the foundational neural network architecture for computer vision tasks, the use of LLM was based on Chiang’s [89] open-source model Vicuna. Through this integration, Fan et al. [82] aligning text and image features during the VLM training. Specific to their domain (an expert system in manufacturing training) [82], the discipline-specific interactions were made possible by the use of You only Look Once (YOLO) API [90] and Character Region Awareness for Text Detection (CRAFT) [91,92]. Despite presenting this novel digital twin system designed to provide immersive manufacturing training through the integration of VLMs and augmented reality, Fan et al. [82] also highlighted inherent challenges associated with VLMs. These challenges include the advocacy for using video training data instead of static images for dynamic scenarios, as well as VLMs’ limitations in accuracy when decoding 3D content and understanding spatial relationships.

3.2.4. Fostering Engaging, Interactive and Immersive Human Experiences through Ambient Intelligence

Ambient intelligence is another concept emerged in the applications of AI in extended reality for learning [93]. One characteristic of extended reality is the blend of virtual and physical environments, which creates a world with “ambient intelligence.” To elaborate, the idea of "ambient intelligence" relates to environments that are responsive to the presence and needs of users, which can work together with extended reality technologies. While ambient ingelligence is not pervasive yet in the included studies, it has been envisioned, conceptualized, and studied for about a decade now [93,94,95]. Ambient intelligence seamlessly integrates with agents, robotics, and extended reality to create immersive and responsive environments that enhance user experiences. Intelligent agents serve as the cognitive backbone of ambient intelligence, enabling systems to perceive, analyze, and respond to user needs in real time. Robotics adds a physical dimension, allowing for interactive and tangible assistance in various settings, from smart homes to healthcare facilities. Meanwhile, extended reality–––encompassing virtual, augmented reality, and metaverse–––offers new avenues for user engagement, transforming how individuals interact with their surroundings. These technologies create a holistic ecosystem where users can benefit from personalized, context-aware interactions, making everyday environments more intuitive and supportive. This synergy not only enhances efficiency and convenience but also opens up innovative possibilities for learning, collaboration, and human performance. Ambient intelligence constitutes mainly with “sensors, sensor networks, pervasive computing, and AI” [94] (p. 277). In Godwin-Jones’s [93] work, the author discussed ambient intelligence’s role in language learning through the theory of sociomaterialism [96], which explained the interrelationships and entanglments between materials worlds, semiotics, and humans. The author also offered several examples of the devices that can entail ambient intelligence such as smart wristbands, apple watches, cameras, and smart speakers.

Wearable data gloves [69,72] is another interesting use of AI-supported multimodal interactions that can be considered as part of the ambient intelligence devices. Focusing on human–robot handover task performance in collaborative environments, Zou et al. [72] demonstrated the use of wearable data gloves in tandem with augmented reality to teach robot on these tasks (Figure 11). This design and development have the potential to advance human-robot collaboration to improve human performance in extended reality. Likewise, in another study, Hughes and colleagues [69] designed a low-cost wearable glove, called MemGlove, using technologies such as knitted resistive sensing and fluidic pressure sensing architecture to support user sensing (e.g., pose/gesture and heart rate), environmental sensing (e.g., sensing temperature, stiffness, and force), and task classification (e.g., object detection and handwriting). Their innovative architecture (i.e., combining resistive and fluidic sensing in wearable gloves) have implications for complex multimodal interactions in extended reality. In particular, the wearable glove has exciting applications for embodied interactions in immersive virtual environments.

4. Discussion

In this systematic review of current status and future outlook applying a machine learning-based semi-automatic approach, we investigated the scope and landscape of the intersection between multimodality, AI, and extended reality in relation to human performance. We employed text mining and topic modeling to map the current literature in this area. Additionally, we conducted a pattern review to supplement our findings and delve deeper into the goals and techniques of the included studies related to multimodal AI-supported extended reality.

4.1. The Trends and the Goals (RQs 1 and 2)

The trend over the past years, specifically, five years, indicates a noticeable increase in attention and investment in research in AI-supported multimodal extended reality. While versions of extended reality technologies are not new and have been around for generations [22,23,24], the integration of transformer-based AI (i.e., LLM, VLM) and advanced technologies for multimodal enhancement represents a more recent innovation. This surge in attention and investment is possibly further fueled by the global pandemic, which has highlighted the importance of remote immersive technologies, as well as by technological advancements driven by the commercial sector. For instance, Facebook's rebranding as Meta in 2021 [97] underscored this shift. These factors may help explain the growing interest and increased number of publications in this field. This growing trend also align with findings from other systematic reviews focused on subset dimensions such as multimodality [98], metaverse in education [99], and AI in education [40,100]. As the trend continue to grow, researchers and developers should continue to approach AI-supported multimodal extended reality with caution and responsible plan and design [101,102] as these technologies are prone to systematic bias, misuse, or overestimated benefits relative to the risks [102].

Another intriguing finding regarding the goals and outcomes of AI-supported multimodal extended reality is the context, setting, and discipline that scholars and practitioners have focused on. Notably, due to the unique characteristics of multimodality and extended reality, rehabilitation has emerged as a prominent area of application, e.g., [67], alongside the well-established field of medical training, see [40]. Additionally, workforce learning and training, particularly in manufacturing, have seen extensive and growing applications. Interestingly, collaboration has become a notable outcome in workforce learning and training within industry. This finding aligns with the characteristics of the metaverse [44,45,46,47,48], where shared and social experiences are essential in this type of virtual environments. Furthermore, it suggests ongoing research efforts to reimagine the future workplace and human performance.

4.2. The Techniques and Apporaches, and Future Directions (RQs 3 and 4)

The topics analyzed highlighted the scope of multimodal techniques, the types of extended reality in use, and, to some extent, the applications of AI technologies present in the current set of literature. Traditional multimodal techniques were identified through text mining and topic modeling, while the pattern review revealed additional novel techniques. While traditional methods such as eye, hand, and brain tracking, as well as emotion analytics, are valuable, the pattern review also distilled that olfactory stimuli could represent an encouraging future direction for research [83]. The exploration of senses, in addition to eyes and haptics, could enhance extended reality by providing a more immersive and comprehensive experience for human performance.

Several emerging techniques are shaping future directions for the design and development of immersive and comprehensive experiences aimed at enhancing human performance. These techniques fall under the overarching concept of ambient intelligence, leveraging advanced technologies such as environmental sensors and wearable devices [69,72,93]. Since the introduction of computer applications utilizing large language models (e.g., ChatGPT), discourse surrounding generative AI has intensified, spanning different disciplines [e.g., [103,104,105]], particularly in the context of text-based applications. However, the application of AI in extended reality with multimodality is essential for driving radical innovations in how humans live, work, and play in alternative environments [44,45,46,47,48], enhancing performance, well-being, and lived experiences. To this end, the integration of ambient intelligence is pivotal despite the need for tremendous investment in research and development.

Finally, methodologically, the current review demonstrated that a semi-automatic approach, which supplements machine learning analysis with human reviews, offers a promising direction for future research focused on collaboration between humans and machines [52,105,106]. While the accuracy of machine learning for analyzing and reviewing literature is constantly improving, human analysis and reasoning remains critical, as machines have limitations in distinguishing nuanced differences and charting future directions. Furthermore, from a human-centered perspective on the use of machine intelligence, decision-making and future planning and orientation should rest with humans, rather than being replaced by machines or AI [101,107].

5. Conclusions

In closing, this systematic review has provided useful and valuable insights into the intersection of multimodality, artificial intelligence, and extended reality, particularly regarding their implications for human performance. Our findings indicate a significant upward trend in research and investment in AI-supported multimodal extended reality over the past five years, driven by technological advancements and a heightened focus on immersive remote technologies. As demonstrated, the integration of traditional multimodal techniques alongside emerging innovations, such as olfactory stimuli, offers exciting avenues for enhancing user experiences. Moreover, the overarching concept of ambient intelligence plays a critical role in shaping future developments; however, it requires substantial investment in research and development to foster continued innovation while also addressing potential biases and the overestimation of benefits. Methodologically, our use of a semi-automatic approach highlights the importance of collaboration between human and machine analysis in systematic reviews. Considering future possibilities, it is essential for researchers and developers to approach these technologies with caution and responsibility, ensuring that they realize their full potential in transforming how we live, work, and engage in diverse environments.