Enhancing Multisensory Virtual Reality Environments through Olfactory Stimuli for autobiographical memory retrieval

1. Introduction

Virtual reality is proving to be an important way to transpose people in space and time to provide them with strong sensory and emotional experiences with considerable psychological impact. Multisensory environments were originally proposed in studies of patients with cognitive and behavioral impairments [1] and have provided encouraging results, particularly in facilitating learning. Within this paradigm, it has been proposed that the human brain has evolved to develop, learn, and function optimally in multisensory environments, therefore multisensory training is a better approximation of the natural setting [2].

The more sensory enriched the multisensory virtual environment created, the more realistic the exposure and, consequently, the psychological effects. Several experiments have tested the effectiveness of multisensory VR exposure in alleviating stress through meditation. Mahalil et al. [3] used a 3D Malaysian environment combined with Islamic ritual prayer and soothing sound, while Perhakaran et al. [4] used the Malaysian 3D environment combined with Mozart music, soothing breathing exercises, and relaxing audio sounds. Both experiments found that the multisensory VR conditions induced a better meditative state compared to the traditional meditation procedure. Studying the effectiveness of a more complex multisensory VR method in inducing presence as well as relaxation, Serrano et al. [5] used VR images, odors, and haptic stimuli individually, VR combined with odor, or VR combined with haptic stimuli. They found that all conditions were equally effective in terms of relaxation, but adding haptic stimuli to VR resulted in a better state of presence. Tomasi [6] explored the therapeutic potential of VR combined with smells and concluded that this multisensory VR is an effective way to treat anxiety, stress, and pain in combination with standard psychotherapeutic and pharmacological approaches.

In this paper, we present a VR system together with an odor device for autobiographical memory retrieval. We evaluated the system through a qualitative study. To the best of our knowledge, we are the first from this series of experiments to add olfactory stimulation (thus representing a mixed reality procedure) and offer the participants the opportunity to interact with the virtual environment. This is an advancement to what exists currently and original research elements for our paper. We did quantitative studies in the past regarding the effects of such using odors for triggering the retrieval of autobiographic memories and these experiments found that odors are effective [7,8]. In this paper, we added VR technology to odors and wanted to complement the previous quantitative studies by doing a qualitative study that is explorative (without numbers and statistics) and that dikes more into the opinions and experiences of the participants. This we consider another novelty of the research reported in this paper - as most of the published research for such systems is quantitative in nature.

The paper is organized like follows. In the next section we present the literature review. In Section 3 we describe the VR system. In Section 4 the qualitative evaluative study is presented while in Section 5 we draw the conclusions.

2. Literature Review

Virtual reality in psychology Virtual reality has emerged as a promising tool in therapeutic settings, particularly in the treatment of anxiety disorders, post-traumatic stress disorder (PTSD), and phobias. Studies have shown that exposure therapy using VR can effectively reduce anxiety and PTSD symptoms. For example, e meta-analysis by Botella et al. [9] demonstrated the efficacy of VR exposure therapy in treating various anxiety disorders. It provided controlled and immersive environments for gradual desensitisation, resulting in significant symptom reduction.

VR technology has revolutionised the study of cognitive processes, allowing researchers to investigate memory, spatial navigation, and attention in ecologically valid environments. A study by Smith et al. [10] utilised VR to examine spatial memory and found that participants’ performance in a VR maze closely mirrored real-world spatial skills, highlighting the potential for studying memory and navigation in realistic settings. Developers have focused on creating increasingly realistic and immersive VR environments. High-resolution displays, improved graphics, and 360-degree tracking have contributed to more convincing simulations, enhancing the effectiveness of VR-based interventions [11]. Haptic feedback technology has seen notable improvements, allowing users to experience tactile sensations within virtual environments. This development enhances the realism of VR experiences and can be particularly valuable in therapeutic contexts, providing tactile cues for interactions [12].

Despite its potential, VR in psychology faces several challenges and limitations. Accessibility and cost remain significant barriers, as high-quality VR equipment can be expensive, limiting its availability in both research and clinical settings. Ethical concerns related to informed consent, potential adverse effects, and data privacy are also important considerations [13]. Moreover, the potential psychological effects of prolonged VR exposure require further investigation. Some individuals may experience disorientation, motion sickness, or other negative psychological reactions when using VR technology [14]. Implementing VR in therapy or research often requires specialized knowledge and training. The availability of professionals with expertise in both VR technology and psychology can be limited, particularly in some regions [15]. Despite those limitations, the use of VR in psychology is a new direction of research that is increasingly used in psychology.

2.1. Multisensory VR in Autobiographic Memory Retrieval

Several studies [16,17] have focused on how virtual reality (VR) can be used to facilitate the retrieval of autobiographical memories, which are memories of one’s own life events. One significant finding in this field is the impact of multisensory stimuli in VR environments on autobiographical memory retrieval. Multisensory information in VR seems to contribute differently to the recollection of events.

This multisensory approach aims to create a more immersive and realistic experience, potentially enhancing the effectiveness of memory retrieval:

• Image and sound: the combination of visual and auditory stimuli in VR is quite common and forms the basis of most virtual experiences. Visual elements create a sense of place and context, while auditory can enhance the realism of the environment and trigger memories related to sound or music.
• Odor: the sense of smell is closely linked to memory and emotion in the brain. By including specific odors that are associated with certain memories or experiences, it’s possible to enhance the recall of autobiographical memories. This can be particularly powerful, as scents are known to trigger vivid and emotional memories.
• Movement: Integrating movement by allowing users to physically move in a VR space, adds another layer of realism. This can be especially effective in recalling memories that involve physical activities or were experienced in a specific location where the user moves around.

Combining these sensory inputs in a VR setting creates a more comprehensive and immersive experience, which can be highly effective for autobiographical memory retrieval. The multisensory approach not only enhances the vividness and emotional impact of the recalled memories but also can aid in memory accuracy. As said previously, to the best of our knowledge we are the first from this series of experiments to integrate olfactory stimulation (thus representing a mixed reality procedure) and offers the participants the opportunity to interact with the virtual environment into an integrated VR system – for example they can interact with the VR environment through virtual hands gives them more control, necessary both to reduce stress and to facilitate the immersive effect. Bellow we describe in more detail the different aspects presented above related to systems for autobiographical memory retrieval.

Various deficits and biases of autobiographical memory are a paramount feature of affective disorders, stress related disorders, personality disorders and some personality traits such as neuroticism [18,19,20,21] and for this reason several clinical interventions that can target these memory deficits and biases have been tested. A general consensus is that odors have an important advantage in triggering autobiographical memory. Hence, in a recent study, [22] found that depressed patients recalled more specific autobiographical memories in response to odor cues compared with word cues. These results suggest that the autobiographical memory deficits mentioned may especially be observed when are used verbal triggers while using odors provides a more efficient method for improving autobiographical memory recall in depressed patients. Virtual reality has the advantage of combining immersion with gradual exposure, making it suitable for treating a variety of stress, anxiety, and mood disorders. This method is known as virtual reality exposure-based therapy (VR-EBT). VR-EBT, which involves immersion in a computer-generated virtual environment, facilitates emotional processing and has been considered an alternative to traditional exposure-based therapies for specific phobias and post-traumatic stress disorder (PTSD) for more than 10 years [23]. However, a further review made by Botella et al. [9] concluded that not all studies using VR-EBT followed clinical guidelines for evidence-based interventions in the treatment of PTSD, while in terms of acceptability, few studies evaluated this topic. VR in phobia therapy allows further exposure to phobic stimuli without any real danger, giving the patient more control over the situation. In this case, the feeling of presence has a significant influence on the outcome of the treatment [9].

Another significant application of VR-EBT is the treatment of moderate to mild depressive symptoms by focusing on positive autobiographical memories. One of the hallmarks of depression is a tendency for negative recall and overgeneralization/lack of specificity when recalling neutral memories. Fernandez-Alvarez et al. [16] used VR-EBT to treat depression using image-triggered autobiographical memories. The procedure involved using Google Earth VR to visit a location where a positive event had occurred in the past. The findings indicate that the treatment had a short-term positive impact on their condition, but the effect was not long-lasting and should be used as an auxiliary tool.

Regarding the use of VR technology to train autobiographical memory, it appears that this technology may be a promising tool for approaching Reminiscence Therapy (RT), which is a method used to train autobiographical memory on clinical populations with memory deficits. In such a study, Chapoulie et al. [17] designed a VR system that enables highly realistic image-based presentation of familiar environments in an immersive setting where elderly subjects with dementia are asked to generate memories. The results show that the number of memories generated for a familiar environment is higher than for an unfamiliar environment presented by the same VR system. In another experiment, Schöne et al. [23] compared two groups - one experiencing a virtual reality (VR) and the other watching a 2D video. Two days later, their autobiographical memory was tested. The VR group showed better recall success and delayed reaction times, suggesting that VR experiences become part of the brain’s autobiographical memory network. This was supported by a second analysis that used EEG and found that VR led to effortless recall and access to memories, as well as a sense of presence that aided recall [24].

In this paper we present a VR system together with an odor device for autobiographical memory retrieval. We evaluated the system through a qualitative study. To the best of our knowledge we are the first from this series of experiments to adds olfactory stimulation (thus representing a mixed reality procedure) and offers the participants the opportunity to interact with the virtual environment. As far as we know nobody done this up-to-now. This is an advancement to what exists currently and original research elements for our paper.

3. VR System for Autobiographic Memory Retrieval

In this section we present the VR System for autobiographic memory retrieval. In Section 3.1 we present the olfactory device while in Section #.2 we describe the VR system.

3.1. Enhancing VR Experience through Olfactory Dimension

To enhance the VR experience, one potential strategy involves integrating an olfactory dimension. This innovative approach can significantly improve user engagement, especially when combined with visual stimuli. In order to trigger the autobiographical recall, we used the combination of smell + congruent visual context (for example, an orchard where there is a box of oranges on a table, which when the subject takes them to the nose - using virtual hands - an odor diffuser releases the smell of oranges). As in our previous experiments, we used common, familiar odors [7,8]. It is known from the literature that the recall effect is accentuated by the familiarity of smells and not so much by their pleasant character [25]. The images were constructed to be consistent (or congruent) with the odours.

Figure 1. Odor Dispenser Mechanism.

Figure 1. Odor Dispenser Mechanism.

In order to disperse scents effectively, a dispersing mechanism was employed, distributing 2 separate odors – orange and chamomile.

The odor-dispensing apparatus comprises a recipient housing the odor and a nichrome resistance, which heats up to approximately 100 degrees Celsius, causing the odor to mix with water. A fan positioned above the apparatus draws air, forcing it over the odor, dispersing it through a hole in the recipient. A directing wall inside the recipient ensures air is directed towards the scent. It’s worth noting that natural oils are preferred over chemical oils to avoid potential unsatisfactory results or intoxication when heated.

Powering the device requires a 220V power source to ensure a steady current flow. Fans operating at 12V and 0.1-0.45A, or 1.2-5.4W, are utilized in the experiment. Both the fans and the nichrome resistance are directly connected to the power source. Thus, activating the device involves first turning on the resistance to heat the odor and release steam, followed by activating the fans to blow the steam through the hole.

The device’s effective range is limited to one meter from the human subject, although a larger fan would extend this distance. It’s recommended to begin with a mild scent and progress to a stronger one towards the end of the process. Additionally, it’s advisable to aerate the area after each human subject and replace the water containing the odor.

Figure 2. 3D rendering of the device, made in Fusion.

Figure 2. 3D rendering of the device, made in Fusion.

3.2. Development and Application of VR Technology

The purpose of the Virtual Reality (VR) application is to simulate the interaction with different types of objects (Figure 3). This interaction alongside the device that disperses different scents corresponding to the elements present in VR, is meant to trigger different memories in the subject.

The VR application was developed using Unity Engine with WebXR and Tillia Integration packages. Unity allows the development of VR applications in WebGL format. The WebGL format refers to applications that can run directly in a Chrome or Edge web browser. This choice was made to make the application easily accessible from any type of VR Device like Oculus Quest2, Rift or other devices without any installation required, this in turn eases the testing phase of the project where we have multiple VR devices to work with.

The WebXR library is mandatory for any VR WEBGL project, it has multiple prefabs, scripts and library meant for web implementation such as the “Locomotors” (Figure 4). The movement is handled by two types of locomotion systems, one for continuous movement using the device’s joystick and the other system using the teleportation system alongside a rotation locomotor. These scripts were added using WebXr and Tillia integration. The part that we had to modify from the packages was the default mapping of the controllers and the new inputs required for menu opening. The camera used is a WebXrCamera specific to VR WebGL applications that only runs in web browsers. Another trait of Web VR applications is that they require an https hosting server to work, for this side we used the github.io pages that allow self-hosting of a webpage where we deployed our application.

To manage the scenes in the applications a menu was created with multiple options, the options were added depending on the availability and relevance of the corresponding scent for the other device. Based on the team choice, 10 scenes were implemented for the following objects/environments: oranges, chamomile, grass, roses, hay, lavender, jasmine, mint, apples, and coffee. The menu (Figure 4) can be hidden with the joystick button (A) or appear if it is hidden, and the user wants to change the scene.

The objects in the respective scenes are intractable as they have a rigid body component attached to allow them to be picked up by the user. All the prefabs used for the scenes are organized in different empty objects which populate a list that is used for enabling or disabling the objects when scenes are changed. We can see in Figure 5 the list with the elements attached and the VR controller attached to interact with the menu.

One limitation with web deployment of VR environments is build size, the application should not surpass 100MB in size, as it will lead to failure when trying to load the application in the web browser. This limitation imposed some constrains regarding the size of the assets used in the application, to solve this issue we used low poly assets as we required a high number of objects and couldn’t surpass the size limit, on average an in-game 3D object used by us is 2-7 MB in size. The textures and other terrain related objects are small in size and didn’t impede us. Within those limitations we developed our system.

4. Procedure

Next, we present the evaluative qualitative research results for the system aimed of retrieving autobiographical memory through the use of multi-sensory virtual reality (VR). VR technology can be used to stimulate and record autobiographical memories. By creating a virtual environment that recreates events from a person’s past or scenarios reminiscent of those moments, individuals can recall and relive those experiences in detail. Using VR, one can create a digital representation of an event from a person’s past. This representation can include visual, auditory, and even sensory details to enhance realism and evoke the feelings associated with that memory. VR technology allows users to immerse themselves in a virtual environment, which can amplify the emotions and feelings associated with memories. Through visual and sound effects, as well as direct interaction with the virtual environment, users can re-experience the feelings that accompanied the original event.

A total of five subjects participated in the study, two undergraduate students, Faculty of Engineering in Foreign Languages, National University of Science and Technology Politechnica Bucharest, Romania and 3 master’s students, in Psychology, Spiru Haret University, Bucharest Romania, four females and one male. Snowball sampling was used. Snowball sampling is a method of recruitment in which research participants are asked to help researchers identify other potential subjects. A sample of the questions from the interview that was taken after the experiment is given bellow. The rest of the questions can be easily deducted from the evaluative analysis presented below.

The subjects (Figure 6) were exposed to the virtual environment and the odour associated with it. We used several virtual scenes and odors – for example a garden with a table on which there was a box of oranges and a meadow with chamomile flowers. In the scene with the box of oranges, the subjects could take an orange with their virtual hand and bring it to their nose. At that moment, the scent diffuser was turned on and sent out the odour of oranges. In both scenes (the meadow with flowers of chamomile and the garden with the box of oranges), the subjects could turn their gaze 360 degrees and in the scene with the meadow with flowers of chamomile they can bend down to see the flowers better. In the scene with the meadow with flowers when they leaned over, the scent diffuser turned on and sent out the odour of chamomile, while in the scene with the garden with a table with a box with oranges, when subject pick an orange and bring it to his/her nose, the scent diffuser sent the odour of orange (those are some examples of how the experiment was carried out for those scences).The subjects were instructed to remember an episode from their life, the memory being triggered by the combination of VR environment and odour. When their memory was activated, they had to relive it and, as they relived it, to describe it in detail. They had to say where they were in that episode, how old they were, what season it was, who they were within that scene, how the people were dressed, what they were doing, what emotions they felt when recalling the episode. They were also asked how pleasant or unpleasant the memory was, how vivid and how personally relevant.

5. Results

In studies that aim to trigger autobiographical memories, it is improbable that the veracity of the evoked memories can be verified. Moreover, it is known that the memories we have of the distant past are rarely kept intact. Each time they are activated, they tend to be filled with different information that they are congruent with. Therefore, the purpose of these experiments is rather on the therapeutic aspect of recall and not on the accuracy of updated information. And in our experiment, the focus was on verifying the effectiveness of the VR setting + congruent smell in evoking vivid memories and in aspects related to the emotion of reminiscence

Following the study conducted on the use of VR technology, the participants were interviewed. For this research, it was important for us to know what feelings the participants had before entering the actual study. The feelings were mixed, from curiosity, good humor, and calm, to nostalgia and concern. Curiosity was especially related to the procedure itself, in which VR technologies would be used.

All participants in our pilot study experienced intense emotions related to childhood or adolescence. Chamomile was associated, for example, with the time when the grandparents used to gather plants to store for the winter. Also, the smell of oranges was most often associated with the winter season when, at grandparents’, with parents, or with dear friends, they could enjoy the desired fruits. The feelings were, predominantly, of joy, nostalgia, and less often of sadness, these being associated with the disappearance of loved ones. One of the participants mentioned that it would be ideal if the images within the VR technologies were personalized. In the opinion of this participant, it is considered that the experiences could be much more intense and much more specific.

Olfactory stimuli were essential for awakening memories. These, together with the images that the participants viewed, were a very good combination for autobiographical memory retrieval in the participants of our study. Some participants considered that both olfactory and visual stimuli contributed equally to the reliving of memories. However, one of the participants mentioned that visual stimuli were more powerful than olfactory ones in retrieving autobiographical memory. With the olfactory stimuli, the memories were not very intense, the number given was 3, on a scale of 1 to 7. The memories were rather at the level of reconnection to a specific state of carefree life, joy, playfulness and motor movement. The colors in the presented images, raw green, white and yellow, were of real use, more than the olfactory stimuli. The odours were a little discreet, the memories evoked being, rather, generic than specific. Last but not least, one of the participants mentioned that odours had a much greater impact on autobiographical memory retrieval than visual stimuli. If we have to conclude, we can mention the fact that the opinions of the participants in this study, vis a vis the olfactory and visual stimuli, were different, some considering them equal in terms of intensity but also in terms of importance in recalling memories, others were stimulated by the visual field more, and others by the odours.

A common fact, accepted by all the participants, was the one related to the pleasant feelings they had during the experiment that persisted even after it ended. Those reliving the moments from childhood or adolescence remained in the memory of the participants for quite a long time, the well-being being the characteristic of all. This is similar with the results from other studies. For example, in one study (Chapoulie et al., 2014) [17] a VR system was designed that enabled highly realistic image-based presentation of familiar environments to elderly subjects with dementia, and their autobiographical memory was retrieved.

VR technologies and interaction had a very important role in the experiment, especially through the prism of isolating the visual field and anchoring it in the suggestive images of scenes such as chamomile and oranges. Two participants thought that a set of images, less artificial, would have a better effect than the one so far. The VR technology used in this experiment had no auditory stimuli. Some participants noticed this, others did not. Those who noticed considered that it might be an element worth taking into account for a future experiment.

The participants all agree that the VR system is well suited for autobiographical memory retrieval with further benefits for improving the psychological state of a person. Our results are similar to other similar studies from the literature. For example, in another study the memory performance of two groups of healthy adult subjects was compared, one experiencing a virtual reality (VR) and the other watching a 2D video, and was found that the VR group showed better recall success [16,17]. Finally, a study designed a virtual reminiscence room using 3D modeling and rendering technology to reproduce life scenes from the 1970s of China and exposed patients with Alzheimer’s disease to this virtual environment to test the efficacy of VR in improving autobiographical memory (Xu and Wang, 2020) [26]. However as compared with other studies, according also to the opinion of the participants, integrating odors and the possibility of interacting with the VR environment creates more immersion and has a stronger effect than just taking one technology in isolation. This integration and immersive interactions are novel aspects of the research presented in this paper.

6. Conclusions

The integration of multisensory stimuli in VR environments offers promising avenues for enhancing therapeutic applications and memory retrieval. This interdisciplinary approach combines technology and psychology to foster stress reduction and improvement of memory, underscoring the potential of VR technologies in psychological therapies.

In this paper, we present a VR system together with an odor device for autobiographical memory retrieval. We evaluated the efficacy of the system through a qualitative study. To the best of our knowledge, we are the first from these series of VR-evoked autobiographical memory to add olfactory stimulation (thus representing a mixed reality procedure) and offer the participants the opportunity to interact with the virtual environment. As far as we know nobody done this up to now. This is an advancement to what exists currently and original research elements for our paper.

After the presentation of the integrated VR system we presented a qualitative evaluation of the system. All participants in our pilot study experienced intense emotions related to childhood or adolescence. A common fact, accepted by all the participants, was the one related to the pleasant feelings they had during the experiment that persisted even after it ended. This is similar with the results from other studies. VR technologies combined with congruent odours and interaction had a very important role in the experiment, especially through the prism of isolating the visual field and anchoring it in the suggestive images of scenes such as chamomile and oranges. The participants all agree that the VR system is well suited for autobiographical memory retrieval with further benefits for improving the psychological state of a person.

This experiment did not set out to make a quantitative comparison between different procedures for triggering memories - such as simple VR and multisensory VR. It is proposed as a pilot study to test the effects of a new combination of VR environment + virtual interaction with objects + odours congruent with the images presented in VR and to gather some qualitative data regarding the subjective experience induced by this combination.

As future work, two participants thought that a set of images, less artificial, would have a better effect than the one so far. This is a direction of improvement for future work. The VR technology used in this experiment had no auditory stimuli. Some participants noticed this, others did not. Those who noticed considered that it might be an element worth taking into account for a future experiment.

The research in this paper was done under Romanian Arut Research Grant no. 27/09/10/2023” Intelligent VR system for treating autobiographical/episodic memory deficits”.