Enabling Exploratory yet Systematic Investigation of Presence Factors in Virtual Reality: Proposed Methododology, Reserarch Tool Development, and Practical Application

1. Introduction

Immersive virtual reality (IVR) has emerged as a profoundly relevant technology for a wide range of user groups, offering immersive experiences and versatile applications that cater to diverse interests and needs [1,2,3]. When designing virtual environments, immersion and sense of presence emerge as key quality metrics from an HCI perspective [3,4,5,6], in addition to the established paradigm of usability. Immersion is defined by Slater and Wilbur as, "the objective degree to which a system is inclusive1, extensive, surrounding, and vivid2" [6]. In contrast, presence is a subjective experience, being generally defined as the feeling of "being there" [7] in an immersive virtual environment (IVE). A plethora of definitions and operationalization of presence has been developed throughout the last three decades of research. However, its subcomponents can be broadly grouped into five categories: spatial [5,8,9], social [5,9], and self-presence [9], as well as involvement [8,10] and plausibility illusion [5,11]. When measuring presence, subjective methods have become the standard approach, offering several valid and reliable questionnaires (e.g., [8,10,12,13]), while objective methods such as physiological and behavioral measures are less established [5,14,15,16]. Presence enables the measurement of IVEs’ effectiveness across different application contexts [17].

1.1. Motivation

Presence research in IVR is impeded by the abundance of factors that can influence IVR-experiences. From a human-centered perspective, these presence determinants can either be classified as internal (dependent on the user) or external (dependent on the system) [18]. The substantial number of identified factors, contradictory results on their influence and lack of clarity regarding their interaction [14,18,19] make the development of design guidelines for presence particularly difficult.

In this work we develop a methodology of systematic investigation with the goal to make presence factors controllable. Internal factors can only be controlled through purposeful recruitment of participants and rigorous documentation of user characteristics. However, this entails the risk of excluding certain user groups based on cognitive, perceptive, and motoric abilities. As characteristics of the IVR system, external factors are the only ones that can be controlled by technological means. When investigating external factors, confirmatory studies (testing hypotheses) are the de facto standard. Typically, these studies compare different versions of an IVE, [20,21,22] or entirely different IVR applications [23,24,25] addressing one or several specific presence factors. Studies that utilize exploratory approaches, which allow for more open investigation of presence determinants, remain scarce (e.g., individual configurability [26,27,28]). As mentioned, confirmatory presence studies often lead to contradictory results (e.g., for factors like display resolution [18], polygon count [29,30,31], emotion [32,33], VR scenario [34,35,36] and age [37,38,39]). In addition, it is hard to isolate the effect that singular factors may have on presence, since they often manipulate a conglomerate of factors within their experimental conditions. A common example is manipulating both display and interaction characteristics in one condition change [40,41,42,43,44,45]. This also complicates the identification of dependencies and interactions between factors.

We argue that a two-stage procedure that combines exploratory and confirmatory approaches is necessary for a more systematic investigation of presence determinants. The Idea is to (1) establish empirically grounded hypotheses via exploratory investigation, to then (2) verify/falsify these hypotheses by means of comparison. This process is described in detail in Section 4. Also, there is a lack of VR research tools allowing for the conduction of systematic investigations of presence factors as IVR applications are typically developed for one-time use in a single study (see Section 3). Thus, they have a clear context to be applied in and mostly do not allow for generalizable results. Furthermore, existing VR research toolboxes do not focus on presence and its determinants [46].

In this work we create a foundation for the conduction of systematic studies on (external) presence determinants by developing a two-component tool that implements the proposed methodology (see Section 5). The VR tool is designed to (1) enable experimental control over external presence determinants, (2) guide the experimental process, and (3) facilitate data extraction. Finally, our developed methodology and VR tool underwent a formative expert evaluation (N =6) and a practical evaluation via exploratory user studies with N=115 participants in the scope of a research project (see Section 6).

2. Presence Factors in IVR: State of Research

Recent literature and meta-analyses addressing determinants of presence aggregate findings on both immersive and non-immersive virtual reality settings [3,5,14,18]. Here, four general categories of presence factors can be identified. These are display fidelity and interaction fidelity [47], as well as engagement and personal characteristics [14]. To provide an overview of the state of presence research that focuses exclusively on immersive VR, we identified 118 studies3 as part of our own literature research. Inclusion criteria for studies were the use of immersive VR technology (e.g., CAVE, HMD) as well as some form of presence measurement (e.g., questionnaires, behavioral measures). These show that the IVR-focussed state of research largely reflects the four categories of presence factors. IVR studies typically investigate the combined influence of several factors from the categories of display fidelity (N=62), interaction fidelity (N=35), engagement (N=16), and user characteristics (N=24). However, in IVR additional factors arise that do not fit into any of the previous categories. 17 studies considered immersion/inclusiveness (see Section 1), by comparing immersive and non-immersive VR technology (e.g., HMD vs. Desktop, [48]). 13 additional factors were classified as others and were mostly examined in individual studies. These factors encompass content variables like character anthropomorphism [49], nested virtual environments [50], realistic anchors [51], and transitional environments [52], as well as special variables like shared VR experience [53] and indoor/outdoor VR usage [54].

Out of 118 researched studies, 70 investigate exclusively external presence factors, 16 excusively internal ones, and 28 investigate both. Research on external presence factors focuses primarily on visual representation [3,55], with 58 of 118 studies examining the influence of visual representations. Most visual characteristics of IVR belong to the category of display fidelity. To shed light on the limitations of the current state of presence research in IVR, namely the unsystematic character of investigations (multifactorial ambiguity) and contradictory nature of the results, we present display fidelity as an example for external presence factors. Subsequently, we describe user characteristics as an example for internal presence factors.

2.1. Display Fidelity

Display fidelity (also stimulus [5] or physical fidelity [3]) is “the objective degree of exactness with which real-world sensory stimuli are reproduced” by a display system [47]. In general, this definition includes all sensory impressions (esp. visual, auditory, haptic, olfactory, and gustatory). Previous VR studies have focused primarily on visual aspects, some studies on auditory aspects, and very few studies on other modalities [3,55]. A framework of the components of visual display fidelity, based on the state of research, was developed by Bowman and McMahan [55,56]. This includes (1) field of view, FOV; (2) field of regard, FOR; (3) screen size; (4) screen resolution; (5) stereoscopy; (6) head-based rendering; (7) illumination realism; (8) frame rate; and (9) update rate. The focus of this model lies on technical properties of the used hardware. Design related characteristics of VEs are not considered (except for illumination realism). Other authors, however, emphasize (7) illumination realism [57,58] (10) texture realism [57] and (11) environmental details [57,58].

IVR studies examined these aspects in various combinations. Significant increases in presence were observed when using more realistic illumination [59,60], or more realistic textures and more environmental objects [61]. An increase in texture and geometric realism either led to significant increases [29,30,62,63] or no significant differences [31] in presence. No significant difference could be found for increased illumination realism in combination with higher texture resolution [64]. An increased texture realism alone did not lead to a significant difference in presence [65]. Still, texture realism contributed more to the sense of presence than the possibility of locomotion [66]. In a comparative evaluation of different VEs, the one with the highest values for illumination realism, texture realism and environmental details led to the highest sense of presence [23]. So, the enhancement of different factors within the category of (visual) display fidelity, in most cases, led to an increased sense of presence. However, most studies show a confounded study design, leading to a lack of factor isolation.

2.2. User Characteristics

User characteristics encompass all presence factors that can be classified as properties, abilities, or states of the user. This includes personal characteristics, such as Personality, Cognition, Imagination, Concentration, Knowledge, VR/Game Experience [14]. The relation between various user charactersitics and presence remains widely unclear. For instance, both a positive [38,67], and a negative correlation [39] between chronological age and presence were identified. Other studies, however, could not find any significant difference in presence between older and younger subjects [37,68,69,70].

Regarding personality traits, extraversion was either found to be positively [71] or negatively correlated with presence [72,73]. Again others found no such correlation [74]. As an example for abilities, studies scarcely found spatial imagination to be significantly correlated with presence [72,73], whereas in the majority of experiments no influence could be observed [33,75,76,77]. 3D gaming experience of users was found to have no association with presence [33,72], in other instances it was negatively associated [78]. Positive associations were found only in behavioral measures [79] or single questionnaire items [80].

Thus, when it comes to internal presence factors these are often well isolated in the experimental design. However, studies often have conflicting results and are generally not sufficiently comparable to explain these.

3. VR Study Tools: State of the Art

Existing VR research toolboxes do not focus on presence and its determinants [46]. In presence research, IVR applications are typically developed for one-time use in a single study (e.g., [62,81,82,83,84,85,86,87,88,89]), limiting their customizability and reusability. Here, experimental conditions are realized either by several separate builds4 or scenes within the respective engine or alternatively by changing configurations directly in the engine’s editor. While the former might be resource intensive the latter might introduce the risk of configuration errors by the experimenter. Game engines, such as the Unity Engine (e.g., [22,40,82,83,84,85,86,87,88,89,90,91,92]) and Unreal Engine (e.g., [23,62,81]) are the most widely used development software for creating such study environments [46]. Others utilize modified [93,94] or unmodified versions [25,33,95,96,97] of proprietary VR applications such as videogames. In addition to the previously mentioned reusability and customizability issues, this approach might also reduce control over presence factors. Furthermore, the VR application is usually decoupled from the rest of the experimental process (e.g., randomization of experimental conditions, tutorials, questionnaires). Therefore, we propose an external interface for experimenters covering the experimental process by streamlining its steps (see Section 5).

Focusing on tools for the general study of human factors in VR, Woelfel et al. [46] analyzed the current state of technology. The authors summarized the full range of feature sets that contemporary tools can provide. Study tools typically only satisfy a certain subset of features, that is beneficial to their respective application context or research domain. (1) Setup & Control refers to the ability of a study tool to facilitate experiment planning, customization, and real-time intervention. A tool should provide process planning before and operator control during the experiment, sample scenes should be available and customizable. Furthermore remote testing (outside the laboratory) and human or agent-controlled social agents should be available. (2) Sensing Participants refers to the variety of sensors provided by VR hardware as a rich source of information (based on informed legal consent) enabling the development of a responsive virtual environment. This encompasses the recognition of controller-input, the tracking of head-orientation, body pose, eye gaze, facial features, brain activities and other biosignals (e.g heart and respiration rate), audio and video recording, as well as the administration of in-VR questionnaires. (3) Representation of the virtual environment and its entities focuses primarily on visual realism and interaction possibilities. This comprises representing virtual entities (e.g., surrounding objects and agents), as well as real entities (e.g, participants, experimenter, physical objects) within the VE. (4) Data Handling refers to the reliability, validity, and transparency of data collection, processing and management by the study tool. This includes the possibility to import, export and stream data, the automation of analyses (e.g., behavour/emotion recognition), the possibility to replay and annotate VR scenes, and to share the experimental environment for replication purposes. (5) Integration addresses the question of how much of the tool’s source code is available open source and how much is proprietary code of the underlying engines (e.g., Unity, Unreal).

Our goal is to develop a VR tool specifically dedicated to investigating presence factors which, based on our literature research, does not exist to date. For this purpose, we developed a research methodology, which is presented in the following paragraph. The development of the modular VR tool based on this methodology is described in Section 5.

4. Research Methodology

As shown by the literature (Section 2), the current state of research does not allow for a well-founded formulation of hypotheses on the relationship between presence and the plethora of internal and external factors. As a direct deduction from this state and as a basis for our research methodology, we propose the following measures for the systematization of presence studies. (M1) Presence factors should be atomic in their operationalization, as far as possible (e.g., breaking down the factor illumination realism into illumination intensity, contrast ratio, lighting algorithm, etc.). (M2) The same presence factors must be investigated analogously in different IVR scenarios, to enhance generalizability and eliminate one possible source of contradictory results. When it comes to control and manipulation of presence factors within an experimental setting, we build on the existing classifications by Felton et al. [18] and Souza et al. [14]. Internal Factors are dependent on the user [18], thus we focus on all abilities, properties, and states of IVR users with potential relevance for presence. External factors must be divided into the ones that cannot be controlled by technological means (we call these environmental factors), and the ones that can (we call these technological factors). Environmental factors include ambient characteristics, namely real world stimuli and the physical surrounding. Technological factors include hardware and software characteristics, namely display fidelity, interaction fidelity and content factors (see Figure 1). Our proposed measures in this regard are the following. (M3) external factors should be controlled by technological means, as far as possible (in our case this refers to the previously defined "technological factors"). Thus, independent variables should be adjustable via the study tool used and all other technological factors should be represented via reasonable default values (derived from the current state of research and capabilities of the technology). External factors relating to the environment should be regulated within a controlled laboratory setting. (M4) Internal factors, i.e., user charactersitics should be controlled through rigorous documentation. Purposeful recruitment by diversifying or narrowing the scope of participant acquisition can help to validify the subject sample within the study context.

Our Methodology prioritizes technological presence factors since (A) they offer the highest degree of experimental controllability, and (B) they hold the greatest practical relevance for designers of virtual environments. In later stages, the methodology’s focus could shift toward human or environmental factors; for now, these are primarily well-documented and controlled rather than manipulated. Thus, for our study design we chose a two-stage procedure consisting of an initial exploratory study followed by a confirmatory study. The exploratory approach is essential to enable the derivation of hypotheses based on the data obtained. In the subsequent confirmatory study these hypotheses are then tested via systematic manipulation of these presence factors, examining them in paired comparisons, enabling the derivation of design guidelines (see Figure 1). This two step procedure is detailed in the following.

4.1. Exploratory Assessment of Preferred Configurations of Presence Factors

An exploratory study design is chosen as the basis for hypothesis formation. We define $T=\{t_1,...t_n\}$ as the set of technological factors, and p as the number of independent variables (IV). Thus, in the exploratory study we investigate possible combinations $t_1\times t_2\times ...t_p$ in order to find the optimal configurations to achieve the highest presence (see Figure 1). Based on the approach by Slater et al. [27], this optimum is to be determined by the subjects themselves. However, since this task is potentially complex, especially for subjects unfamiliar with IVR, the configurations will be made by the experimenter in a guided process similar to subjective refraction5. To identify an appropriate subset of T to be used as IV in a study, the trade of between comprehensiveness (considering as many factors as possible) and complexity must be taken into account. In our first practical application of the methodology, we chose $p=2$ as the minimal reproducible example, and conducted two studies with different pairs of IV (see Section 6 and Figure 2). The general procedure of such an exploratory study is described below.

Before the IVR experiment, control over presence factors that are not considered as IV must be ensured. The laboratory environment is set up in order to provide stable conditions in terms of real world stimuli and physical surroundings. User characteristics with potential relevance for the IV are assessed via suitable instruments (for an example of how to identify and document user characteristics, see Section 6.2.1). Technological factors that are no IV are set to their respective default values. At the beginning of the IVR experiment, subjects are introduced to the concept of presence to ensure correct understanding of this term. Then subjects are informed that they will repeatedly experience two different versions of an IVE. Each time they will be asked to indicate in which of the two they feel more present. A short narrative introduction to the respective IVR scenario is given. Subjects put on the HMD, and the IVR application is started. The experimenter increments one presence factor at a time, starting at the lowest possible value. The IV $t_1,...t_p$ are manipulated in randomized order to compensate for sequence effects. For instance, the experimenter might start by presenting the IVE with $t_1$ at level 0 vs. level 1, while $t_2,...t_p$ stay at their default values. If subjects state that they feel more present at level 1, the experimenter will increase $t_1$ presenting level 1 vs. level 2. This process is repeated until the lower value is preferred or subjects state, that they do not recognize any differences between variations or if the maximum value is reached. After that, the same configuration is carried out for $t_2$ and all other IV up to $t_p$. Subsequently a refinement step is performed starting again with $t_1$ moving up to $t_p$ varying all IV in a smaller range. This is intended to address a possible interdependence of presence factors that may not have become apparent from the default values. After this, the configuration is finished and all values are saved. With the resulting configuration, the subjects must then perform an IVR task involving locomotion in the IVE and interaction with virtual objects. As a control measure, all presence subcomponents are assessed after IVR exposure using a short scale (see Appendix, Table A1) based on established presence questionnaires (e.g., PQ [10], IPQ [8], SUS [12], MPS [13]).

To enable the derivation of hypotheses for subsequent studies, the resulting individual configurations $t_1\times t_2\times ...t_p$ are clustered. In the scope of possible hypotheses, all possible cases must be covered. These are (H1) that clusters found relate to certain user characteristics, indicating a specific relationship between these user characteristics and the optimal configuration for presence. (H2) that clusters found are independent of user characteristics, representing different patterns of experiencing presence. It is also possible that no clusters are found at all, suggesting either (H3) a universal optimal configuration for presence or that (H4) optimal configurations can only be predicted for each user individually.

4.2. Confirmatory Study on the Combined Influence of Presence Factors

To test the developed hypotheses, a confirmatory study will be carried out. This study forms the empirical basis for deriving presence-related design guidelines for IVR Design. Measures to control environment factors and non-IV technological factors as well as the assessment of user characteristics will be performed analogous to the previous study.

In this comparative study, presence factors $t_1,...t_p$ are systematically varied in several conditions, which are predefined depending on the hypothesis derived from the results of the preceding study. Thus, the hypothesized optimal configurations $O_{cluster}(t_1,...t_p)$ for either (H1) each profile of user characteristics identified by the cluster analyses, or for (H2) each presence experience pattern identified by the cluster analyses, are tested against each other. Alternatively, the hypnothesized (H3) universal optimal configuration $O_{uni}(t_1,...t_p)$ or (H4) optimal configuration for each individual user $O_{user}(t_1,...t_p)$ is tested against a certain number of hypothesized suboptimal configurations $S_i(t_1,...t_p)$ that derivate strongly from it. In the IVE the same IVR task as in the exploratory study is carried out once for each experimental condition (randomized order). After each condition presence is recorded in detail via established presence questionnaires (e.g., PQ [10], IPQ [8], SUS [12], MPS [13]).

The data of the main studies will be examined having the experimental conditions (configurations of $t_1,...t_p$) as within-subject factor, and in the case of H1 or H2 user characteristics or experience patterns as between-subject factors. The analysis addresses the research question of, how the examined presence factors differ regarding their effect on presence in IVEs. For this purpose all five subcomponents of presence, namely spatial, social, and self-presence, involvement, and plausibility illusion (see Section 1) will be analyzed via the subscales of PQ [10], IPQ [8], SUS [12], and MPS [13]. This analysis is used to derive design recommendations for maximizing presence in IVR, which are either aimed at specific user groups or provide universal solutions.

5. IVR Study Tool Development

To enable the conduction of the planned studies (see Section 4), our study tool requires the following central functionalities: (1) manipulation of IV (technological presence factors), (2) support of different IVR scenarios, (3) guidance of the experimental process, (4) IVR interaction techniques for certain tasks and respective tutorials, (5) data formatting and extraction. Furthermore, the system needs to be modular and expandable, especially in terms of manipulable IVs. Our tool was developed in the context of the AgeVR6 research project, which focuses on the influence of age-related user characteristics and visual display fidelity on presence. Accordingly, the four exemplary IV that were implemented are in the category of visual display fidelity, namely illumination intensity (I), contrast ratio (C), objects density (O), and texture resolution (T). The proposed study process (see Section 4) and an analysis of the state-of-the-art features of industrially produced IVEs form the basis of our development process. For the industrial state-of-the-art analysis ten contemporary IVR games7 (published 2018-2022) were investigated focusing on visual fidelity and its configurability, but also to identify industry-standard baselines for auditive and haptic fidelity. Most of the configurable visual fidelity aspects are covered by the four IV (I, C, O, T), for the remaining aspects like anti-aliasing or color correction common default values were chosen as baselines. The development process resulted in two main components: an IVR application and an experimenter interface (see Figure 3), which are described below. A formative evaluation with six field experts was conducted at the end of the first iteration (see Section 6.1). Based on this feedback, the study system was refined (see Section 4.1). Both components of the tool have been made freely available on github.com8 as an open source template that can be used to create custom IVR studies.

5.1. IVR Application

The IVR application is powered by Unreal Engine 5 including the virtualized geometry system Nanite9. It displays the IVR scenarios with which subjects are presented and can interact during the experiment. The IVR application listens for instructions provided by the experimenter through the web interface. These instructions include e.g., manipulating the IV and loading new IVE. The subjects’ data (IV configurations, task completion scores and times) is captured during and exported after the experiment by the IVR application. Unreal Engine 5 was chosen, because its one of the leading engines used for IVR research (next to Unity, see Section 3), enables high-fidelity graphics and physics, its low-level entry point for novice developers (through constructs such as Blueprints), and its modularity and expandability.

5.1.1. Base System

To support the creation of adaptable scenarios, a series of systems were implemented using the Blueprint10 system provided by the Unreal Engine. The manipulation of the IV (which are referred to as Adaptation Dimensions (AD) within the tool) is handled by the Adaptation Dimension Manager (ADManager). This is achieved through either direct control of the player camera contrast (C), scaling the intensity of all light sources (I) in the scene, the execution of console commands to manipulate the resolution of all textures11 (T) or toggling the visibility of objects in the scene (O). The latter was implemented by extending the existing Unreal Engine construct of the Actor, creating the so called Adaptation Dimension Actor (ADActor), which allows the systematic manipulation of this Actor’s visibility. When an ADActor is placed in the scene, it is randomly determined at which degree of the AD (O) it becomes visible. At the maximum degree of O, all ADActors are visible. To address more complex object compositions, e.g., involving gravity (objects placed on top of each other), the ADComplexActor was created as an extension to the ADActor. It allows the developer to group multiple objects together and predetermine a specific order in which the individual objects appear (with the range of AD (O) degrees still being determined randomly). Figure 4 shows a bookshelf implemented as a ComplexADActor. In this example, at the AD degree of $O=5$ (left) there is only one book visible, with each degree more books get added, these can lay or lean on previous books (middle), at the degree of 16 all books are visible (right).

Another basic functionality is the management of the IVR task. The collective of task related objects (TaskObjects) gets managed by the TaskManager Blueprint. At its current state the IVR task consists of a simple search task in which the participants have to move through the IVE and search for and pick up certain objects. Thereby, the level of presence developed by subjects, is based on an interactive experience of the IVE. The teleportation-based locomotion and the interaction with TaskObjects, were implemented using native functions of the Unreal Engine. The completion of (sub)tasks (indivitual TaskObjects) is assessed automatically by the system. For complex tasks monitored by the experimenter a checklist template is provided via the interface.

5.1.2. IVR Scenarios

We developed three types of IVE for the study tool: A sample IVE, a tutorial IVE, and several specific study IVEs. The Sample IVE serves as a template for the creation of new experimental IVE. It contains sample versions of the ADActor, ComplexADActor, and TaskObject Blueprints, which can be copied and customized to fit the needs of the specific experiment and/or IVR scenario being depicted.

The Tutorial IVE was designed with the goal of communicating and practicing the study procedure and relevant IVR interactions. The IVR scene is kept intentionally abstract, having no corresponding real world scenario. It consists of a walled zone with a few interactive objects (see Figure 5) . The participant is required to perform three tasks: (1) practice room-scale walking in the IVE, (2) practice teleportation in the IVE, (3) practice object grabbing and repositioning, (4) learn how the configuration of IV works in the exploratory study process. When practicing the movement, the participant is asked to walk or teleport to certain positions marked on the floor. When practicing object interaction, the participant needs to sort different objects according to their type, by taking them and placing them in specific containers. Configuring the IV is learned via the example of adjusting the color of an object in the scene. This way, the participant learns the process and necessary interaction with the experimenter on an abstract level, without biasing later configurations during the actual experiment.

The four specific study IVE were developed using the Qixel Megascans Libary12, which provides high quality 3D-scanned assets. To derive generalizable design guidelines for presence, these IVE encompass a wide range of settings, spanning from artificial to natural spaces and from indoor to outdoor areas (see Figure 6). All IVR-Scenarios are based on real life scenarios and are designed based on photo-references to enable an objective assessment of visual fidelity based on a real-world paradigm. The Forest IVE depicts a large forest with paths through a wide ravine. The task is to find three metal watering cans hidden near the paths (Figure 6, left). The Warehouse IVE portrays the inside of a warehouse with multiple aisles of shelfs, storing cardboard boxes on wooden pallets. The task is to find and collect red paint cans (Figure 6, middle-left). The Antique Shop IVE presents a small, cluttered indoor shop filled with old objects of varying value. The windows offer a view into a small and quiet street. The task is to locate and collect hammers hidden throughout the shop (Figure 6, middle-right). The Urban IVE shows a large courtyard, surrounded by different buildings, containing parking lots, small green spaces, and a construction site. The task is to find and collect scuffed soccer balls (Figure 6, right).

5.2. Experimenter Interface

The experimenter interface is implemented as a web app using the JavaScript framework Vue.js13. It is designed to navigate the experimenter through the study and thereby preventing possible operating errors. To ensure the double-blind principle [99] it limits the information the experimenter receives during the experiment to minimize biases in the interaction with the participant. Therefore, the view of the subject in the IVE is not shown to the experimenter during the experiment.

5.2.1. UI and Study Modes

At the beginning of each experiment, the interface allows the selection of one of three study modes, (1) explorative study, (2) comparative study and (3) debug mode (Appendix, Figure A1). The latter is allowing for more operating freedom and insights regarding the experiment status (e.g., for pre-testing; Appendix, Figure A2). The comparative study mode will be added as studies progress according to the methodology (see Section 4).

After entering the subject ID and choosing the explorative mode, the system determines randomly in what order IV are configured and the order in which the IVE are presented, all this information is hidden for the experimenter. The next view contains the four configuration steps needed for the exploratory study (Appendix, Figure A3). The experimenter increases, decreases or continues to the next step according to the exploratory study procedure (see Section 4.1). After the fourth step is completed, the task can be started through the interface. During the task, the interface presents the subtask progression to the experimenter, but without revealing the status or type of specific subtasks. The configuration and IVR task get repeated for every IVE. Lastly, the experimenter has the option to add notes of unusual observations or occurrences during the experiment and to export all data recorded by the tool.

5.2.2. Communication Between Interface and IVR Application

The communication between the two components is by definition unidirectional, with the web-based experimenter interface sending requests to the IVR application. This communication is realized through http requests that get sent to a web server running in the Unreal Engine. This server is part of the remote control API14 and allows the execution of functions in instantiated objects at the currently loaded level (IVE), for example, the manipulation of IV (see Figure 7).

6. Evaluation and Practical Application

Our developed methodology (Section 4) and study tool (Section 5) underwent a formative expert evaluation to test the general usability and applicability for exploratory VR studies. To further evaluate our approach, we conducted exploratory user studies in the scope of the AgeVR research project with N=115 participants. So far, our evaluation covers the initial half of our methodology, up to the stage of concluding the exploratory study.

6.1. Formative Expert Evaluation

6.1.1. Method

The evaluation was conducted with six experts (6 male, 22–32 years), with prior scientific or development experience in IVR. The explorative study procedure (Section 4.1) was simulated by conducting a role play in pairs of two with one expert, as experimenter and the other as participant. The participant’s role was to configure two IV and complete a task within the IVE. The experimenter’s role was to control the experiment through the interface, following a written step-by-step instruction. Subsequently, the experimenter evaluated the usability of the interface through the ISONORM 9241/110-S usability questionnaire [100], while the subject role’s presence was measured via the IGroup Presence Questionnaire (IPQ) [8] and Slater-Usoh-Steed presence questionaries (SUS) [12]. Afterwards, the experiment was simulated a second time, swapping the roles. Finally, a semi-structured interview was conducted with each individual expert. It focused on the adequacy of the study design and the tool’s implementation of the methodology as well as its feasibility for large and diverse subject samples. Considering its formative character, the evaluation was limited to the forest IVE (Section 5.1.2).

6.1.2. Results

Semi-Structured Interviews15

Considering the suitability of the proposed methodology and the corresponding study design, it was stressed that it is essential for participants to have the correct mental model of the sense of presence in order to carry out the configuration task in an effective manner (E2). Furthermore, unknown limits of the IVR-hardware (e.g., maximum brightness of the HMD) could lead to confusion (E1), as well as certain IV starting at zero level, such as illumination intensity and contrast ratio (E3, E5). It was suggested that some of these IV should rather be manipulated on a continuum, rather than in discrete steps (E3). The importance of considering interference between IV as well as auditory aspects in the development of design guidelines was emphasized (E2). Three experts questioned the double-blind principle, saying that it might hinder communication with participants (E1, E3, E4). However, it appears to reduce biases, as four experts wanted more feedback on the status of the subjects (E1, E2, E4, E5), which was justified, e.g., by "being able to already think about result analysis during the experiment" (E1). Addressing the diversity of participants, experts stated that interaction with IVR technology poses different difficulties depending on the user groups (E4, E6). Users with an affinity for technology may expect more configuration options and wider ranges (E5), while too many options may lead to mental fatigue, not only in unexperienced users (E2). It was suggested to simplify the IVR task by adding optional task objects (E1). Being concerned with mobility requirements, especially for older users, experts suggested to reevaluate the time intervals that users were required to stand (E6) and how intensively they must bend down and pick up objects (E5). For the developement of adequate IVR scenarios, E6 recommended focusing on the plausibility of IVE in terms of boundaries and lighting. Depending on the scenario, changes in IV may not be recognizable (e.g., if there are few objects/texture details) (E5, E2). Additionally it was stated that, when it comes to presence, there may be several different IV configurations that can be considered realistic (e.g., depending on the time of day or the underlying narrative) (E5). Furthermore, E4 stressed that IV may have different influences in each scenario, which hints at the importance of our multi-scenario approach to achieve generalizability and counteract contradictory results.

Regarding how well the study tool implements the methodology and study procedure, two experts stated that they had not experienced any problems with the system during the experiment. In the following section, encountered problems and positive feedback by the experts is listed separately for the experimenter interface and the IVR application respectively.

When asked about the experimenter interface, five of the six experts stated that they were adequately supported by the system during the procedure and that they could well imagine using it to conduct a large-scale exploratory study. The main reason given was, the linear and stringent process guidance circumventing possible diversions (E2, E5, E6). Four experts described the UI as "tidy", "uniformly designed", and "easy to use" (E1, E2, E3, E4). The use of a web application was seen as a technological advantage (E1). Several experts noted that error prevention and management could be improved. This included having clearer step descriptions and confirmation prompts for critical actions (E1), as well as checking subject IDs for duplicates when creating the study (E3). Another suggestion was the option to decrement IV, to enable direct correction of configuration errors by the experimenter (E1, E2, E5, E6), The interface seems to comply with the double-blind principle, as four experts stated that they did not know which specific IV they were manipulating (E1, E3, E5, E6). While one expert described labeling of the buttons as "unambiguous" (E3), another expert stated that the labels could easily be confused with each other (E1). It was also noted that progress bars were not directly recognized as such (E6). Additional features suggested include personalization, e.g., a dark mode (E1), keyboard shortcuts (E1), and the automatic loading of subject IDs into the system, e.g., via a text file (E3).

When asked about the IVR application, five experts described the forest IVE as very realistic, due to its high level of fidelity, detail (E1, E2, E4, E5, E6) and large size (E1, E5). Two experts addressed navigability and wayfinding in the IVE, stating that, the IVE and the teleport locomotion were suitable for carrying out the search task (E4), but especially for older users this should be retested (E3). One expert confirmed that the task fulfilled its purpose of ensuring that users actively explore the IVE (E1), however he found the distance of the teleport impractically short. Three experts wished for the IVE’s boundaries to be visualized more prominently (E2, E4, E5). Furthermore, an additional tutorial IVE was suggested to learn the teleport locomotion and picking up objects (E3). Regarding the task objects, marking already found objects was suggested to avoid confusion, as well as more realistic physics when picking it up (E3). Some experts mentioned performance problems, i.e., frame drops (E5, E6) and aliasing effects (E4, E6). It was suggested to perform additional performance tests to lower the risk of cybersickness (E3). W

Questionnaires16

Results of the ISONORM 9241/110-S usability questionnaire ranged from moderate to high values $(Min=4.2,Max=5.95,M=5.43,SD=0.65)$. In line with the interview results, the subscales of self-descriptiveness $(M=4.61,SD=0.92)$, error tolerance $(M=4.39,SD=0.59)$, and individualizability $(M=4.50,SD=0.29)$ were noticeably lower than those of task appropriateness $(M=5.95,SD=0.63)$, expectation conformity $(M=5.28,SD=1.59)$, learnability $(M=5.56,SD=0.59)$, controllability $(M=5.39,SD=0.69)$. However, the variance between experts was higher for individualizability, than for self-descriptiveness and error tolerance (see Figure 8).

Presence ratings obtained were divergent, with SUS scores ranging from 2.5 to 6.67, $(M=5.06,SD=1.62)$ and the SUS counts ranging from 0 to 6 $(M=3.50,SD=2.59)$. The IPQ total score ranged from 3.64 to 5.43 $(M=4.68,SD=0.66)$, the subscales of General Presence $(M=5.83,SD=1.33)$, Spatial Presence $(M=5.30,SD=0.59)$ were considerably lower than those of Involvement $(M=4.21,SD=0.73)$ and Realness $(M=4.22,SD=1.24)$. However, it is unclear which phase of the IVR-exposure the presence rating refer to (during configuration or IVR-task).

6.2. Conduction of Exploratory IVR Studies in the Context of a Research Project

As a first practical application of the proposed methodology (Section 4), and the developed study tool (Section 5), we conducted an exploratory study with 115 participants in the context of the AgeVR research project. The study tool was previously adapted based on the results of the formative expert evaluation: duplicate checking for subject IDs was implemented, button labels were unified, and possibilities to correct IV configuration errors (decrement button) and to annotate experiment data (notes text field) were added. Furthermore, to prevent frame drops and aliasing effects a high-performance VR-PC17 was used in combination with a high-resolution HMD18.

6.2.1. Method

This evaluation serves as an instantiation of the proposed methodology, within the context of a concrete research project, which specifies concrete IV as well as human and environment factors (see Figure 2). The AgeVR research project aims to identify age-specific design guidelines for presence in IVR. The presence factors focused on in this project are those of visual display fidelity and associated age-related user characteristics. Technological factors serving as IV for the experiment were the exemplary ones implemented in Section 5 (I, C, O, T). As a minimal reproducible example, only two IVs were analyzed at a time. Each of the two pairs (I, C and O, T) was configured in two different IVR scenarios (forest and warehouse, see Section 5.1.2) according to the exploratory study process (Section 4.1), leading two four IVR exposures for each participant. Age-related user characteristics were assessed prior to each study (visual short-term memory (block span), spatial imagination, visual system performance (visus, stereopsis, phorieder, color sense, visual field, contrast vision, glare sensitivity). The main Environment factors were the room characteristics stabilized by conducting all experiments in the same IVR laboratory, as well as a tracking/operating area of 4x4m. The developed presence short scale (see Appendix, Table A1) was administered after each IVR exposure. An age-diverse sample was recruited (N=115, 60 female, 55 male, age = 18–88 , $mea{n}_{age}$ = 45.71, SD = 18.77).

6.2.2. Results

The following analysis is restricted to data that is essential for assessing the applicability of the methodology and the correct execution of the experiments. Data addressing the research questions will be presented in a subsequent publication.

Frequently Encountered Problems

All 115 participants successfully completed the exploratory study process, despite 84% of participants having little or no IVR experience (55% were novice users). This means that all participants understood how to configure the IV in the IVR scenario and how to find and collect the task objects in the IVE. 63 of the studies did not encounter any problems during the study procedure. The remaining 52 studies encountered problems that can by classified as technological errors, errors by the experimenter, and errors relating to the methodology (see Figure 9). In terms of technological errors, malfunctioning of IVR software, i.e., the Unreal Engine application (see Section 5.1.1) occurred in 20 studies. This includes unforeseen crashes of the application, physics glitches in task objects, and the teleport locomotion not working properly. Malfunctions in the IVR hardware occured in 34 studies, specifically non-functioning controllers and blackscreens within the HMD, due to loose or damaged cable connections (as the material was heavily stressed by over 400 IVR exposures in total). Errors by the experimenter can be attributed to a lack of attention (25), such as forgetting to turn on the controllers before the experiment, or to plug the headphones into the HMD, or incorrect operations in the experimenter interface (2), such as incrementing / decrementing an IV by mistake and accidentally loading the next IVR scenario. In terms of methodology, communication errors between experimenter and participant occurred in 13 studies, as task instructions or term definitions (i.e., presence) appear to have been ambiguous since they were not correctly understood. Furthermore, the methodological requirements could not be met in 19 studies, as the experimenter sometimes had to give additional information for solving the search task, had to physically intervene (restart controller, adjust HMD/cable), or the experimenter’s voice lead to a break in presence for some participants.

Configuration of IV for Optimal Presence

When analyzing the presence ratings, seven datasets had to be excluded, due to having more than one blackscreen, or data loss due to system crashes. In the IVR exposure, participants were tasked with finding their personal optimal setting for presence, by configuring the two IVs (IxC or OxT respectively). As a control measure, presence was assessed after performing the IVR search task with the set configuration. Figure 10 shows the distribution of post-hoc ratings for all presence subcomponents (data were summarized for both IVR scenarios, thus with N=108 pariticipants, there are now 216 values for each subcomponent). There were only minor differences between the types of IV configured, IxC and OxT show roughly the same distribution (${\Delta }_{mean}=2\%,{\Delta }_{max}=5\%$, for each level of the 5-point Likert scale). However, the subcomponents themselves differ substantially in how well participants were able to achieve their optimal configurations. In terms of the subcomponents of general presence and involvement, configuring the IV worked best with 94-96% of ratings being in the positive spectrum. 60-63% of participants reached the maximum value for involvement, while for general presence it were 47-49%. Regarding plausibility illusion, 75% were in the positive spectrum in both IV configurations (IxC and OxT), but only 27-29% at maximum value. Ratings for spatial/physical presence and self-presence ratings were less clear, both having 59% positive ratings after the OxT configuration and 62% after the IxC configuration and both having more negative (27-28% for spatial/physical presence, 25-29% for self-presence) than neutral ratings (11-14% for spatial/physical presence, 12-14% for self-presence). Regarding social presence, participants were least likely to achieve their optimal configuration as only 15-17% of ratings were positive. Here, most ratings were in the neutral spectrum (49-52%). All subscales violate the assumption of normal distribution.

7. Discussion

Research on presence factors is impeded by the substantial number of identified factors, contradictory results on their influence and lack of clarity regarding their interaction (Section 1.1). Based on the analysis of current literature, we proposed a research methodology to address these problems using a two-step procedure, combining exploratory and confirmatory paradigms (Section 4). As current IVR study tools do not focus on presence and its determinants (Section 3), we developed our own tool consisting of two components: An an IVR app based on the Unreal Engine for designing an displaying various IVR scenarios and a browser-based experimenter interface, enabling precise control of presence factors and study procedure (Section 5). A formative expert evaluation of both the methodology and the study tool was performed by role-playing experiments (Section 6.1). A first practical application in exploratory studies of the AgeVR research project, enabled a first evaluation with study participants (Section 6.2).

Qualitative expert feedback suggest that the methodology can reduce biases (double blind principle) and enhance generalizability of results (multi-scenario approach). Experts had some concerns regarding understandability, physical strain and cognitive load for certain user groups (unexperienced users, older adults). However, this has not been confirmed in the exploratory studies, as all participants were able to complete the study process, despite being an age-heterogeneous sample (18-88 years) and despite the majority being unexperienced or novice users. Methodological problems that did occur, were primarily communication errors between experimenter and participant as well as additional assistance that had to be given (additional information or physical intervention). Thus, task instructions and term definitions (i.e., presence) must be reformulated to make them unambiguous in further studies. Another issue of the proposed methodology was that the experimenter’s voice lead to a break in presence for some participants. This could be addressed by giving the experimenter a virtual representation (avatar) within the IVE or by implementing a (virtual) communication device that is coherent to the IVR scenario (e.g., phone or walkie-talkie).

Considering the study tool, five of the six experts felt adequately supported during the study procedure and could well imagine conducting a large-scale IVR study using the experimenter interface. Expert usability ratings for the study tool reached moderate to high values. Self-descriptiveness, error tolerance, and individualizability were rated noticeably lower than the other scales. Since the tool is supposed to strictly guide the experimental process and minimize derivation, individualizability is not a desired property. Thus, only measures were taken to improve self-descriptiveness by unifying button labels and to increase error-tolerance by implementing duplicate checking for subject ID and possibilities to correct IV configuration errors as suggested by the experts. Consequently, in the subsequent exploratory studies with participants only two operating errors occured in all 115 studies. Performance issues mentioned by the experts were addressed by upgrading the hardware used (VR-PC and HMD) and did not reoccur in the exploratory studies. However, software and hardware errors remained the most frequent issue within the exploratory studies. The former shows that additional debugging of our Unreal Engine-based IVR application is necessary to prevent unforeseen crashes, physics glitches, and other errors, while the Web-based experimenter interface worked without difficulty. The latter demonstrates how high frequency use can put a strain on IVR equipment and material. We learned from over 400 IVR exposures that e.g., cables simply reach their physical limits and must be replaced at some point.

To assess the combined effectiveness of the methodology and the study tool, the presence ratings obtained from the exploratory studies can be consulted. These show how well participants were able to achieve their optimal IV configuration for presence. While the ratings only differ marginally between the two pairs of IV (I,C and O,T), substantial differences could be identified between the different subcomponents of presence. Almost all participants were able to achieve a sense of general presence and involvement with their configurations, and two-thirds of participants had a plausibility illusion (Section 6.2.2). Only slightly more than half of the participants achieved a sense of self-presence. This was likely due to the chosen default value for the avatar, being a hands-only representation which may not have met the participants’ expectations for a realistic virtual body. Furthermore, the only IV that affected the avatar was the texture resolution variable, while others had a more indirect influence (illumination intensity, contrast ratio). Spatial/phyisical presence yielded ratings similar to social presence. Contrary to social presence, however, the IV configuration strongly influenced the visual appearance of virtual objects in the IVE. It is therefore not clear why about half of the participants did not manage to achieve spatial/physical presence with their IV configuration. One possible explanation might be that participants rated their spatial/physical presence for the whole IVR exposure (including the configuration task) and not only for the search task (as instructed). Thus, the great alternations in visual appearance might have led to conflicting perceptions regarding spatial/physical presence. Only one-sixth of participants achieved a sense of social presence with their IV configurations. The reason for that is probably the choice of IVR scenarios and their respective default values, as neither the forest nor the warehouse IVE included any form of social actor. Also, the IV configuration would not have influenced social actors. Thus, in terms of self-presence and social presence, results imply that (1) the IV investigated (I, C, O, T) seem to have no influence. As the study tool will be extended with additional presence factors, their influences might be identified in future studies. (2) Default values play an important role within the IVR scenarios, as the ones chosen for self-presence and social presence (hands-only avatar and no social actors, respectively) seem to have inhibited the experience. Especially, if IV with greater impact on self-presence and social presence are to be investigated in future studies, more sophisticated body representations and the addition of agent or user-controlled social actors might enable a more precise investigation of their influence.

8. Conclusion

In this paper, we have systematically analysed the state of research and derived measures for the systematization of presence studies (Section 4). Our proposed methodology addresses the fundamental problem that the isolation of presence factors is not always possible. The practical workaround is achieved by (1) always focusing on the combined influence of several presence factors, and (2) following a two-step procedure where an exploratory investigation serves as a basis for subsequent hypothesis formation and confirmatory investigations. Our exemplary case was to enable age-differentiated investigations of different presence factors by customizing IVEs to meet varying age-specific needs. The feasibility of our methodology and study tool has been confirmed by expert feedback, as well as data from 115 successfully completed exploratory studies with participants of various ages. The exploratory study procedure works for general presence, involvement and plausibility illusion. Measures that will enable the seamless investigation of the remaining presence subcomponents have been proposed (see Section 7). Our next step is to perform the cluster analysis prescribed by the methodology to develop and test hypotheses in the confirmatory studies. Beyond our own research agenda, the developed study tool was made publicly available as an open source project to enable other researchers to conduct custom experiments on presence factors in IVR. Currently the tool supports the manipulation of four IV (presence factors) in the category of visual display fidelity (I,C, O T). As our system is designed with expandability in mind, additional presence factors, IVR scenarios, and IVR tasks can easily be added.

9. Limitations

The evaluation of the proposed methodology was limited to $p=2$ independent variables. If the study procedures work for $p>2$ is yet to be shown, since the complexity of configuration and the interdependence of the presence factors will increase substantially as additional IV are added. So far, our evaluation covers only the initial half of our methodology, up to the stage of concluding the exploratory study. This is because the AgeVR research project used for validation has not yet progressed beyond this step. From the experimental studies, only data relevant to the methodology’s applicability and experiment execution were analyzed; data addressing the project’s research questions will be published separately. Thus, our next step is to perform the cluster analysis, derive hypotheses, and to conduct confirmatory studies, serving (1) the further evaluation of our methodology and (2) the explicit goals of the research project (age-specific design guidelines for presence).

The proposed methodology itself is limited in its applicability, as it currently accounts only for technological presence factors, while human and environmental factors remain in a controlled but undynamic state. To enable a holistic investigation of factors contributing to the sense of presence, ways must be found to incorporate these presence factors as IV into IVR experiments. Furthermore, in our studies we chose discrete steps for the manipulation of presence factors, while for certain factors a continuous approach might be more adequate (as suggested in the expert evaluation). This could on the one hand enhance the precision with which optimal levels of presence can be detected; on the other hand, this approach could confound optimal values due to subject’s reaction times.

Regarding the implemented study tool, the IVR task is currently limited to simple teleport locomotion and object grasping. It remains unclear how well the results of the studies can be transferred to more intricate IVR interactions, such as hand gestures, passive or active haptic feedback, or even redirected walking techniques. Furthermore, the IVEs implemented so far focus on replications of real-life contexts (forest, warehouse, courtyard, shop). While this may serve as a good starting point for investigating presence factors, it does not exhaust its possibilities. We plan to add various other virtual scenarios (e.g., fantasy settings, abstract scenarios) in the future, to fully leverage the design space for virtual environments.