Evaluating the Use of 360 Degree Video in Education

1. Introduction

In the past two decades numerous studies have demonstrated the benefits of Virtual reality (VR) over multiple levels of education [1,2] and in many application domains [3,4].

The term VR encompasses the array of software and hardware that creates a digital representation of a 3D object or environment, with which a user can interact and feel a sense of immersion. Immersion is one of the primary motivations for educators employing VR, because a sense of immersion in students can foster engagement with learning content [5,6,7].

The definition of immersion is widely debated. In this paper we use an integrative definition from Mütterlein, describing immersion as “the subjective experience of feeling totally involved in and absorbed by the activities conducted in a place or environment, even when one is physically situated in another” [8]. This activity-centric definition allows immersion to be differentiated from the related concept of presence, which describes ‘how realistically participants respond to the environment, as well as their subjective sense of being in the place depicted by the Virtual Environment’ [9]. A heightened sense of presence in a virtual environment is associated with increased immersion in the learning activities performed there [4].

VR has encompassed experiences on many different devices with different levels of immersion and interactivity, from low–immersion, 2D computer screens; to fully–immersive, VR head-mounted displays (HMDs). However, there has been a recent surge in consumer interest in VR and the concurrent release of a new generation of affordable HMDs and peripherals, greatly reducing the barrier of entry to immersive VR [10,11,12,13]. This has transformed the landscape of research into VR and seen educational literature focus on immersive VR as a newly accessible option.

Immersive VR provides a complete simulation of a new reality by tracking the user’s position and creating multiple sensory outputs through a VR HMD [14]. It has been successfully applied to education in many fields, including engineering [15], architecture [16], medicine [17], history [18], and music [19]. Educators using immersive VR often rely on constructivist pedagogy, because the medium facilitates the experiential and environment-driven learning at the centre of this methodology [20,21]. Studies have consistently found positive learner attitudes towards immersive VR [22], and it has been shown to improve time on task [23], motivation [6], and knowledge acquisition [24].

Despite these findings, immersive VR has failed to achieve widespread adoption in education. Traditionally this failure was attributed to the high cost and questionable usability of the technology [2,3,20,25]. With modern VR systems available, this hesitance now arguably comes from educators weighing two other factors: whether there is evidence that VR has enough of an impact on students’ learning to justify costs; and whether they can guarantee the availability of VR content in the long term [22].

Currently, if educators wish to create their own VR content, they either have to learn how to program 3D graphical applications themselves, or hire content developers to create it for them [26,27]. Novel technologies such as neural radiance field [28] and photogrammetry [29] have made it easier to create customized 3D content, but content creation is still a major barrier towards a wider use of VR in education.

In an attempt to address these barriers to VR adoption, we investigated 360° video as an alternative type of educational VR content. 360° video, when viewed inside an HMD, can be considered a subset of immersive VR [30,31]. It provides a movable, user-centric viewpoint inside an immersive, digitally projected environment. However, it is limited in that the user cannot directly interact with or navigate through the environment.

360° video is much easier to create than interactive VR. Spherical cameras that capture 360° video have followed a similar trajectory to VR HMDs, with a recent generation of consumer–focused models greatly reducing the cost barrier [32,33]. In response to this, popular video sharing platforms YouTube and Facebook added support for 360° video in 2015, and later Vimeo in 2017 [34]. Once captured, 360° video can be edited using traditional video editing software in the same file formats as standard videos [35,36]. This means that 360° video is cheaper to make than immersive VR, more easily sharable, and requires less upskilling of current content creators. Recent research has looked into making 360° video more useful by proposing more powerful editing algorithms [37] and mixing it with other media [38,39].

Research into educational 360° video is a relatively new, but it has already been applied to many fields, including medicine [40,41], language [30], sports [42], business [43], collaborative design [44], education and training [45,46,47,48], sustainability [49], and marine biology [50]. The literature suggests many of the same benefits as immersive VR, such as enjoyment, motivation, and improved learning outcomes cite, though more research is needed in this area [34,51,52].

Many of these studies were applied to domains that the researchers believed would particularly benefit from 360° video [30,42]. The evaluated videos were often hard to compare to standard videos because they were embellished with features that do not translate between the mediums [40,50]. Additionally, the studies directly comparing 360° video to standard video only evaluated learning through immediate knowledge tests [40,42,43].

In this paper we describe a randomized, crossover study in which we sought to compare the effects of 360° video and standard video. Twelve videos were recorded and then produced in both 360° and standard video formats.

To minimise the effect of domain and prior knowledge, the content presented was believable but fictitious. To ensure a comparable learning experience from the same video presented in either video format, we placed certain restrictions on the production of the 360° videos, which incidentally lead to parallel production processes between the two video types. And to measure learning outcomes that had not yet been investigated, we included long-term retention and special cases for active and passive visual recollection in our study design.

Participants were divided into two groups, each experiencing six of the videos in one format, followed by the other six in the other format. After watching the videos they were asked to complete both a User Evaluation and a Short-term Retention Test. Six weeks later, participants were asked to complete a follow–up Long-term Retention Test.

This paper reports on both the results of the User Evaluation, and the comparative short and long term learning outcomes of participants who viewed the same video in 360° and standard formats. This is, to the best of our knowledge, the first time long–term learning retention has been investigated for 360° video.

1.1. Research Questions

RQ1:

What differences in user experience exist when presenting educational content in 360° video on a virtual reality head-mounted display, compared to standard video on a desktop PC?

RQ2:

What differences in short and long term learning retention exist when presenting educational content in 360° video on a virtual reality head-mounted display, compared to standard video on a desktop PC?

2. Materials and Methods

This is a randomized, crossover study intended to compare the learning experience of 360° video, viewed on a VR HMD, to standard video, viewed on a desktop PC. The study was spread over a six-week period and collected results about the user experience and short and long term learning retention of participants.

2.1. Participants

In total 20 tertiary students participated in the study, 16 of which were undergraduate and 4 postgraduate. Of these students, 18 were male and 2 were female. The mean age of participants was 25, however the ages of the participants varied greatly (standard deviation=9.02, minimum age=18, maximum=55). No participants dropped out of the study, and all successfully completed all activities.

2.2. Educational Content

There were two primary considerations when designing the educational content used during this study. Firstly, we wanted to limit the effect that application domain may have on learning outcomes of students. Participants may have prior knowledge in different domain areas, affecting the study, and we wanted to confirm that the benefits of 360° video were not constrained to especially well-suited subjects. Secondly, we needed to create videos covering the exact same learning content in 360° and standard formats.

As a result of these considerations we created a series of 12 short lectures in both video formats, filmed at 12 visually distinctive locations (e.g., a forest, a lecture theatre, a playground, inside a car). In 10 of these locations, a teacher would present a series of facts about that particular location, sometimes referring to visible features around them. The other two locations will be discussed in Section 2.3, Special Case Videos.

To address the effect of domain and prior knowledge, the location names and information presented in the videos were designed to be believable, but were actually fictitious. For example, in the Whiterock Bush video recording, participants were informed that the area had been the site of `the frequent illegal dumping of rubbish’. Whiterock Bush is not a real location, nor is the presented fact true. Participants were not informed that the information was fictitious until after the study was complete, as that knowledge could have an effect on learning outcomes.

Ensuring the two video formats provided the same information placed two limitations on video development. Firstly, as extra information is visible when viewing a 360° video, we needed to ensure learning did not rely on content outside of the frame captured in the standard video. Secondly, the quality of accessible 360° cameras varies considerably, and while the pixel resolution is high compared to standard video, it is stretched over the full sphere when viewed. This is compounded by re-projection through VR HMDs of varying resolutions, leading to 360° video experiences often being low quality with poor text legibility. It is for this reason we chose to use verbally-delivered lectures in visually distinct locations as the educational content.

We also decided to reduce the resolution of the standard videos to match that of the 360° videos. Although current technology results in a difference in resolution between typical 360° and standard desktop video, we anticipate that in the future the gap will close and the experience of resolution in each technology will be more similar. We adjusted the resolution to ensure our results are more robust to changes in technology and not significantly influenced by the current technology limits.

2.3. Special Case Videos

There were 3 additional factors of interest we wanted to measure, for which we designed 2 special case videos, and modified one of the location-based lectures.

Firstly, as arguably the most common setting for tertiary educational videos, we included a lecture theatre as one of the locations. Rather than presenting facts about the location in this recording, we instead performed a traditional lecture where information was presented both verbally and via an overhead projector. This was done to evaluate whether increasing the feeling of presence in a lecture theatre would affect learning outcomes.

We were also interested in whether viewing a video in 360° would affect the active and passive recollection of visual information. To investigate active visual recollection, we included a recording located inside of a car. Participants were not presented with any facts, but were instead asked to memorize as much detail as they could about the interior of the vehicle. They were later tested on this information.

To investigate passive visual recollection, a bright red ribbon was tied around a tree in one of the existing locations. The ribbon was clearly visible in the recordings, however the ribbon was never explicitly mentioned by the speaker at this location. Participants were later asked to recall where they had seen this ribbon among 3 visually similar locations.

2.4. Technologies

The 360° videos were captured using a Ricoh Theta S spherical camera, and were displayed on a Samsung Gear VR HMD (used in conjunction with a Samsung Galaxy Note 5).

The standard videos were obtained using the free and open-source OBS Studio to take screen recordings of the 360° videos, running on the free Ricoh Theta desktop video player software. They were recorded at 60fps and adjusted to 1920x1080 resolution, to match the 360° video resolution. Standard videos were played on a 23” PC monitor with a resolution of 1920x1080.

In most of the standard videos, all necessary information was contained withing the intial field of view. However, in some locations small rotations of the camera were needed to include other details. These rotations were integrated into the standard video recordings, not controlled by or performed in front of the participants.

In total 24 video clips were created, both the 360° and standard versions of each location and lecture. For a detailed description of the content creation process, as well as the issues we encountered see [53].

2.5. Study Design

Participants were randomly divided into Groups 1 & 2. Both groups were shown all 12 recordings, however Group 1 was shown locations 1–6 in 360° video on a VR HMD, and locations 7–12 in a standard video on a desktop PC. Group 2 was shown the opposite videos on the HMD and PC (see Table 1). This is a randomized, crossover design for the user experience of the video formats, with both groups experiencing both treatments, then comparing them through a User Evaluation.

This study is additionally a randomized design for learning outcomes between the video formats, achieved by comparing the test scores of participants experiencing the same recordings in different formats. These scores were obtained from both a Short Term Retention Test and Long Term Retention Test.

The groups compared in this second design are not Groups 1 & 2, but rather the pseudo-groups HMD and PC. This is because the scores associated with experiencing 360° video will include question scores from both Groups 1 and 2, depending on the recording being assessed by each individual question. In fact, all participants conditionally appear in both pseudo-groups, since all participants experience both 360° and standard videos.

2.6. Assessment Instruments

Three assessment instruments were used in this study: a User Evaluation, a Short Term Retention Test, and a Long Term Retention Test.

The User Evaluation was 12 questions long, comprising 4 Likert Scale questions, 1 short answer question, and 7 open-ended questions. In all questions participants were asked to compare 360° and standard video experiences, except the final question, which directly asked participants for their opinion on 360° video as an educational tool.

The User Evaluation included questions about participants’ senses of enjoyment, immersion, and engagement while watching the different video formats. These are some of the demonstrated benefits of immersive VR, and we wanted to validate that they were evident in 360° video as well.

The Short Term Retention Test was 20 questions long, comprising 14 multiple choice questions and 6 short answer questions about the content of the video recordings. This test also included questions about the active and passive visual recollection from the special case videos.

The Long Term Retention Test was 21 questions long, comprising 15 multiple choice questions and 6 short answer questions. It is identical to the Short Term Retention Test except for 4 questions, which are slight simplifications of 3 of the original questions. 2 of these let the participants recall information from an image instead of a location name, and 1 short answer question asking for two pieces of information was split into both a short answer and multiple choice question.

The Short and Long Term Retention Tests were divided into 2 sections, Content Retention and Location Recognition. Questions about Content Retention focused on the recall of the information participants were taught during the lectures, while the Location Recognition questions targeted visual information about the recording locations. The questions were designed to be unambiguous and binary in nature, allowing for a rigid marking rubric.

2.7. Study Procedure

Participants were first asked to fill out a Demographic Questionnaire. This questionnaire collected basic information, including the participants’ age, area of study, experience with VR HMDs, and whether they had any issues affecting their vision (so the HMD could be adjusted accordingly).

Next participants were asked to put on the HMD, and instructed to complete both a pre-installed Oculus Tutorial as well as watch a 2 minute long introductory lecture in the HMD. This was to familiarize the participants with 360° video.

After the familiarization protocols, participants were asked to watch the first set of videos on either the HMD or the PC, depending on the group they had been assigned to (see Table 1). These video sets were each comprised of 6 location recordings stitched together, forming one continuous recording approximately 5 minutes long.

Participants were then asked to perform an unrelated reading task for 3 minutes, designed to act as a distractor between the 2 video sessions. After the distractor task, they were shown the second set of videos in the other video format.

When they had finished viewing the video content, participants were asked to complete both the User Evaluation and Short Term Retention Test. 6 weeks later, participants were asked to complete the online Long Term Retention Test.

3. Results

This section is split into three primary subsections. Section 3.1 discusses the results of the score-based questions in the User Evaluation, before Section 3.2 reports on our thematic analysis of the open-ended responses. Section 3.3 then outlines our statistical analysis of the results of the Short and Long-term Retention Tests.

3.1. Quantitative User Evaluation

The User Evaluation was a combination of Likert Scale, short answer, and open-ended questions. It was completed by all participants after they had finished viewing both video formats.

This section reports the quantitative results from the 4 Likert Scale and 1 short answer question, discussed in two groups: Subjective Experiences and Study Habits.

3.1.1. Subjective Experiences

In the first 3 Likert Scale questions, participants were asked to rate the degree to which they felt enjoyment, immersion, and engagement while watching the two different video formats. These questions used a 5-point Likert scale, from -2 (Strongly Disagree) to 2 (Strongly Agree), with 0 corresponding to Neutral.

Immersion was described to participants as the degree to which they forgot about their surroundings and ’lost themselves’ in the experience. Engagement was described as the degree to which a learning experience holds their attention and makes them actively focus on their learning.

Table 2 presents the means and medians of the responses to these 3 questions, which range between -2 and 2. It also presents the results of Wilcoxon signed rank tests between the scores for 360° video and standard video, and the Cohen’s d effect sizes where the tests were considered significant at the $\alpha =0.05$ threshold. Cohen’s d values of 0.20, 0.50 and 0.80 correspond to small, medium and large effect sizes respectively. Wilcoxon tests were used because Likert responses are ordinal, the sample size is small (n = 20), and each group of scores heavily defied a Shapiro-Wilk test for normality.

Based on these analyses, we found statistically significant results across all 3 questions, with participants expressing strong preference towards 360° video over desktop video in all factors.

As Figure 1 shows, participants rated their sense of Enjoyment much more highly for 360° video than standard video. The majority of respondents, 12 of 20, responded neutrally to standard video, with only 5 responding positively. Conversely, 18 participants responded positively to 360° video, with 11 of them saying they “Strongly Agree” that they enjoyed the experience. This is strong evidence that the students enjoyed the 360° versions of the videos more than the standard versions, with a large effect size in our sample.

Figure 2 shows the results of the participants rating their sense of immersion during the 360° or standard videos. Of these 3 subjective Likert questions, this one had the smallest p-value and largest effect size, suggesting that the participants felt much more strongly immersed in the 360° video than the standard one. 19 of 20 participants responded positively to this question regarding 360° video, while 12 responded negatively regarding desktop video.

Regarding engagement, the responses were again much more in favor of the 360° video approach, with 16 of 20 participants rating their engagement level positively, compared to just 5 rating positively for standard video.

Unlike the previous 2 questions however, 2 participants responded negatively regarding their feeling of engagement with the 360° content. While the p-value was significant and the effect size large, this question had the smallest difference between groups of these 3 subjective Likert questions. Potential reasons for this will be expanded on in the discussion, after evidence about attention and distraction is presented in Section 3.2.1.2 of the thematic analysis.

3.1.2. Study Habits

The next two questions concerned the potential study habits of participants using the two video formats. Participants were asked how long they believed they could comfortably study by watching 360° video on a VR HMD, or by watching standard video on a desktop PC. They were then asked how likely they would be to use educational materials of a similar nature to those they had been shown, as a supplementary (but optional) material for their own courses. These were a short answer and a 5-point Likert scale question, respectively.

The differences between the responses to the two video formats were analysed using Wilcoxon signed rank tests, and the effect sizes estimated by Cohen’s d when the result was significant. The results are given in Table 3 and illustrated in Figure 4.

These responses indicate that participants believed they could study using the different video formats for very similar amounts of time. The mean reported time for standard video was 87.1 minutes, with 79.4 minutes for the 360° ($sd=57$). A Wilcoxon signed rank test on the reported times found $p=0.91$, showing no significant difference between these times.

Assuming that they had equal and easy access to a desktop PC and a VR HMD, participants showed a statistically significant preference to the idea of using 360° video as supplementary content to their own courses. Most participants answered that they were Likely or Very Likely to use both 360° video and Standard formats, but this number was larger for 360° and weighted towards Very Likely.

3.2. Thematic Analysis

This section outlines the results of our thematic analysis of the open-ended responses to the User Evaluation. The evaluation included 6 questions in which participants were asked to comment on both the standard and 360° video experiences, and a final question in which they were asked their opinion on 360° video as an overall tool for education.

Participants were asked about their senses of immersion and engagement, their level of distraction, their preference of format for lecture recordings, and any issues that were caused by video and screen quality, motion sickness, or other factors.

Qualitative responses were examined for key themes, patterns and outliers based on the Coding in Detail approach outlined by Bazeley and Jackson [54]. NVivo was used for the coding process, and as a result we created 29 codes organized into 4 primary themes: engagement, immersion, learning and technology limitations. These codes were applied a total of 268 times to the data.

To test the validity of this coding scheme, two reviewers independently coded a subset of the data. Cohen’s Kappa was then calculated, resulting in a coefficient of $k=0.8705$ suggesting a high degree of inter-rater agreement.

The 4 primary themes relate to the questions asked, but codes for each theme were spread among responses for every question. The following sections will discuss the findings in each theme, with notable sub-themes presented in subsections where appropriate.

3.2.1. Engagement

Of the 4 primary themes we identified during our analysis, engagement was the most prevalent. Participant responses were coded 117 times as relating to engagement, accounting for 43% of the total coded data.

Responses in this category can be grouped into two major sub–themes: enthusiasm and enjoyment, which follows expected trends based on the quantitative results above; and attention, which revealed substantially more mixed opinions and interesting insights about the video formats. These sub–themes are discussed in further detail in the following two sub–sections.

Enthusiasm and Enjoyment

Mirroring the quantitative results from Section 3.1.1, participants expressed much more enthusiasm and enjoyment regarding learning in 360° video. 12 of 20 participants described it as `enjoyable’ or `fun’, while 4 participants explicitly described the standard video as `boring’.

The major factor to which participants attributed their enjoyment in 360° video was the interactive, immersive environment. One participant said that `the experience of being able to look around and follow visual cues was engaging’, while another described 360° video as `Dynamic’ compared to the `Stagnant’ standard video. This idea of passivity in the standard video was mentioned by another participant, who stated that `It was quite boring to just stare dully at the screen, and try to just take in all the information’.

Another factor that 3 participants mentioned was the novelty of experiencing 360° video. Only 7 of the 20 participants claim to have used a VR HMD for more than an hour in the demographic questionnaire, and many of the others reported an increased sense of curiosity and interest in the immersive environment. Novelty will be discussed in Section 4 as it is likely to impact enjoyment, and also affected the leniency reported in Section 3.2.4.3 below.

Overall, these responses clearly show that participants enjoyed the 360° video format more, but whether this translated into engagement with the learning content itself will be discussed further after more evidence is provided.

Attention

While participants rated the overall engagement of 360° video more highly than standard video, in asking specifically about distractions we were able to find more nuance in this response.

Surprisingly, 11 of the 20 participants claimed to be more distracted in 360° video. Only 5 felt more distracted watching standard video, while 4 claimed no difference or did not compare the formats directly. The interesting factors here, however, are the vastly differing reasons students felt distracted.

Participants who felt more distracted in standard video often cited boredom, repetitive content, and a lack of focus or attention. They also mention distractions from the real world, and that 360° video has an advantage because `it hides distractions in the surroundings.’

For the larger group who felt more distracted in 360° video, the major reason was the extra information available in the surrounding environment. One participant reported they `can’t focus on the speaker because [they’re] curious with the surrounding things’, while another said that 360° video `was very immersive... But I don’t know how much the distraction of looking around was combating the usefulness of it.’ Another participant claimed they `enjoyed being able to look at random things while listening’, but did concede they `got distracted more with this.’

5 participants additionally reported accidental encounters with the real environment as a distraction in 360° video. For example, 2 of these participants specifically mention bumping their foot against the wall while wearing the HMD.

Interpreted together, the responses regarding distraction highlight the distinctions between the two video formats. Watching a standard video is a less engaging activity, but the limited information can help students identify what to focus on for their learning. 360° video is much more fun and engaging, but the immersive environment can pull attention away from the learning content if it is not directly relevant. This is a key finding that will be expanded on further in Section 3.2.3.1 and Section 4.

3.2.2. Immersion

A sense of Immersion was commonly mentioned by the participants, with responses coded to this theme 57 times (accounting for 21% of the coded data). 16 of the 20 participants described experiencing immersion in the 360° video, whereas only 1 of the participants described finding the desktop video (`somewhat’) immersive.

The responses regarding immersion were frequently strongly emotive in nature, with participants using superlatives including `surreal’, `incredible’ and `powerful’ to describe their experiences. As noted in Section 3.2.1.1, immersion in the environment was often given as the reason a participant enjoyed 360° video more than standard video.

Immersion is a primary motivation for using 360° video, so these results are encouraging but not surprising. However, the sources of immersion are useful to understand so that they can be taken advantage of.

Two distinct groupings emerged from this theme: immersion through a heightened sense of presence and realism in the environment; and immersion from a sense of intimacy and closeness to the presenter.

Presence

A feeling of presence in the 360° video was explicitly mentioned by 7 participants, while it was not applied to standard video at all.

One participant wrote `I can feel the surround[ings] with me, just like I am there’, while another said they `Forgot [they were] actually still in a room for a moment...’. A couple of participants related very powerful feelings of presence, one stating `I felt as if I was a new person and I’m in a different world.’

2 participants described the lack of presence in standard video, with one saying it `felt more like looking through someone’s perspective’. Another participant, when comparing the video formats, said of standard video `I imagine I was there, imagining what was there, in contrast to feeling `like I am there’ in 360° video.

A sense of presence can be reinforced by a sense of realism in a virtual environment, and the realism of the 360° video was mentioned by 3 participants. One said `it allows me to step into the content and explore...so it feels like the experience is more real’, and another described 360° video as `a more “real” experience’ than standard video.

The sense of presence potentially afforded by 360° video is a major motivation for its use over standard video, which is reinforced here by our study results.

Intimacy

Another way in which participants experienced immersion in 360° video was through an increased sense of intimacy with the speaker. 3 participants mentioned this directly.

One of the participants listed as a benefit of 360° video: `I found the headset gave me a greater sense of interaction with the presenter, I could understand him a bit more’. Another wrote as a criticism of standard video, `The experience was almost less personal than in VR, so I felt like I could pay less attention to the facts’. This feeling of obligation to the presenter was echoed by a third participant, who wrote `I feel like I have to pay more attention because the person speaking feels more like a real person’.

3.2.3. Learning

This theme encapsulated any discussion about how the video formats may impact learning outcomes, or the learning content of the videos themselves. This was the least common theme, and was only coded 28 times during the analysis (approximately 10% of the coded data).

3 participants said they felt 360° video helped with their learning outcomes, one stating `it seemed like [my] brain learnt by itself as I do not need to concentrate too much’. The other 2 directly mentioned it helping with memory retention, one postulating that `perhaps it also engages spatial processing in the brain more which creates more robust memories’. The other attributed the better retention to novelty and immersion, saying they `felt HMD is the more memorable experience hence [they] would be able to remember and recall more things’.

However, most responses under this theme were about which types of learning content are appropriate for standard and 360° video.

Content

Generally, responses about content suggested that effectiveness of 360° video would depend on the content being taught and how it was presented.

When asked about their preference of video format to watch lecture recordings, 9 participants said they would prefer 360° video, 8 would prefer standard video, and 2 said it would depend on the content of the lecture.

When participants preferred 360° video for lecture recordings, they cited increased immersion and novelty as the reason. One participant even claimed they would use 360° video until their novelty wore off, and likely switch to standard video after that.

The reasons participants would prefer using standard video for lecture recordings included that they can more easily take notes, they would be less likely to encounter distractions, and that a lecture recording is not appropriate content for 360° video. Integrating these last two reasons, one participant wrote `If it’s a lecture recording; there’s no need to be aware of my surroundings - it’d only be an unnecessary distraction’. Another wrote quite directly `The content of the lectures doesn’t get improved by HMD. A lecture is a lecture!’

The topic of content-appropriateness was again mentioned by several participants when giving their opinion of 360° video as an educational tool. In one participant’s words: `For material which benefits from being in the environment, it’s a much more enjoyable way to learn it. For material which has no benefit from being in the environment (e.g., a traditional lecture), the desktop is more comfortable to use.’

A couple of participants linked this idea back to distractions, one saying `the content should be related to the spherical video, otherwise it’s too easy to get distracted.’ This was echoed almost verbatim by another participant, who responded `immersing students in a subject is how you best learn. If you could immerse a student and make sure the distractions are informational, you have a very powerful tool.’

The presentation of learning content is closely tied to the themes of attention and distraction reported above. The implications of these findings will be explored in the discussion.

3.2.4. Technology Limitations

The final theme in our thematic analysis encompasses any issues relating to the technologies themselves. Responses were coded as relating to this theme 85 times (approximately 31% of the coded data), and came largely from questions on the User Evaluation about screen and video quality, motion sickness, and other issues encountered by participants.

The nature of the issues discussed varied considerably, however they can be classified as relating to either the comfort of the participants, or issues with the viewing experience and video quality. The other interesting sub–theme to emerge here is leniency; multiple participants exhibited forgiveness towards technical issues with 360° video. These three sub–themes are discussed below in the final sections of this thematic analysis.

Comfort

Problems with the physical comfort of the HMD were among the most common issues with 360° video, being mentioned by 8 of the participants. Of these, 2 described experiencing a sense of disorientation when removing the HMD, while another expressed difficulty swivelling the computer chair while watching 360° video. The remaining 5 participants all mentioned that the HMD was heavy, with several stating that it was uncomfortable on their neck.

Surprisingly, and despite being explicitly asked about it, only 1 participant reported any motion sickness while wearing the HMD.

The final issue of comfort while viewing 360° video was not physical, but was instead the fact that students cannot take notes while wearing a VR HMD. This was explicitly mentioned by 3 of the participants, 2 of whom provided it as their justification for why they believed that 360° video may not be an effective tool for education overall.

Viewing Experience

An equal number of participants reported issues with the quality of the 360° video and the standard video. 6 participants mentioned their learning was impacted by video or screen quality in each format.

This is surprising because the videos were of the same resolution, but the 360° video is spread over the full visual range of the viewer, generally decreasing perceived quality. We anticipated more issues with 360° video than standard video, but 2 participants even stated they believed the 360° video had a higher resolution than the standard video. This shows a level of leniency that the participants had towards 360° video, which will be noted in the next section.

3 participants encountered a specific issue with the viewing quality in 360° video: the Screen Door Effect. These participants described being able to identify individual pixels, with one writing `Because the pixels were visible, my immersion and attention were both negatively affected’. This was the only viewing quality issue ascribed to the screen itself, instead of the resolution of the video.

Leniency

In reviewing responses about the limitations of 360° video, and interesting trend emerged of participants expressing leniency towards these limitations. This forgiveness was expressed by 6 of the participants, and usually came in the form of a criticism followed immediately by a justification as to why it was unimportant.

For example, when asked if video quality affected their 360° experience one participant wrote: `Yes! But overall the experience felt quite real so the low quality was a compromise’. Another similarly responded `Yes, resolution could have been better, but I think that being able to interact with the environment lessened the impact of this’.

This leniency could have a couple of causes, including novelty and social desirability bias, which will be expanded on during the discussion in Section 4.

3.3. Learning Retention

The second major section of results involves learning retention. Learning retention was measured through the Short Term Retention Test, taken by participants directly after watching the videos, and the Long Term Retention Test, taken 6 weeks later. The tests were comprised of multiple choice and short answer questions separated into two sections: Content Retention and Location Recognition.

Test scores for short and long term retention were analysed by comparing the scores between participants that experienced the same recording in 360° and standard video. This involved comparing the pseudo–groups HMD and PC, which conditionally contained all participants, but for questions about different locations. For example, for questions pertaining to the James Cook Domain recording, scores from Group 2 were considered HMD because they experienced it in 360° video, while scores from Group 1 were considered PC. For questions about the Rawene Nature Reserve recording, scores from Group 1 were considered HMD, and scores from Group 2 were PC.

This setup effectively splits the sample into two groups, each with n = 10. The reduced sample size means that statistical comparisons are less powerful than would be desired, but they can still indicate if an effect may be present and could warrant further investigation. None of the score distributions for HMD and PC in either test was significantly different to normal, according to Shapiro-Wilk tests, so they were compared using two-tailed two-sample Student’s t-tests assuming unequal variances. These results are reported in Section 3.3.1 and Section 3.3.2.

The Short and Long Term Retention Tests also included questions about the Special Case videos, which sought to test active and passive visual recollection and recollection from a lecture theatre environment. These questions were part of the Content Retention section of the tests, and are analysed individually in Section 3.3.4 below.

3.3.1. Short Term Retention

The Short Term Retention Test was scored out of 32 total marks. 27 marks were available for Content Retention, and 5 for Location Recognition.

From Table 4 we can see that total mean marks of both the HMD and PC group are very similar, at 21.65 and 21.45 respectively. These means are 67.65% and 67.03% of the available marks.

The overall t-test found $p=0.6174$, suggesting that there is no statistically significant difference in the learning outcomes of the two groups.

While the HMD group did appear to perform better in the Location Recognition questions, a second t-test run on just these results found $p=0.2248$, indicating that the difference was not statistically significant.

3.3.2. Long Term Retention

The Long Term Retention Test was scored out of 33 marks, with 27 available for Content Rentention, and 6 for Location Recognition.

From Table 5 we can see that again, while the HMD group performed better on average than the PC group, the difference in learning outcomes was not statistically significant. The HMD group achieved on average 15.25 (46.21%), while the PC group averaged 14.7 (44.55%), and the t–test found $p=0.1615$.

As the HMD group appeared to perform better on the Location Recognition questions, a second t-test was run these results for the 2 groups. However, the result was not significant at the $a=0.05$ threshold, with $p=0.3575$.

3.3.3. Question Performance

Interestingly, from Table 6 we can see that the HMD group performed better on more than twice as many questions as the PC group in both the Short and Long Term Retention Tests. We investigated the probability of this occurring by chance using a binomial test, assuming equal probability that either group would outperform the other on a question. The cumulative probability that the HMD group performed better on 24 questions across the two tests is $P(X\ge 24)=0.174$, under the null hypothesis that video format had no effect.

While this value is not significant at the $a=0.05$ threshold, this is an indication that a study with more power has the potential to detect an effect on learning, or that the impact of video format may vary considerably between types of question.

3.3.4. Special Case Questions

As outlined in Section 2.3, participants were tested for 3 special interest cases: recollection from a lecture theatre, and active and passive visual recollection. Content from the lecture theatre video accounted for 4 marks in both the Short and Long Term Retention Tests, and 1 mark was assigned to each of the visual recollection cases. The results of these questions are in Table 7, along with students t–test results comparing the performance between the HMD and PC groups.

From Table 7 we can see that the HMD group performed better on all of the cases across both tests, with the exception of long-term retention of the passive visual information, for which the 2 groups performed equally. Though this may suggest a trend, t-tests again failed to find any of the differences on the individual factors significant.

4. Discussion

RQ1. What differences in the user experience exist when presenting educational content in 360° video on a virtual reality head-mounted display, compared to standard video on a desktop PC?

The results of our User Evaluation analysis highlighted many of the key differences between using standard and 360° video.

In terms of general preference, participants were found to enjoy 360° video more, feel more engaged by it, and would prefer to use it as a supplementary material to their learning, provided they could access it as easily as standard video. This was largely attributed to the increased sense of immersion and interaction in the environment in 360° video, amplified by feelings of presence and realism. These results are consistent with existing research around immersive VR and validate our motivations for using 360° video as a learning tool.

Looking further into the thematic analysis, we see a more complex relationship between immersion and engagement in the two video formats. While 360° video was more engaging, participants commonly reported being more distracted while using it, primarily by the interesting, immersive environment itself. The extra visual information pulled attention away from the speaker in the video who was delivering learning content verbally.

This illuminated a distinction in attention and focus between the video formats. Participants watching standard video were more likely to get bored, and be distracted by elements of the real world. However, they paid more attention to the core learning elements of the video. Viewers of 360° video were more engaged, but that did not translate directly to engagement with the learning content itself.

While a couple of participants took this as evidence that standard video was the better teaching tool, more noted the potential strength of harnessing distractions in 360° video. They noted that if the learning content is reinforced by the visual environment, instead of being distracted by it, 360° video has the potential to create greater engagement with learning content than standard video.

This related directly to another major emergent theme: content–appropriateness. Participants identified that 360° video would be more suited to topics that relied on visual and environmental information, instead of those that are taught verbally or through text.

Our study design should be taken into account when discussing distractions and content–appropriateness. We included videos that were visual–information heavy (e.g., memorising details inside a car), and verbal–information heavy (e.g., a traditional lecture recording), incidentally allowing participants to compare the two kinds of content. We also tried to create as comparable an experience as possible between video formats, by restricting the learning content in the 360° videos to a frame that could also be captured as standard video. This meant most of the environmental information in the 360° videos was not part of the learning by design, which would have increased the effect of distractions.

Interestingly, the sub–theme of intimacy suggested that 360° video may have an advantage with some verbal content. Some participants felt an increased sense of connection to the speaker due to the presence and realism of the video format, which translated into a sense of obligation to listen to them. This kind of authentic connection can be particularly useful in situations like language learning [30].

Enhanced intimacy could even be harnessed to create better learning resources for students from cultural backgrounds that emphasise the relationship between student and teacher as a part of learning. For instance, Reynolds suggests that for students of Pacific Island background, `teu le va [the nurturing and valuing of a relationship] between a teacher and student is crucial because a student’s identification with a subject can come through a positive connection with a teacher’ [55]. With the increasing prevalence of digital learning resources, consciously maintaining an element of human intimacy as part of their design will benefit some students.

The final major difference between the video formats to mention here is usability. Watching 360° video was reportedly less comfortable, with a few participants noting the weight of the HMD or a sense of disorientation from using it. Participants also noted that you cannot take written or typed notes while in immersive VR. Overall, however, these limitations did appear not offset the benefits of 360° video, as participants also claimed they could study from both formats for similar amounts of time, and would prefer to use 360° video to do so.

RQ2. What differences in short and long term learning retention exist when presenting educational content in 360° video on a virtual reality head-mounted display, compared to standard video on a desktop PC?

Overall, our analysis of the results of the Short and Long Term Retention Tests found no statistically significant differences in learning retention between the HMD and PC pseudo–groups. This is true of the overall scores, the separate scores for the content retention and location recognition subsections of the tests, and the scores associated with the three special case videos.

This means that 360° video was as effective as standard video in conveying learning content, even with the limitations we placed on production. There were no major differences in short and long term retention in the context of this study.

It should be noted that the small effective sample size (n = 10) limited the statistical power of this study. When looking at the results in aggregate, the HMD group outperformed the PC group in every measure except two (short-term content retention and long-term passive visual recollection), and scored better in over twice as many total questions. As mentioned, none of these results were statistically significant at the $a=0.05$ threshold, but there is an indication that a study with greater strength may be able to detect an effect.

5. Limitations

Novelty is a commonly observed confounding effect in studies involving new learning technologies [56,57], including immersive VR [22]. It can lead to increased motivation or perceived usability of a technology, which may translate into increased attention and engagement in learning activities [7]. The responses to the user evaluation showed evidence of the novelty effect among participants, which should be taken into account with these results. The novelty effect also manifested itself as leniency towards issues with 360° video.

Connected to this novelty is the potential for anchoring bias in our study results. Participants will have likely watched many standard videos in the past, giving them high quality reference points or `anchors’ against which to evaluate the study videos. However, they are much less likely to have encountered 360° videos before, and therefore may not have any expectations of quality in that format. The anchoring effect can lead to more severe criticisms of experiences that can be compared, and more lenient evaluations of novel experiences [58].

Another factor that could generate leniency is social desirability bias. This is where survey responses are informed by a participant’s desire to project a favourable image to others [59,60]. In this study participants may have been less critical of 360° video, aware that we (the researchers) were studying the format, and mistakenly believing we may perceive them undesirably for answering negatively.

As noted previously, the other major limitation of this study is the small sample size. This caused our statistical analysis to have limited power to detect differences in learning outcomes between the HMD and PC groups. Future studies comparing 360° and standard video will need larger sample sizes to conclusively comment on any effects on learning retention.

6. Conclusions

In this paper we discussed the results of a study looking into the differences in user experience and short and long term learning retention in tertiary students viewing educational videos in both 360° and standard desktop formats. We found that participants retained the same amount of learning from both types of video, and engaged with and enjoyed 360° video more. We also found that participants believed they could study using either format for a similar amount of time, and would generally prefer to use 360° video as supplementary materials to their coursework, provided it was accessible.

To create a more comparable experience between the 360° and standard videos used in this study, we placed restrictions on the production of the 360° videos. That means these results were obtained using 360° videos produced in an accessible way, parallel to standard video production. This indicates that 360° video is a viable way for institutions to continue generating value from an investment in immersive VR technology, without having to upskill their current learning content creators or pay for expensive bespoke VR development.

Our user responses also produced the insight that though this would be effective, 360° video has much greater potential for learning engagement if additional effort is put into incorporating the learning content into the surrounding environment. Our results suggest that at its most accessible level, 360° video is equally effective and more engaging than standard video. However, it is a different medium with a different user experience, creating engagement and enjoyment through a sense of immersion in the wider environment. To unlock the potential of 360° video for education this environment should be harnessed to reinforce learning objectives.