Reliability of a Smooth Pursuit Eye-Tracking System (EyeGuide Focus) in Healthy Adolescent and Adults

1. Introduction

Eye-tracking has been well established to be able to determine cognitive states in neurological health and injury, or with neurological disorders [1,2,3]. Recently, devices that assess oculomotor eye movements have been developed to improve the objectivity and time taken in order to be able provide a neurological assessment. This in turn has placed oculomotor testing as a convenient technology to assess neurological injuries such as concussion.

Induced by biomechanical force directly or indirectly to the brain [4], a concussion causes an assemblage of transient signs and symptoms representing complex pathophysiological processes across multiple cortical areas of the brain [4]. Moreover, concussion is an evolving injury whereby symptoms can change (or worsen) over time reflecting the changing physiology as the brain recovers from the insult.

The current consensus on concussion in sport relies heavily on symptom reporting and presentation, prior to further testing such as neurocognitive assessment, and this may be open to manipulation or under-reporting [5,6]. For example, in elite sport, ~6% of male and female professional athletes in the Australian Football League have admitted to hiding symptoms of concussion [7].

Consequently, there is increasing discussion and emerging evidence on the benefits of utilizing supplementary markers within the concussion assessment protocols [8,9,10]. Incorporation of assessments such as oculomotor smooth pursuits provide objective screening to assist sports club medical staff who may suspect a concussion, or during the recovery phase after sport-related concussion [11,12].

While the physiological mechanisms of smooth pursuits is still yet to be fully understood, this involves saccadic and cognitive requirements of focus tracking on a moving object [12,13]. Multiple cortical and subcortical areas are involved in eye movement (see review [14]), making eye-tracking a suitable proxy for a measurement of cognitive functioning after a concussion [11,15].

One eye-tracking device recently developed is the EyeGuide Focus. The system is portable and simple to utilize involving a 10-second assessment, quantifying oculomotor smooth pursuit or Dynamic Visio-Motor Synchronization [16]. There are emerging studies employing the EyeGuide Focus system, not only across concussion management [17,18,19] but also studies in fatigue [20]. However, to date, there are no published studies on reliability of the system. The aim of this study was to determine intra-session reliability and differences in healthy adolescent and adult cohorts.

2. Materials and Methods

A sample of 75 healthy participants were recruited (adolescents: n=28; female=11, male=17, mean age 16.5 ± 1.4 years; adults n=47; female=22; male=25, mean age 26.7 ± 7.0 years). All participants were physically active, but not involved in elite sport. Participants had reported no history of concussions or other traumatic brain injury; and no pre-existing visual, neurological or psychiatric conditions, nor were on any medication. The sample size was determined using an a-prior power analysis (G*Power V3.1.9.6, Germany)[21] for within-between groups repeated measures with an assumed effect size of f =0.25; α<0.05; power (1- β)=0.95 would require a minimum sample size of 44. Prior to testing, all participants (or their parent) completed voluntary informed consent. Protocols were approved by La Trobe University Human Research Ethics (HEC18207) conforming to the Declaration of Helsinki.

Oculomotor smooth-pursuits eye-tracking was completed using the EyeGuide Focus (EyeGuide, Texas, USA). The protocol, following EyeGuide protocols, involves the participant seated with their head placed in a chin/head rest, to reduce unnecessary head movements (Figure 1a), and tracking a white dot against a black background moving anti-clockwise, then clockwise through one cycle of a horizontal “lazy 8” or “infinity-shape” path for 10-seconds (Figure 1b). The fixed position digital camera follows the participant’s pupils, at 60 Hz frequency, comparing the position and movement speed of the pupils to the white dot stimulus.

Participants undertook testing in a quiet room, away from potential distractions. Participants were also allowed usual eyewear such as prescription contact lenses or glasses. Each participant completed three trials with a 30-second rest between each trial to avoid potential fatigue. A successful trial was defined as a ‘tick’ indicating that their score (see section 2.1) fell into the EyeGuide community range [17]. If a trial was unsuccessful, that is their result was 1SD above the mean, the EyeGuide would indicate the trial was not successful, and participants were allowed to repeat that trial. If three successful trials were not able to be obtained after an arbitrary 10 attempts, the participant’s data was not included.

2.1. Data and Statistical Analysis

Eye-tracking score was calculated as the cumulative distance between the stimulus data and the pupil position [22]. A greater error was indicated by a larger test score. Participants and test administrators were not blinded to test results, with a classification of scores provided at the end of the test (e.g., “Superior”, “High Average”, “Above Average”, “Average”, “Low Average”, “Impaired”, “Severely Impaired”), however exact test scores were not displayed. Data collected by the EyeGuide system was imported into Jamovi (V2.3, Sydney, Australia). Data was screened for normal distribution using the Shaprio-Wilk tests and found to be normally distributed (W=0.98, p=0.18). Test-retest reliability was calculated via Intraclass Correlation Coefficient (ICC). ICC values were selected using the criteria by Koo and Li [23] : >0.91: excellent reliability, 0.75–0.9: good reliability, 0.51–0.74: moderate reliability, and ≤ 0.5: poor reliability. Data for ICC is presented as value (±95% CI), and comparisons between groups using mean (±SD). Cohen’s d effect sizes were utilized to describe differences between group means [24]. EyeGuide scores are reported in arbitrary units (AUs).

3. Results

All but three adult participants completed testing. In three adult participants data was not able to be collected due to EyeGuide being unsuccessful in tracking pupil movement.

3.1. Reliability Analyses

ICCs for all participants (n=72) showed good reliability across three trials (0.79, 95%CI: 0.70, 0.86). Sub-comparisons between trials 1 and 2, trials 1 and 3, and trials 2 and 3 showed good reliability (ICCs: 0.76 [95%CI: 0.62, 0.84], 0.77 [95%CI: 0.64, 0.85], 0.83 [95%CI: 0.73, 0.89], respectively).

ICCS in all adolescents showed overall good reliability (0.78, 95%CI: 0.62, 0.88) with moderate reliability seen between trails 1 and 2 (0.74, 95%CI: 0.51, 0.87) and between trials 1 and 3 (0.65, 95%CI: 0.35, 0.82). A good reliability was found between trials 2 and 3 (0.81, 95%CI: 0.63, 0.91). When comparing reliability by sex, males showed good overall reliability (0.81, 95%CI: 0.61, 0.82) and across trials 1 and 2, trials 1 and 3, and trials 2 and 3 (0.77 [95%CI: 0.47, 0.91], 0.81 [95%CI: 0.54, 0.92], 0.87 [95%CI: 0.67, 0.95] respectively). In females, overall good reliability was observed (0.76, 95%CI: 0.44, 0.92). A good reliability across trials 1 and 2, and trials 2 and 3 were seen (0.71 [95%CI: 0.22, 0.91], 0.77 [95%CI: 0.35, 0.93] respectively), but poor reliability was seen between trials 1 and 3 (0.48, 95%CI: 0.14, 0.83).

In all adults, ICCS showed overall good reliability (0.80, 95%CI: 0.67, 0.89) and good reliability between trails 1 and 2 (0.79, 95%CI: 0.61, 0.88), trials 2 and 3 (0.82, 95%CI: 0.67, 0.90), and between trials 1 and 3 (0.84, 95%CI: 0.71, 0.91). When comparing reliability by sex, adult males showed good reliability (0.79, 95%CI: 0.60, 0.90), and across trials 1 and 2, trials 2 and 3, and trials 1 and 3 (0.80 [95%CI: 0.58, 0.91], 0.80 [95%CI: 0.50, 0.91], 0.83 [95%CI: 0.63, 0.92] respectively). Similarly in females, overall good reliability was observed (0.86, 95%CI: 0.73, 0.94), and between trials 1 and 2, trials 2 and 3, and trials 1 and 3 were seen (0.86 [95%CI: 0.70, 0.94], 0.86 [95%CI: 0.70, 0.94], 0.84 [95%CI: 0.65, 0.93] respectively).

3.2. Comparisons Across Trials and Groups

A one-way ANOVA for all participants (n=72) showed a significant improvement in scores with subsequent trials (F_(2,140)=28.2, p<0.001; Figure 2). Post hoc comparisons revealed significant differences between trials 1 and 2 (t₍₇₁₎=4.12, p<0.001, d=0.33), trials 1 and 3 (t₍₇₁₎=7.21, p<0.001, d=0.57), and trials 2 and 3 (t₍₇₁₎=t₇₁=3.69, p=0.001, d=0.26).

Comparison between groups revealed an interaction effect between groups (F_(2,140)=10.3, p<0.001; Figure 3). Post hoc testing between groups showed that there was a statistically significant difference for trial 1 (t₍₇₀₎=4.97, p<0.001, d=1.20), but no differences for trial 2 (t₍₇₀₎=2.11, p=0.58, d=0.52) or trial 3 (t₍₇₀₎=2.79, p=0.10, d=0.68).

There was no interaction effect found between sexes across trials (F_(2,140)=0.36, p=0.69; Figure 4). A main effect was observed for trials (F_(2,140)=29.08, p<0.001) with post hoc testing revealing significant difference between trials 1 and 2 (t₍₇₀₎=4.16, p<0.001, d=0.34), and trials 1 and 3 (t₍₇₀₎=7.21, p<0.001, d=0.58), and between trials 2 and 3 (t₍₇₀₎=3.64, p=0.002, d=0.26). No differences were seen between sexes (F_(1,70)=1.38, p=0.24).

Sub analyses between sexes were conducted in adolescent and adult populations separately (Figure 5 and Figure 6 respectively). In the adolescent group (Figure 5) there was no interaction effect found (F_(2,52)=2.55, p=0.09) or main effects between groups (F_(1,26)=0.16, p=0.69). A main effect across trials was observed (F_(2,52)=30.45, p<0.001) with post hoc comparisons showing significant differences between trials 1 and 2 (t₍₂₆₎=5.77, p<0.001, d=0.87), and trials 1 and 3 (t₍₇₀₎=6.55, p<0.001, d=1.04). No difference was seen between trials 2 and 3 (t₍₇₀₎=1.84, p=0.23, d=0.12).

Similarly in the adult group (Figure 6), no interaction effect was observed (F_(2,84)=0.99, p=0.37) or main effect between groups (F_(1,42)=3.14, p=0.08). A main effect across trials was observed (F_(2,84)=11.58, p<0.001) with post hoc comparisons showing significant differences between trials 1 and 3 (t₍₄₂₎=4.78, p<0.001, d=0.36), and trials 2 and 3 (t₍₄₂₎=3.52, , d=0.27). No difference was seen between trials 1 and 2 (t₍₄₂₎=0.98, p=0.99, d=0.08).

4. Discussion

To the best of the authors’ knowledge, this is the first study to examine the reliability of the EyeGuide Focus as a tool for oculomotor pursuits. Overall findings suggest that the EyeGuide Focus was found to have good reliability, but familiarisation effects were observed in both healthy adolescent and adult cohorts with subsequent attempts. Of note, familiarisation effects in adolescents contributed to the moderate reliability in subsequent trials.

Reliability in EyeGuide was comparable to other eye movement cognitive tests (e.g., King-Devick). For example, studies in adolescent populations have reported moderate to good ICCs of 0.81 (95%CI: 0.73, 0.87), and 0.91 (95%CI: 0.86, 0.95) [25,26]; and in adults with good ICCs of 0.95 (95% CI: 0.91, 0.97) and 0.91 (95%CI: 0.80, 0.96) [27,28]. The data in the present study is also consistent to previously published EyeGuide data. In 2017 Kelly [17] reported in adolescents (12 – 18 years) a mean score of 29633.05 (± 9209.83) AUs and this is comparable to our results for trial 1 (30976.57 ± 7099.87 AUs). In adults the mean from trial 1 (22503.33 ± 7014.32 AUs) was equivalent to ‘high average’ to the EyeGuide normative range (21066.64 ± 1858.57 AUs) [17]. Of interest, the baseline range in the current study being classified as ‘high average’ was above the baseline reported, ‘average’ classification, in a cohort of surgeons (no raw data presented however) by Puckett et al [20]. However, in studies by Kelly [17] and Puckett et al [20], only a single baseline trial was recorded.

The finding of familiarization/learning effects in this sample is interesting but not surprising. Well described in cognitive psychology, learning/familiarization effects are noteworthy improvements that occur with repetition of a task until scores no longer notably change and achieve stability [29]. The results in this study concur with previous findings in other eye-movement tests (e.g. King-Devick) with improvements in subsequent trials [30]. Consequently, based upon the results of this study, it is suggested that a minimum of three successful trials are completed with the best score used as the player’s baseline.

The data also showed that, overall, adults performed testing with ~20% less error than adolescents (Figure 3). Similar to studies in other eye-movement tests that show improved performance with age [31], these differences reflect developmental changes in eye-movement and cognition more generally. Mechanisms contributing to differences between adolescents and mature oculomotor systems include longer saccade latencies, reduced smooth pursuit gain, and increased general variability reflecting maturation differences between groups [32,33].

The concern that reliability is affected by improvement in scores, suggesting familiarization effects, should not necessarily be interpreted as a limitation but useful in developing standardized protocols for future use. The diminishing familiarisation effect observed between trials 2 and 3 suggest that three trials should suffice to determine a baseline. Given data is collected within a 10-second time frame, this should not be too onerous on both operator and participant. However, further studies are suggested that could determine the optimal number of baseline trials to reduce familiarization effects, particularly in junior athletes undertaking baseline concussion testing.

Further limitations of this study include the relatively young adult population utilized in this study. We focussed on a younger adult sample to emulate the type of adult population likely to be using EyeGuide Focus (athletic cohorts). However, concussion is not a ‘young person injury’ and indeed many older athletes participate in contact sports and are at risk of concussion. Future studies should consider assessing reliability in an older population cohort.

While previous studies have investigated the effect of workplace fatigue [20] future studies should also look at the effect of acute exercise and fatigue effects on oculomotor smooth pursuit tracking. Other eye-movement tests have found influence of exercise on outcomes [34], and with these systems being suggested as a ‘sideline’ assessment tool to aid medical decisions, it is not yet known the effect of fatigue on eye-movement performance outcomes.

In conclusion this study has demonstrated reliability with the EyeGuide Focus, suggesting that this system is suitable for concussion recognition. However, with familiarization effects apparent, baseline determination in adults and adolescents should employ a standardized protocol of a minimum of three trials.