Dynamic Emotion Recognition and Social Inference Ability in Traumatic Brain Injury: An Eye-Tracking Comparison Study

1. Introduction

Traumatic brain injury (TBI) is a global public health concern with approximately 69 million new cases each year [1]. The long-term sequelae of TBI include physical (i.e. headaches), cognitive (i.e. executive function, concentration), affective (i.e. mood disorders), and social cognition (i.e. diminished theory of mind) impairments [2]. Importantly, social cognition impairments are documented across all severities of TBI [3,4], and diminished abilities are associated with poor patient outcomes [5-7] and negatively impact long-term family functioning and wellbeing [8]. Despite the likelihood that social cognition impairments after TBI are underreported [9,10], the evidence base for their prevalence is strong [11-14]. One prominent area of disruption post-TBI is facial emotion recognition [11,14-16], with estimates that up to 39% of individuals with moderate-severe TBI may exhibit this impairment [17]. This is highly problematic as facial expressions are one of the richest tools for social inference and communication [18]. Additionally, individuals with TBI display difficulties in interpreting social cues such as tone of voice and gestures which impede the understanding of other people’s feelings, thoughts, and intentions [14,19,20].

Although emotion recognition impairments are well documented post-TBI, the mechanisms underpinning these changes are not fully understood. Specifically, it is unknown whether they stem from disruption to early visual processing (i.e. aberrant eye fixation patterns) or social cognition, or a combination of both. An association between eye movement dysfunction and poor emotion recognition has been documented for neurological disease and injury [21-27]. Although research on TBI populations is scant, it is unsurprising that eye fixation patterns are disrupted as, anatomically, several frontal brain areas modulating eye movements are susceptible to pathology due to their position within the skull and the trajectory of the brain on impact [28,29]. Furthermore, emotion and social inference recognition are complex processes, requiring several synchronised visual strategies (e.g. saccades, fixations) to the body and face. Complexity is increased as different emotional expressions elicit distinct fixation scan path patterns, for example, angry and sad faces produce earlier and longer fixations to the eye region compared to happy faces where the mouth receives more attention [30-33]. Unfortunately, the emotion recognition field has been hampered by an experimental bias towards behavioural data with a paucity of studies exploring visual strategies of real-time physiological data. Furthermore, there has been a growing discussion regarding the poor ecological validity of frequently implemented static facial expression stimuli and a call for increased use of dynamic stimuli [11,34]. The current study had three aims:

(1)

To explore dynamic emotion recognition and social inference abilities between adults with and without TBI

(2)

To determine if adults with TBI exhibited different fixation patterns compared to adults without TBI in response to dynamic social interactions

(3)

To investigate relationships between fixation duration and fixation counts to AOI and emotion recognition and social inference accuracy scores

2. Materials and Methods

2.1. Participants

The study included 18 TBI participants and 18 matched non-TBI participants (N = 36). Due to a dataset error, there was one less non-TBI participant for the social inference minimal (SI-M) and social inference enhanced (SI-E) components (N = 35). Ethical approval was obtained through Leeds East NHS Research Ethics Committee and Sheffield Hallam University Faculty Research Ethics. TBI participants were recruited through two brain injury rehabilitation services within the UK National Health Service. The non-TBI participants were recruited via stratified opportunity and snowballing sampling and matched to the TBI participants for sex, age, and education. All participants provided informed consent and completed all three components of The Assessment of Social Inference Test (TASIT)[14] whilst undergoing eye tracking.

TBI information was acquired from medical records, including scan data and hospital admission notes. Participant pathology was heterogeneous, including brain haemorrhage, skull fracture, and contusion, which is typical for this patient cohort [22]. Mechanisms of injury included assault, road traffic accidents, falls, and pedestrian collisions. Thirteen participants had frontal lobe pathology, five had pathology outside of frontal cortices to other cortical and/or subcortical regions (see Appendix A). As anticipated, we recruited more males than females which is in line with existing evidence that males are more likely to sustain a brain injury compared to females [23,24]. Years of education did not differ between groups but the group with TBI fell significantly below the non-TBI group for all three components of the IQ test. This is a pattern frequently reported in the brain injury literature [18,19]. The non-TBI group all reported as neurotypical with no history of brain injury or concussion. See Table 1 for the demographic and clinical characteristics of TBI participants and non-TBI groups.

2.1.1. Inclusion/exclusion criteria

TBI inclusion criteria

TBI sustained in adulthood is typically determined by assessment at hospital admission or brain pathology based on imaging scans.

• Participants were required to be at least one-year post injury to ensure that chronic rather than acute effects of brain injury were measured
• Aged between 18 and 65 to account for any effects of natural aging

Exclusion criteria for all participants

• History of psychiatric illness
• Severe recent drug and alcohol abuse assessed by Michigan Alcohol Screening Test (MAST) [25] and Drug Abuse Screening Test (DAST) [26]
• Significant depression and anxiety using the Hospital Anxiety and Depression Scale (HADS) [27]
• Visual deficits using the Visual Object and Space Perception Battery (VOSPB) [28] Warrington & James, 1991), and Cortical Vision Screening Test (CORVIST) [29]

None of the participants had visual defects measured by screening tasks and none met the criteria for severe anxiety or depression. However, the TBI group did have a higher overall HADS score (anxiety M = 8.11, SD = 4.61, depression M = 6.00, SD = 3.31, overall HADS M = 14.11, SD = 6.28); matched non-TBI-group (anxiety M = 4.94, SD = 2.04, depression M = 2.67, SD = 1.94, overall HADS M = 7.61, SD = 3.48), but this pattern is not unusual in post-TBI populations, even if the injury is mild [30,31].

2.1.2. Injury severity

Currently, there are two primary brain injury severity measurements, post-traumatic amnesia (PTA) and loss of consciousness (LOC) based on the Glasgow Coma Scale (GCS) [32]. PTA can be defined as the period between a TBI occurring and the patient regaining full awareness and standard memory function [33]. PTA less than 24 hours is indicative of mild TBI, greater than 24 hours but 7 days or less moderate and PTA longer than seven days indicates severe injury. This study cohort included 16 severe, one moderate, and one mild TBI. The GCS assesses motor, verbal, and eye responses providing a score between 3 and 15. Scores between 3 and 8 indicate severe injury, between 9 and 12 moderate and between 13 and 15 mild injuries. Collaborating referring clinicians determined severity of injury rating based on standard procedures – LOC and PTA.

2.2. Design

A quasi-experimental design was utilised to compare social cognition abilities of participants with and without TBI. A Tobii T120 Eye Tracker [34] recorded participants eye movement metrics including (1) time to first fixation, (2) first fixation duration, (3) total fixation duration, and (4) total fixation count. Three areas of interest (AOI) were chosen for the study, the eyes, nose, and mouth of the actors in the TASIT task.

2.2.1. Stimuli and procedure

The TASIT [14] comprises video vignettes of professional actors engaged in social interactions. The actors were employed to use a ‘method style’ of acting where they were required to demonstrate an emotion relevant to the situation. This method provided a realistic, spontaneous, and natural test context. The TASIT includes three sub-component measures: the emotion evaluation test (EET), the social inference-minimal test (SI-M), and the social inference-enriched (SI-E) test. All three sub-components of the TASIT were administered during the study. In brief, the EET assessed the understanding of communication components such as facial expressions, tone of voice, and gestures. The SI-M provided a measure of the participant’s ability to read paralinguistic features, facial expressions, and tone of voice in either sincere or sarcastic situations. Finally, the SI-E assessed the participant’s ability to use contextual cues (i.e. tone of voice and facial expressions) to determine if everyday conversations were deceptive or sarcastic. The primary behavioural score for each component was accuracy scores (i.e. identifying the correct emotion or expression). A more detailed description of the TASIT sub-components can be found in Appendix B. Only correct responses were included in all analyses. There was one practice trial at the beginning of each sub-component and participants verbally indicated their response which the researcher documented.

The TASIT is a superior social inference task as it includes naturally occurring visual cues during social situations (demeanour of the speaker and reaction of the listener) which many other experimental inference tasks do not. For example, while the Movie for the Assessment of Social Cognition (MASC) [36] includes dynamic social cues, it does not explore the processing of sincere and sarcastic exchanges in as much depth as the TASIT. It has been reported that the TASIT has good reliability and high test-retest reliability levels (r = 0.74-0.88) as well as good construct validity, low practice effects, and high ecological validity [14,37]. The TASIT is gaining traction in brain injury research [38-40].

2.2.2. Apparatus Eye-Tracker

A Tobii T120 eye-tracker and Tobii Studio Eye Tracking Analysis software [41] was used to collect and analyse eye-tracking data. The eye-tracker has a 17-inch thin-film-transistor screen (1280 x 1024 pixels) with an embedded infrared camera. The technology does not have a head frame ensuring participants are free to exhibit natural head and eye movements. Eye tracking data was sampled at 120 Hz with an accuracy of 0.5°. The default Tobii fixation filter algorithm was used for all three components of the TASIT (fixation threshold at 35 pixels for velocity and 35 pixels for distance per sample). An eye movement velocity above the 0.5° per second threshold was classified as a saccade sample and below it was classed as a fixation. The standard five-point calibration of each eye was undertaken for each component of the TASIT for each participant. Participants were seated approximately 62 cm away from the screen in a stationary chair.

2.2.3. Eye tracking metrics

Eye tracking metrics were calculated by the Tobii software [41]. Those analysed were ‘total fixation duration (seconds)’ which was the sum of all fixations within an active area of interest (AOI) e.g. how long the participant spent looking at the eyes, the nose and the mouth; and ‘fixation count (count)’ which indicates the number of times the participant fixates on an active AOI. Fixation counts are amalgamated throughout the experiment as participants redirect their fixations back and forth between AOIs.

3. Results

Parametric assumptions were checked for raw data. The behavioural data was relatively normally distributed but the eye-tracking data had moderate violations of normality regarding skewness and kurtosis, homogeneity of variance, and outliers. As violations were moderate and not severe, and because the study had approximately equal sample sizes, the effects of violation were thought minimal [42]. Furthermore, to guard against the correction of accurate but ‘non-normal’ data and to avoid the challenges associated with transforming skewed data parametric analyses were conducted on untransformed data [43]. Although the present study recruited similar participant numbers to previous dynamic social inference research in different populations [44,45], the sample size is still modest and, as such, the α for main inferential analyses was set at .05 with and adjusted α = .01 for all post-hoc follow-up analyses (α = .01).

3.1. Emotion Evaluation Test

For the EET, a 2 (TBI vs. no TBI group)*(7 - emotion) ANOVA was conducted to explore potential differences in emotion perception. A further 2*(7) *(3 – eyes, nose, mouth AOI) repeated measures ANOVA was conducted to investigate potential differences in fixation duration and count between the TBI and non-TBI groups.

3.1.1. Behavioural data

As demonstrated in Table 2, the descriptive statistics indicated that the group with TBI had lower overall accuracy scores on the EET compared to the group without TBI.

The analysis showed a significant main effect of emotion, (F (6, 204) = 6.85, p ≤ .001, ηp2 = 0.17), but the interaction between emotion and group was non-significant, (F (6, 204) = 1.08, p = .375, ηp2 = 0.03). The test of between-subjects effects was also significant, (F (1, 34) = 20.28, p ≤ .001, ηp2 = 0.37) with the group means indicating that the TBI participants had significantly fewer correct responses across the EET compared to the non-TBI group. When comparing the descriptive statistics, it appeared that the TBI group scored lower on negative emotions compared to positive, particularly interpreting sad, anxious, and revolted displays of emotion, compared to the non-TBI group, although these differences were not statistically significant.

Eye-tracking data

Fixation duration across the EET in seconds

The group with TBI had shorter fixation durations to the eyes and nose compared to participants without TBI, while both groups displayed similar fixation durations to the mouth (Table 3).

The analysis showed a significant main effect of emotion, (F (1.83, 62.17) = 38.99, p ≤ .001, η_p² = 0.53), and a significant interaction between emotion and AOI, (F (4.19, 142.35) = 12.29, p ≤ .001, η_p² = 0.27). The main effect of AOI, (F (1.70, 57.87) = 0.09, p = .884, η_p² = 0.003), and the interactions between emotion and group, (F (1.83, 62.17) = 1.67, p = .198, η_p² = 0.05), AOI and group, (F (1.70, 57.87) = 2.17, p = .130, η_p² = 0.06), and emotion, AOI and group, (F (4.19, 142.35) = 1.26, p = .289, η_p² = 0.04) were all non-significant. The tests of between-subjects effects was also not significant, (F (1, 34) = 2.84, p = .101, η_p² = 0.08).

The significant interaction between emotion and AOI was explored through three one-way ANOVAs, with emotion as the independent variable (IV) and fixation duration as the dependent variable (DV). There was a significant effect of eyes, (F (1.40, 48.90) = 36.35, p ≤ .001, η_p² = 0.51), nose, (F (2.53, 88.41) = 5.99, p = .002, η_p² = 0.15) and mouth, (F (1.40, 48.90) = 36.35, p ≤ .001, η_p² = 0.51) on fixation duration in response to different facial expressions, with follow-up paired-samples t tests in Appendix C. In brief, the post-hoc tests indicated that participants made longer fixations to the eyes when viewing sad faces compared to all other emotions. Participants also spent significantly longer looking at the eyes of neutral faces compared to the other emotions (bar sad). The group elicited longer fixations to the nose when viewing angry, sad, and neutral faces and more fixations to the mouth when looking at happy and sad faces.

Fixation count across the EET

The group affected by TBI had fewer fixations compared to those without TBI across all of the emotions and across most of the AOIs within the emotions (Table 4).

The analysis showed a significant main effect of emotion, (F (3.40, 115.65) = 41.78, p ≤ .001, η_p² = 0.55) and a significant interaction between emotion and AOI, (F (5.85, 198.88) = 10.77, p ≤ .001, η_p² = 0.24). The main effect of AOI, (F (1.60, 54.41) = 0.44, p = .602, η_p² = 0.01) and the interactions between emotion and group, (F (3.40, 115.65) = 1.79, p = .146, η_p² = 0.05), AOI and group, (F (1.60, 54.41) = 2.26, p = .124, η_p² = 0.06), and emotion, AOI and group, (F (5.85, 198.88) = 1.88, p = .088, η_p² = 0.05) were all non-significant. The tests of between-subjects effects was also not significant, (F (1, 34) = 2.96, p = .094, η_p² = 0.08).

The significant interaction between emotion and AOI was explored through three one-way ANOVAs, with emotion as the IV and fixation count as the DV. There was a significant effect of eyes, (F (1.40, 48.90) = 36.35, p ≤ .001, η_p² = 0.44), nose, (F (4.17, 146.02) = 5.87, p ≤ .001, η_p² = 0.14) and mouth, (F (3.94, 138.06) = 30.97, p ≤ .001, η_p² = 0.47) on fixation count in response to different facial expressions, with follow-up paired-samples t tests presented in Appendix D. In summary, the group made more fixations to the eyes and nose when viewing neutral and sad faces. The mouth was looked at more when participants viewed happy and sad faces.

Correlations

One-tailed Spearman’s Rho correlations were run to explore possible relationships between the behavioural data (accuracy for emotion recognition) and eye-tracking data (fixation duration and fixation count). When the groups were combined and the α-level had been adjusted to account for multiple comparisons (α= .01) there were no significant correlations (all, r's ≤ 0.29, all p’s ≥ .043). When the groups were separated there were also no significant correlations for the group with TBI (all r’s ≤ -0.23, p ≥ .352) or non-TBI group (all r’s ≤ 0.58, p ≥ .013).

Social Inference-Minimal

A 2 (TBI vs. no TBI group)*(3 – simple sarcasm, paradoxical sarcasm, sincere) ANOVA was conducted to investigate group differences during the understanding of conversational meanings. A further 2*(4 – intentions, meaning, beliefs, feelings) ANOVA was conducted to investigate the difference between the groups in understanding different facets of social interactions. With eye-tracking data, a 2*(3) *(3) mixed-design ANOVA was conducted to investigate potential differences in fixation duration and count between the TBI and control groups.

Behavioural data

Accuracy for conversation style

As demonstrated in Table 5, the descriptive statistics indicated that the TBI group had lower overall accuracy scores across the conversational styles (simple sarcasm, paradoxical sarcasm and sincere) compared to the control group.

There was no significant effect of conversation style, (F (1.59, 52.47) = 1.37, p = .260, η_p² = 0.04) and no interaction between conversation style and group, (F (1.59, 52.47) = 1.63, p= .203, η_p² = 0.05). The tests of between-subjects effects was significant, (F (1, 33) = 21.03, p ≤ .001, η_p² = 0.39) and referring to the estimated marginal means, the group with TBI scored significantly lower across the three conversation styles of the SI-M (M= 15.54, SD= 3.65) compared to the control group (M= 18.55, SD= 1.94).

Accuracy

for comprehension probes

The descriptive statistics for the four different comprehension probes (beliefs, meanings, intentions, feelings) suggested that the group affected by TBI were less accurate at understanding what the actor was trying to do, what they were trying to say, what they were thinking, and what they were feeling (Table 6). See Appendix B for a comprehensive description of the TASIT behavioural metrics.

There was a significant effect of comprehension probe, (F (2.38, 78.36) = 3.19, p = .038, η_p² = 0.09) but there was no interaction between comprehension probe and group, (F (2.38, 78.36) = 1.37, p= .261, η_p² = 0.04). To further explore the significant effect of comprehension probe, independent t tests were conducted. When post-hoc correction was applied, the intentions, meaning, beliefs, and feelings probes were significant (p ≤ .001). The tests of between-subjects effects was significant, (F (1, 33) = 17.67, p ≤ .001, η_p² = 0.35) and referring to the estimated marginal means, the group with TBI scored significantly lower across the four comprehension probes of the SI-M (M= 11.67, SD= 2.29) compared to the control group (M= 13.84, SD= 1.00).

Eye-tracking data

Fixation duration across the SI-M in seconds

The mean scores indicated that participants spent more time fixated on the mouth than the nose and eyes during simple sarcasm and sincere videos (Table 7). The sincere videos appeared to elicit the longest fixations, followed by the sarcastic and then paradoxical videos.

The analysis showed a significant main effect of conversational style, (F (1.34, 44.16) = 23.59, p ≤ .001, η_p² = 0.42) and AOI, (F (2, 66) = 6.29, p = .003, η_p² = 0.16). The interactions between conversation style and group, (F (1.34, 44.16) = 0.25, p = .78, η_p² = 0.01), AOI and group, (F (1.78, 58.82) = 0.39, p =0.657, η_p² = 0.01), conversation style and AOI, (F (2.39, 78.98) = 2.62, p = .070, η_p² = 0.07), and conversation style, AOI and group, (F (2.39, 78.98) = 0.15, p =.894, η_p² = 0.01) were all non-significant. The tests of between-subjects effects was also not significant, (F (1, 33) = 0.52, p = .476, η_p² = 0.02).

To explore the significant main effect of conversation style, two-tailed paired samples t tests were conducted and the post-hoc correction was applied. Participants generated significantly longer fixation durations during the sarcastic videos compared to paradoxical videos, (t (34) = 5.67, p ≤.001, d = 0.96, CI= 0.61-1.30), sincere videos compared to sarcastic videos, (t (34) = -3.00, p =.005, d = 0.51, CI= -1.82- -0.35), and sincere videos compared to paradoxical videos, (t (34) = -6.50, p ≤.001, d = 1.09, CI= -2.68- -1.41).

Additional two-tailed paired-samples t tests were conducted to investigate the significant main effect of AOI. When the post-hoc correction was applied, participants exhibited significantly longer fixations to the mouth (M =2.30, SD= 1.69) compared to the eyes (M =1.13, SD= 1.07), (t (34) = -3.24, p = .003, d = 0.55, CI= -1.90- -0.44).

Fixation count across the SI-M

From the mean scores presented in Table 8, there appeared to be a similar pattern for both the TBI and control participants, with both groups having higher fixation counts for the sarcastic and sincere compared to paradoxical videos.

The analysis showed a significant main effect of conversational style, (F (1.39, 45.91) = 22.40, p ≤ .001, η_p² = 0.40). The other main effect of AOI, (F (1.69, 55.77) = 2.96, p = .068, η_p² = 0.08), and the interactions between conversation style and group, (F (1.39, 45.91) = 0.37, p = .617, η_p² = 0.01), AOI and group, (F (1.69, 55.77) = 0.38, p = .648, η_p² = 0.01), conversation style and AOI, (F (2.26, 74.71) = 2.40, p = .091, η_p² = 0.07), and conversation style, AOI and group, (F (2.26, 74.71) = 0.44, p =.671, η_p² = 0.01) were all non-significant. The tests of between-subjects effects was also not significant, (F (1, 33) = 0.002, p = .965, η_p² ≤ 0.001).

To explore the significant main effect of conversation style, two-tailed paired-samples t tests were conducted with post-hoc corrections applied. Participants generated significantly more fixations during the sarcastic videos compared to paradoxical videos, (t (34) = 7.24, p ≤.001, d = 1.22, CI= 1.83-3.26), and to sincere (M = 5.73, SD = 3.41) videos compared to paradoxical videos, (t (34) = -6.30, p ≤.001, d = 1.06, CI= --4.97- -2.54).

Correlations

One-tailed Spearman’s Rho correlations were used to investigate potential relationships between the behavioural data (accuracy for conversation style and comprehension probe) and eye-tracking data (fixation duration and fixation count). When the groups were combined and once the α-level had been adjusted to account for multiple comparisons (α= .01), there were no significant correlations (all, r's ≤ 0.001, all p’s ≥ .110). When the groups were separated, there were still no correlations for the group with TBI (all r’s ≤ 0.51, p ≥ .032) but there were significant correlations for the control group between fixation duration to the eyes and the intentions comprehension probes (r= 0.74, p= .001), fixation count to the eyes and the intentions comprehension probes (r= 0.73, p= .001) and between fixation duration to the eyes and simple sarcasm comprehension probes (r= 0.69, p= .002).

SI-E

A 2 (TBI vs. no TBI group)*(2 – sarcastic and lie) ANOVA was conducted to investigate group differences during the understanding of conversational meanings. A further 2*(4 – intentions, meanings, beliefs and feelings) ANOVA was conducted. Again, this analysis investigated the difference between the groups in understanding different facets of social interactions. A 2*(2) *(3 – eyes, nose and mouth AOI) repeated measures ANOVA was conducted to investigate potential differences in fixation duration and count between the TBI and control groups.

Behavioural data

Accuracy for conversational style

As demonstrated in Table 9, the mean scores indicated that the group affected by TBI had lower overall accuracy scores across the two conversational styles (sarcastic and lie) compared to the control group.

The effect of conversation style was non-significant, (F (1.00, 33.00) = 0.64, p = .428, η_p² = 0.02) but there was a significant interaction between conversation style and group, (F (1.00, 33.00) = 9.86, p= .004, η_p² = 0.23). Simple effects analysis revealed that the group with TBI scored significantly lower on the sarcastic vignettes (M = 22.56, SD = 4.77) compared to controls (M = 29.41, SD = 3.37) (p ≤ .001). The tests of between-subjects effects was also significant, (F (1, 33) = 16.31, p ≤ .001, η_p² = 0.33). Referring to the estimated marginal means, the group affected by TBI had significantly lower accuracy scores across the two conversation styles of the SI-E (M= 23.94, SD= 4.78) compared to the control group (M= 28.58, SD= 3.24).

Accuracy for comprehension probes

The descriptive statistics for the four different comprehension probes (intentions, meanings, beliefs, and feelings) suggested that the group living with TBI were less accurate at understanding what the actor was trying to do, what they were trying to say, what they were thinking, and what they were feeling (Table 10).

There was a significant effect of comprehension probe, (F (2.46, 81.19) = 3.25, p = .034, η_p² = 0.09) and a significant interaction between comprehension probe and group, (F (2.46, 81.19) = 4.69, p= .008, η_p² = 0.12). Simple effects analysis revealed that the group with TBI scored significantly lower on the ‘intention’ probes (M = 11.28, SD = 2.49) compared to controls (M = 14.82, SD = 1.63) (p ≤ .001) and the same for the ‘meaning’ probes (TBI M = 11.17, SD = 2.60; control group M = 14.47, SD = 1.70) (p ≤ .001). The tests of between-subjects effects was also significant, (F (1, 33) = 21.41, p ≤ .001, η_p² = 0.39). Referring to the estimated marginal means, the group affected by TBI scored significantly lower across the four comprehension probes of the SI-E (M= 11.97, SD= 2.34) compared to the control group (M= 14.47, SD= 1.63).

Eye-tracking data

Fixationduration across the SI-E in seconds

Table 11 shows that individuals with TBI had shorter fixation durations to the eyes, nose and mouth compared to controls (Table 43).

The analysis showed a significant main effect of conversational style, (F (1, 33) = 44.18, p ≤ .001, η_p² = 0.57) with participants exhibiting longer fixations for sarcastic videos compared to lie videos. The main effect of AOI, (F (1.22, 40.16) = 0.92, p = .405, η_p² = 0.03) and the interactions between conversation style and group, (F (1.00, 33.00) = 0.002, p = .969, η_p² ≤0.01), AOI and group, (F (1.22, 40.16) = 0.01, p = .991, η_p² ≤ 0.01), conversation style and AOI, (F (1.61, 53.26) = 2.54, p = .099, η_p² = 0.07), and conversation style, AOI and group, (F (1.61, 53.26) = 1.42, p =.249, η_p² = 0.04) were all non-significant. The tests of between-subjects effects was also not significant, (F (1, 33) = 1.13, p = .296, η_p² = 0.03).

Fixation count across the SI-E

From the descriptive statistics presented in Table 12, the TBI cohort appeared to produce fewer fixation counts to all AOI across the sarcastic and lie conditions compared to control participants. Both the TBI and control participants generated fewer fixation counts during the videos depicting lies compared to videos where the actors were being sarcastic.

The analysis showed a significant main effect of conversational style, (F (1, 33) = 75.77, p ≤ .001, η_p² = 0.70) with participants displaying more fixations during the sarcastic vignettes compared to vignettes depicting lies. The main effect of AOI, (F (1.36, 44.95) = 0.28, p = .673, η_p² = 0.01) and the interactions between conversation style and group, (F (1, 33) = 0.08, p = .780, η_p²= 0.002), AOI and group, (F (1.36, 44.95) = 0.20, p = .732, η_p² = 0.01), conversation style and AOI, (F (1.58, 52.23) = 1.29, p = .279, η_p² = 0.04), and conversation style, AOI and group, (F (1.58, 52.23) = 1.49, p =.236, η_p² = 0.04) were all non-significant. The tests of between-subjects effects was also not significant, (F (1, 33) = 0.76, p = .390, η_p² = 0.02).

Correlations

One-tailed Spearman’s Rho correlations explored the relationships between the behavioural data (accuracy for conversation style and comprehension probe) and eye-tracking data (fixation duration and fixation count). When the groups were combined and once the α-level had been adjusted to account for multiple comparisons (α= .01), there were no significant correlations (r ≤ 0.39, p ≥ .022). When the groups were separated, there were still no significant correlations for the group with TBI (all, r's ≤ 0.56, all p’s ≥ .016) or controls (all, r's ≤ 0.56, all p’s ≥ .019). Although the analysis did not reach the .01 set for post-hoc significance, the results are nearing significance and are significant at the typically accepted α level of .05.

4. Discussion

We compared the emotion recognition and social inference abilities of participants with TBI to those without TBI. As expected, the TBI group were significantly less accurate in identifying emotions on the EET than those without TBI. In line with our previous work[20], there was a distinct deficit for negative compared to positive emotions, particularly interpreting sad, anxious, and revolted displays of emotion. Individuals with TBI also scored significantly lower across all three of the conversation styles of the SI-M, in comparison to the control group, with simple-sarcasm vignettes scoring lowest followed by sincere, and then paradoxical-sarcastic exchanges. This effect was mirrored in the SI-E where the group with TBI were significantly poorer at recognising sarcastic conversations compared to those without TBI. Interestingly, there was no significant difference between the two groups during the lie vignettes. These findings are consistent with McDonald and colleagues who reported that participants with TBI displayed impairments in deciphering sarcastic exchanges but scored in a typical range when interpreting lies [14]. These results imply that while TBI diminishes the ability to comprehend sarcasm, the aptitude to recognise lies remains intact. This model has previously been proposed by McDonald [46] and is based on the conceptual differences between lies and sarcasm. For example, it has been proposed that understanding sarcasm requires more cognitive effort compared to understanding sincere and untruthful exchanges, mainly because understanding sarcasm necessitates first-order theory of mind (ToM; understanding what someone is thinking or feeling) and second-order ToM (predicting what one person thinks or feels about what another person is thinking or feeling) [47]. Analysis of the comprehension probes on both the SI-M and SI-E also demonstrated that the group with TBI had difficulty understanding the actor’s intentions, feelings, beliefs, and the meaning of the conversation compared to the group without TBI. In summary, these results align with prior research that highlights disruptions in emotion recognition and social inference following a traumatic brain injury (TBI) [11,14-16,40,48]. However, our study contributes to the field by pioneering an investigation into the potential involvement of visual impairments in this issue, utilising ecologically valid assessments.

There were no significant differences between the groups in terms of eye fixation patterns and no associations between fixation patterns and behavioural accuracy scores. This differs from previous studies that have reported TBI groups exhibiting abnormal fixation patterns compared to non-TBI groups [24-26]. Interestingly, these studies utilised static stimuli which have been criticised for potential low ecological validity, for instance, lack of motion, depth cues, and contextual cues [23,27,49-52]. Evidence has demonstrated that static and dynamic stimuli activate different brain networks [53,54] as well as different levels of attention. For example, Kujawa and colleagues reported that individuals with impaired consciousness resulting from brain injury exhibited higher visual attention to dynamic compared to static stimuli [55]. Others have also shown that eye fixation patterns differ in normative populations in response to dynamic and static stimuli [56,57]. It appears that when more real-world dynamic social stimuli are implemented, there are no group dissimilarities in terms of eye fixation patterns between those with and without TBI, a finding we replicated in a previous study [23]. These results might suggest that the challenges in everyday emotion recognition and social inference following a TBI aren't necessarily linked to visual alterations. Nevertheless, in the present study, we did observe distinctions between our TBI and non-TBI groups at a descriptive level. Additionally, considering previous reports on abnormal eye fixation patterns post-TBI in response to static stimuli, it may be rash to entirely dismiss the possibility that visual changes might exert some influence on social cognition. It seems that there is a multifaceted relationship between vision and social cognition after TBI, necessitating further investigation, potentially with larger sample sizes and increased statistical power to detect subtle group differences. Currently, it remains unclear what is driving these social impairments but, speculatively, they may be attributed to a combination of low-level visual functions and higher-order functions such as working memory load [48], education and/or executive functioning [40].

There was a significant interaction between AOI and emotion during the EET with visual strategies appearing to reflect attention to the most diagnostic area of the face for each emotion (e.g. mouth for identifying happiness and nose for recognising disgusted faces). These findings indicate a goal- and stimulus-driven influence on eye gaze patterns, as distinct patterns of fixation count and duration were observed for both emotional and neutral faces and for conversational styles. This finding is in line with previous research [31-33]. During the SI-M, all participants generated longer fixation durations and counts during the sarcastic videos compared to paradoxical and sincere compared to paradoxical videos. For the group without TBI, there was a correlation between fixation duration to the eyes and intentions, fixation count to the eyes and intentions, and between fixation duration to the eyes and simple sarcasm. Recognising sincere, sarcastic, and deceptive communication requires the integration of verbal, non-verbal, and paralinguistic cues and is paramount to everyday interactions. Despite this, previous research exploring this complex area has heavily relied on written and static stimuli. Our findings conflict with existing research which reported that individuals had longer fixation durations to nonliteral compared with literal interactions [58]. Gaining a deeper understanding of typical literal and non-literal language perception can inform existing theoretical frameworks as well as aid clinical rehabilitation areas, such as TBI, where these skills are frequently diminished [59].

The primary limitation of the present research was the relatively small sample size. Future work might include larger samples, with more female participants, or more conservative alpha levels during significance testing. Demographically, TBI and non-TBI groups were matched for age, sex, and years of education, but those with TBI fell below the non-TBI group on IQ measures. This is typical for TBI populations [60] but if social impairments post-TBI are related to working memory deficits then future studies could control for this.

5. Conclusions

Our results demonstrate two things. First, eye fixation patterns for dynamic social stimuli do not statistically differ between individuals with and without TBI but do differ on a descriptive level, and second, dynamic emotion recognition and social inference abilities are impaired after TBI. These combined findings seem to indicate that impairments are not related to abhorrent eye fixation patterns and must be driven by a disruption outside of the low-level visual system. Our research also highlights the important distinction between static and dynamic stimuli when trying to map structure to function in eye-tracking research and we urge other research teams to implement dynamic stimuli in the future.