Content Moderator Mental Health and Associations with Coping Styles: Replication and Extension of Previous Studies

1. Introduction

Content moderators (CMs) are the professionals who analyse and remove online content that has been reported by users or artificial intelligence systems as potentially harmful, illegal or breaking the platform’s policies and guidelines (Caplan, 2018; Gerrard, 2022). They are required to work across cultural contexts and understand, what can at times be, coded nuances to make decisions about the suitability of content (Gillespie, 2018; Roberts, 2019), these decisions often need to be made within approximately 10-30 seconds (Barrett, 2020; Common, 2020), sometimes thousands of times over any given shift (Barrett, 2020; Newton, 2019). Therefore, alongside banal content such as spam, CMs can be exposed to high levels of graphic images, videos and text that feature hate speech, violence and sexual abuse material (Steiger et al., 2021).

There is a large evidence base for more established professions such as police and social workers that demonstrates exposure to the suffering of others can result in detrimental mental health outcomes such as secondary traumatic stress, vicarious trauma, depression and anxiety (e.g., Bride, 2007; Caringi et al., 2017; Foley et al., 2022; Hurrell et al., 2018). Alongside mental health aspects, workers may also experience burnout (Frieiro Padín et al., 2021; Queiros et al., 2020), as well as increased sickness rates and compassion fatigue (Körlin et al., 2009; Ondrejková & Halamová, 2022). A systematic review by Iversen and Robertson (2021) found there was a high prevalence of secondary trauma in legal professionals who were exposed to clients who have experienced trauma. They also found having a personal history of trauma and higher levels of exposure were associated with secondary trauma.

Alongside this literature there is a growing evidence base which outlines the potential harm that CMs can also experience. Studies with volunteer CMs, those who moderate community platforms like Twitch or Reddit without payment (Caplan, 2018), find it is common to experience harassment and abuse (Blackwell et al., 2018; Lo, 2019). They often report experiencing feelings of under-appreciation and guilt (Wohn, 2019), as well as burnout from the amount of work and their exposure to harmful content, some of which affects their offline experiences (Seering et al., 2019) and can result in people leaving moderation (Lo, 2019; Schopke-Gonzalez et al., 2022). Commercial CMs, those who are paid to moderate platforms, also experience a range of issues from PTSD symptoms (Parks, 2019; Roberts, 2019) to more common mental health problems such as anxiety and depression (Steiger et al., 2021). In a study by Spence et al., (2023) CMs reported experiencing intrusive thoughts and sleep disturbance, as well as feelings of cynicism, desensitisation and detachment.

Nevertheless, there has been growing recognition of what has been described as a replication crisis in the social sciences (e.g. Tackett et al., 2017; Wiggins et al 2019). The assumption of psychological studies is that most statistically significant findings can be replicated using new data (Shrout & Rodgers, 2018). This ability to reproduce the results in novel datasets then suggests the results are both reliable and generalisable rather than due to some methodological or sampling quirk (Stanley et al., 2018). Theories can then be developed based on the empirical patterns of results taken from numerous studies, enhancing knowledge construction and avenues for future research, which further accumulates new knowledge (ap Sion & Earp, in press).

Ideally, any insights should be confirmed using new data and this is no less true for trust and safety research. Especially given that moderators are a worldwide workforce and their working conditions vary considerably, from volunteers to direct employees to those employed by outsourced third party vendors (Caplan, 2018; Roberts, 2019). However, a significant amount of research occurs and remains inside private companies. This can mean instead of working collaboratively towards industry-wide best practices on topics like trust and safety, research tends to be somewhat internally-focused and it is unknown to what degree results may be industry-wide rather than company specific. Nevertheless, there are good reasons why companies may want research to stay private, for example disclosures made through describing the research process or environment may benefit a company’s competition or create negative publicity. For example, in 2014 Facebook released the results of their emotional contagion study where users’ feeds had been manipulated to be either positive or negative to explore the emotional impact of Facebook (Kramer et al., 2014). However, this study sparked a wave of publish backlash and criticism for breaching ethical guidelines (Arthur, 2014). Furthermore, results may open up companies to legal challenges. This could be particularly problematic for studies that investigate employee mental health. For example, a Spanish court ruled a CM’s mental health condition was due to work-related issues (Faus & Demony, 2024) and Meta settled in a court case where they were accused of failing to protect CMs from psychological injuries resulting from exposure to graphic content (Wiessner, 2021).

Previous studies by Spence et al., (2024a, 2024b) have found high levels of psychological distress, with over a third (34.6%) of CMs surveyed scoring in the moderate to severe range of the Core-10 and almost half (47.6%) scoring 13 or over, which is associated with having clinical depression (Barkham et al., 2013). They also found over a quarter (27.4%) scored in the low wellbeing range of the Short Warwick-Edinburgh Mental Wellbeing Scale (SWEMWBS; Tennant et al., 2007). There was also a dose response effect between frequency of exposure and psychological distress and secondary trauma, with daily exposure associated with greater scores than exposure occurring less than monthly. An exploration of coping strategies found the majority used approach-based coping, and that this was associated with a reduction in psychological distress and secondary trauma and greater wellbeing, whilst avoidant coping was associated with the opposite. Although both approach and avoidant coping were measured through behavioural indicators such as smoking or talking to colleagues rather than a standardised measure of coping. Additionally, they found that using wellbeing services were not significantly associated with mental health or wellbeing and service use was not predicted by concerns about confidentiality or the belief that they did not understand the pressure of the role (Spence et al., 2024b). However, there may be other perceived benefits employees garner from wellbeing services beyond symptom reduction.

Both these studies relied on an anonymous, online sample. This therefore raises concerns over the validity and reliability of the data. For one, the population to which the survey was distributed cannot be described, this means respondents may be unrepresentative in some meaningful way (Andrade, 2020). For example, it is impossible to track non-response rates (Andrews et al., 2003) and it will likely suffer from self-selection bias (Thompson et al., 2003). Those who are traumatised by their job may have been more motivated to respond. Similarly, online surveys may suffer from coverage limitations (Provost, 2002), whereby those who completed the survey may not currently be or have ever been CMs (the target population). These limitations make it harder to make generalisations from the study findings but can be ameliorated through replication (Wright, 2005).

This paper seeks to replicate key findings from the previous studies and extend them using a large sample of professional CMs working for an international company. Specifically, we aim to a) explore rates of psychological distress, secondary trauma and low wellbeing, b) determine whether there is a dose-response effect between frequency of exposure and greater symptomatology, c) establish whether there is an association between wellbeing service use and a reduction in symptomatology and if there are other perceived benefits employees gain from wellbeing services and d) explore the relationship between coping and symptoms using a validated measure of coping styles.

2. Materials and Methods

2.1. Participants

Participants were 160 CMs recruited from an international company that provides content moderation services in the entertainment industry. The company employs approximately 200 CMs so this represents an 80% response rate. The participants completed an anonymous online survey that collected demographic information (e.g., sex, age, location), mental health symptoms by way of the Core-10, a measure of psychological distress, the Secondary and Vicarious Trauma Scale and the Short Warwick Edinburgh Mental Wellbeing Scale, and their working conditions.

The link to the online survey was shared with trust and safety professionals at the company who disseminated it to their frontline CMs. The survey included a range of questions about themselves, their job and their work environment. This included what sort of distressing content they were exposed to, if they use the wellbeing services available at work (often/occasionally/never), concerns about confidentiality (yes/somewhat/no) and whether the professional understands the pressure of the role (yes/somewhat/no), as well as if they thought the wellbeing service had had an impact (more productive/fewer sick days/improved mental health/feel heard/valued/other). Participants were also asked if they had moved from a customer support agent role to their current role (yes/no) and if their wellbeing had changed as a result (improved/stayed the same/gotten worse). Mental health and wellbeing were measured through the Core-10, Short Warwick Edinburgh Mental Wellbeing Scale and the Secondary and Vicarious Trauma Scale, coping strategies were measured using the Coping Orientation to Problems Experienced Inventory Brief. Participation was unpaid, anonymous and voluntary and all participants read through an information sheet about the study and gave consent before they began. The study was approved by the university psychology department’s ethics board (Ref: 21657).

2.2. Measures

2.2.1. Short Warwick Edinburgh Mental Wellbeing Scale (SWEMWBS; Tennant et al., 2007)

The SWEMWS is a seven item self-report measure which asks about thoughts and feelings over the previous two weeks. Items are scored from 1 ‘none of the time’ to 5 ‘all of the time’, then scores are summed and transformed into metric scores using the SWEMWBS conversion table. Scores range from 7 to 35 with higher scores indicating greater wellbeing. High wellbeing is scored 27.5 and above, whilst low wellbeing is scored 19.5 and below.

2.2.2. Clinical Outcomes in Routine Evaluation (Core 10; Barkham et al., 2013)

The Core 10 assesses psychological distress over the previous week. It consists of 10 items which are scored from 0 ‘not at all’ to 4 ‘most or all of the time’ and then summed. The items cover anxiety, depression, trauma, physical problems, functioning and risk to self. Higher scores indicate higher levels of general psychological distress, with severe psychological distress scored 25 and above and the non-clinical range scored 10 and below.

2.2.3. Secondary and Vicarious Trauma Scale (SVTS)

The SVTS is a 17 item self-report measure assessing aspects of secondary and vicarious trauma. Items are scored from 1 ‘strongly disagree’ to 5 ‘strongly agree’, and covers thoughts (most people are not trustworthy), emotions (I often feel sad), and intrusion (I find it difficult to separate my work and personal life). Scores are summed with higher scores indicating higher levels of secondary and vicarious trauma.

2.2.4. Coping Orientation to Problems Experienced Inventory (Brief-COPE; Carver, 1997)

The Brief-COPE is a 28 item self-report measure of effective and ineffective coping strategies. Each item is scored from 0 ‘I have not been doing this at all’ to 3 ‘I have been doing this a lot’. Scores are then summed across three subscales: Problem-Focussed Coping, Emotion-Focussed Coping and Avoidant Coping. Problem focused coping is where someone directly addresses the source of a distressing situation using strategies like acting to make the situation better or planning what steps to take. Emotion focused coping focuses on reducing emotional stress, often through strategies such as seeking social support. Avoidant coping is characterised by trying to avoid dealing with the stressor through strategies like denial or substance use.

2.3. Analysis

Descriptive statistics established the frequency of demographic characteristics, type of content CMs were exposed to, frequency of exposure and to quantify coping strategies employed by CMs. A series of multivariate general linear models (MGLMs) were conducted. The first assessed the relationship between frequency of exposure and psychological distress, secondary trauma and wellbeing. Frequency of exposure was trisected into ‘daily’, ‘weekly/monthly’ and ‘less often’ and entered as a predictor. To control for their effects, sex, location and duration in role were entered as fixed factors, whilst age was centred and entered as a covariate. The total scores on the Core-10, SVTS and WEMWBS were entered as the dependent variables. Due to small numbers, participants from Europe, South America or who did not identify as male or female were removed from the MGLM analysis. A second MGLM was conducted to explore the relationship between wellbeing service use, confidentiality concerns and wellbeing services understanding the pressure with psychological distress, secondary trauma and wellbeing as dependent variables. The predictor variables were dichotomised into ‘yes/no’ with ‘often’ and ‘occasionally’ recoded as yes and ‘never’ recoded as no for wellbeing service use, and ‘yes’ and ‘somewhat’ recoded as yes and ‘no’ kept as no for concerns about confidentiality and professional understands the pressure of the role. Lastly, the different total scores on the coping style sub-scales of problem focused, emotion focused and avoidant coping were entered as predictors with the Core-10, SVTS and WEMWBS total scores as the dependent variables.

3. Results

The majority of participants were male (63.75%) and located in Asia (62.50%). The moderators were aged between 19-48 (mean = 29.68, SD = 5.10) and had most commonly been in the role between 6 months to 2 years. Compared to the respondents in Spence et al., (2024a, 2024b) this sample contained significantly more males (χ² = 13.98, p < .01), was significantly younger (F = 30.60, p < .001) and average time in the role was shorter (F = 12.02, p < .001). The sample location was also significantly different (χ² = 180.84, p < .001) (see Table 1). The majority of participants (52.4%) scored in the low to mild range of the Core-10, with a further 13.8% scoring in the healthy range and a third (33.8%) scoring in the moderate to severe range. There were 74 (46.3%) that scored 13 or greater on the Core-10, suggesting almost half of the sample had symptoms associated with clinical levels of depression. Over a quarter of participants (28.8%) scored in the low wellbeing range and 13.5% scored in the high range. The Core-10 was demonstrated a significant negative relationship with wellbeing (r = -.57, p < .001) and significant positive relationship with secondary trauma (r = -.61, p < .001). Wellbeing was negatively correlated with secondary trauma (r = -.62, p < .001).

3.1. Exposure to Content and Mental Health

Over three quarters (76.9%) of CMs reported being exposed to hate speech, and over a third reported exposure to humiliation (35.6%) and child sexual abuse material (CSAM; 34.4%) (see Figure 1).

Most frequently, CMs were exposed to content they experienced as distressing less than monthly, but over a quarter (28.7%) were exposed on a daily basis and over a fifth (21.3%) on a weekly basis. The MGLM showed there was a dose response effect between frequency of exposure and psychological distress (F = 5.16, p = .007) and secondary trauma (F = 8.72, p < .001) but not wellbeing (F = 1.19, n.s.) (see Figure 2). Bonferroni post-hoc tests demonstrated daily exposure was associated with significantly greater scores on the Core 10 and STVT than exposure occurring either weekly-to-monthly or less often.

3.2. Wellbeing Services and Mental Health

Using wellbeing services was not significantly associated with psychological distress, or wellbeing but those who used wellbeing services had significantly lower secondary trauma symptoms. Concerns about confidentiality were significantly associated higher psychological distress, secondary trauma and lower wellbeing. There was no association between the services understanding the pressure of their role and wellbeing, psychological distress or secondary trauma (see Table 2). An interaction effect between concerns about confidentiality and using wellbeing services was checked but not significant for any of the outcomes.

In terms of perceived benefits, the majority of participants (60.6%) reported the wellbeing service made them feel valued/heard, 40.0% thought it improved their mental health, a third (33.1%) stated it made them more productive and almost a tenth (9.4%) reported they took fewer sick days. Interestingly, of the 89 (55.6%) participants who had moved from a customer safety role into content moderation, half (50.0%) asserted it had improved their wellbeing.

3.3. Coping Styles and Mental Health

All three coping styles were significantly associated with psychological distress, secondary trauma and wellbeing (see Table 2). An increase in Core-10 scores was associated with increased emotion-focused coping and avoidant coping, whereas problem-focused coping was associated with a decrease in Core-10 scores (see Figure 3).

Similarly, an increase in SVTS scores was associated with greater emotion-focused and avoidant coping and less problem-focused coping (see Figure 4).

Lastly, wellbeing demonstrated the opposite pattern where it increased with problem-focused coping and decreased with greater emotion-focused and avoidant coping (see Figure 5).

4. Discussion

This study sought to replicate and extend findings from previous studies investigating CM mental health and wellbeing using a defined sample of CMs. The response rate in the current study was 80% meaning there can be confidence that these results are representative of CMs currently employed at this company. The current study sample was significantly different on a number of demographic dimensions from the previous sample, yet the results pertaining to mental health were consistent. In both this and the study by Spence et al., (2024a) approximately a third of CMs scored in the moderate to severe range of the Core-10, just under half had scores suggesting clinical depression and over a quarter of moderators scored in the low wellbeing range. The dose-response effect between frequency of exposure to content and levels of psychological distress and secondary trauma was replicated, with individuals exposed to content on a daily basis reporting significantly higher psychological distress and secondary trauma symptoms than those exposed less often. Similarly, the lack of a significant association between exposure and wellbeing was reproduced.

Research using the Core-10 in non-clinical studies is lacking, however a study by Bewick et al., (2008) found 17.2% of students scored in the moderate to severe range of the Core-10. A rate that was almost half that found in the studies of CMs. Additionally, Connell & Barkham (2007) found a mean Core-10 score in the general population (n = 535 adults) of 4.7, indicating that moderators again skew high. Nevertheless 38.2% of moderators in the current sample scored in the healthy-low range, 18% (n = 11) of whom were exposed to distressing content on a daily basis. Given that wellbeing service use was not significantly associated with psychological distress, this suggests there are individuals who are naturally more resilient to this type of work. Future research should focus on exploring what factors are associated with resiliency in this high stress role, both in terms of screening for features that may make individuals less vulnerable but also what can be learned from resilient individuals to influence wellbeing policy more widely.

The lack of association between using wellbeing services and mental health and wellbeing found in Spence et al., (2024b) was partially replicated. Using the wellbeing services was not significantly associated with psychological distress or wellbeing but it was associated with reduced secondary trauma. These results highlight the need to formally evaluate “what works” across therapeutic services and the intervention-specific components which leverage change in terms of different issues, whether it be alleviating secondary trauma symptoms or psychological distress more generally. Where Spence et al., (2024b) found not understanding the pressure of the role was associated with higher psychological distress, secondary trauma and lower wellbeing, there were no significant relationships in the current study. Lastly, the relationships between concerns about confidentiality and psychological distress, secondary trauma and lower wellbeing were the same. This underlines the importance of assuring CMs the psychological support offered to them at work is confidential. As before, concerns with confidentiality did not significantly interact with using services, therefore CMs may still use services but not engage with them in the same way. Future research should try to tease out why CMs are worried about confidentiality and how this drives greater symptomology and reduced wellbeing.

Nevertheless, beyond reducing symptoms there were other perceived benefits of having wellbeing services available. Most notably, the majority felt that having it available made them feel heard/valued. This has numerous benefits beyond mental health including increased job satisfaction and motivation (Pfister et al., 2020; Seitl et al., 2024) and decreased intention to leave (Liu et al., 2024). Additionally, almost half thought it improved their mental health. The Core-10 is a measure of psychological distress and it is possible that other mental health scales would discern a reduction in symptoms. Nonetheless, mental health is not merely the absence of mental illness and the two are related but distinct (Westerhof & Keyes, 2010), therefore wellbeing services may also help to improve mental health without reducing symptoms. Interestingly, half of respondents who moved from a player support role, where the work typically involves responding to players queries about their accounts, gameplay etc., into a moderation role reported an improvement in their wellbeing. Given the more stressful nature of moderation this was unexpected. However, although the company provides a general wellbeing service across all teams, the one for the content moderators has been specifically tailored to address their unique challenges, which might explain why self-reported wellbeing improves. This again highlights the need to tease out what components of wellbeing services are working and what different issues they serve to address in content moderators.

This study used the Brief-Cope scale to extend the previous results concerning coping strategies CMs use and their effect on mental health and wellbeing. Emotion-focused and avoidant coping were associated with increased psychological distress, secondary trauma and decreased wellbeing, whereas problem-focused coping was associated with the opposite. This is consistent with previous literature that shows problem-focused coping is associated with greater resilience and fewer mental health symptoms, whilst avoidant and emotion-focused coping are associated with poorer mental health outcomes (e.g., Chang et al., 2024; Herman-Stabl et al., 1995; Völlink et al., 2013). These results show a potential avenue for training that increases problem-focused problem solving as a wellbeing initiative in trust and safety workplaces.

Limitations

Although many of the previous findings have been replicated using a new sample, and crucially, one that is significantly different in terms of demographics from the original sample, it has not escaped the authors that we are replicating our own work with data we cannot publicly share. Other researchers need to work with industry partners to publicly replicate these results. Nevertheless, the replication of the original results adds more credence to their generalisability and reliability. Additionally, the results are cross-sectional and anonymous, whereas longitudinal studies that track individuals over time are critically needed to assess how scores may change and for whom.

5. Conclusions

The results of the current study mostly replicate previous findings that used a large anonymous online sample. This indicates that the results are generally robust and suggests approximately a third of CMs are likely to demonstrate raised rates of psychological distress and a quarter will experience low wellbeing. This, combined with the consistent finding that wellbeing services demonstrate limited effectiveness, highlight the ongoing importance of discovering ‘what works and for whom’ with this population of workers. One potential avenue is exploring training that increases problem-solving focused coping strategies. The lack of longitudinal research also urgently needs to be addressed, especially when many of these workers are young and the long-term effects are currently unknown.