Development of Cut-Scores for Feigning Spectrum Behavior on the Orebro Musculoskeletal Pain Screening Questionnaire and the Perceived Stress Scale: A Simulation Study

1. Introduction

Central to the management of compensable injury is addressing psychosocial barriers to recovery [1,2,3]. Chronic pain and associated syndromes, including opioid misuse, represent a substantial cost to society and these conditions are predicted to increase in the future [4]. The successful management of chronic pain, if possible, would result in significant economic savings, reduction of personal suffering, and increase in productivity for the individual and society. Similarly, feigning spectrum behavior (FSB) for the pursuit of secondary gain, such as work avoidance and access to pharmaceuticals, represents a significant cost to society and is detectable in 29% of personal injury at formal psychological assessment [5]. As a medium of service delivery telehealth has significant benefits in terms of convenience, cost savings, and capacity to reach clients in remote locations bringing expertise otherwise unavailable due to distance. With the onset of Coronavirus, there was increased interest in telemedicine and adoption of these media for service delivery, rather than face to face consultation [6], even among primary healthcare providers and there is increasing consumer acceptance of services delivered via telehealth platforms. Yet little is known about how feigning spectrum behaviour may manifest via telehealth. Inherent to compensation systems is the notion of assigning a dollar value to injury, illness, and distress. In such a system the report of financial duress could be seen as a way of assessing the motivation or intent behind feigning spectrum behaviour. Binder and Rohling conducted a meta-analysis showing a moderate effect size for financial incentives and reported abnormality and disability despite less severe injury in closed head injury [7]. Financial stress also plays a complex role in health, and structural equation modelling revealed that people reporting major financial stressors reported more interpersonal stress, psychological distress and lower levels of psychological well-being, and their psychological well-being rating was associated with elevated interleukin 6, an inflammatory cytokine marker of stress [8]. In compensable injury there appears to be an inextricable relationship between report of symptoms and secondary gains that may be exploited by some individuals.

The Orebro Musculoskeletal Pain Screening Questionnaire (OMPSQ) [9] is a widely used measure of the impact of injuries and this scale indexes psychosocial factors that complicate recovery. It has moderate predictive power to identify persisting pain and disability in patients with back pain [2], sciatic leg pain [3] and acute and non-acute pain [10]. The Perceived Stress Scale (PSS) [11] has been correlated with a variety of factors including health and smoking status, and help seeking behaviour [12], and has been used to investigate work stress and associated health behaviours [13]. Both the OMPSQ [14,15,16,17,18] and PSS [19,20,21] have been validated in several languages which is advantageous to the generalization of use in a multicultural society like Australia, although some qualitative research with culturally diverse people indicates that use of standardized questionnaires may have some unintended consequences such as negative impact on client-practitioner relationship [22]. The multilingual validation of these measures demonstrates the robustness of the constructs measured by these scales and the impact of psychosocial factors to recovery and health regardless of culture and language. The OMPSQ and PSS have short forms, the ten item OMPSQ-SF [23] and the four item PSS-4 [24,25] respectively, which have similar predictive validity in health. The PSS has been validated for telephone use and the OMPSQ-SF has been used telephonically, however, there has been no formal validation of the OMPSQ-SF for telephone administration. Unlike general health settings, the treatment of compensable injuries often involves complex interactions between the patient, treatment providers, agents of insurers, insurers, and legal service providers. Such complex systemic interactions can produce behaviours including malingering, factitious presentations, and false imputation of causes of conditions [26]. A comprehensive review of the literature across various search engines including MEDLINE via OVID, Scopus, PsychInfo via OVID, Web of Science, and Google produced no results for either the OMPSQ or PSS being validated with a simulation design to see if a feigned presentation can be distinguished from a genuine presentation.

The most successful method of detecting feigned mental disorder is the “Rare Symptoms” and “Quasi-rare symptoms” detection strategies [27]. These strategies detect feigning spectrum behaviour by indexing the overreporting of infrequent symptoms or the reporting of symptoms to an exaggerated extent. That is, on measures of clinical symptoms people engaging in feigning tend to endorse more pathology and rarer pathology than genuine respondents, producing inflated scores exceeding those generated by injured people reporting their pathology accurately. Usually, this strategy of detection generates a large to very large effect size [28]. Simulation studies involve the determination of cut-points for feigning based on the probability of genuine responding by comparing people with the condition to people without the condition who were asked to fake the condition. Criterion referenced or “known group” studies involve classifying people with the condition and those without the condition by some means, such as another measure or a known diagnosis, and then administering the subject measure. Research on the Minnesota Multiphasic Personality Inventory-Restructured Format (MMPI-RF), show that both simulation studies and criterion referenced studies generate large effect sizes [28,29]. Similar effect sizes were also found regardless of what disorder or symptom was to be faked. However, the effect size reduced to the moderate range if the simulator was educated about the disorder or the measure, so called “sophisticated” simulators [29,30].

An alternative strategy to detecting feigning spectrum behaviour is forced choice testing, and this has been applied to cognitive and memory impairments, and malingered pain [31], using the Test of Memory Malingering (TOMM). However, the medium of telephone limits the stimuli to auditory and the TOMM is a visual task. A verbal forced choice test, akin to the 21-Item Test [32], a verbal list learning task, would offer a further detection strategy. This method identifying feigning spectrum behaviour were explored by the present study in a “simulation” design and a 10-item Brief Forced Choice Test (BFCT) of verbal learning was created to test this method against the cut-score approach.

The present study aims to develop cut scores to identify FSB on the OMPQ and PSS and their short-forms and to see if a novel forced choice strategy or financial rating can identify feigning. The design is a simulation study comparing the pattern of responding of injured workers (IW) with Community Sample (CS) of non-injured people instructed to simulate a workplace injury (CSS) and respond genuinely (CSG). A further aim was to see if multiple exposures to test materials influenced the strategy to engage in FSB, and if practice improved a non-injured person’s ability to simulate an injury. The study also investigated the degree of reported financial duress to see if simulators equate financial distress with the intent to feign an injury, and if there is any discriminant value in enquiring about financial status. It was hypothesized that the CSG will score significantly lower than IW on all measures, and that CSS will score significantly higher on all symptom measures and significantly lower on the BFCT and financial rating than IW.

2. Materials and Methods

Study Design and Participants

Given the expected large effect size and comparisons to be made a sample size of 40 participants in each group would have sufficient statistical power [33] with a power estimate of 0.98 [34]. The IW were the first 40 referrals who consented to pre-screening for an independent support service for people who had suffered compensable workplace injuries. The pre-screening included assessing suitability for the program, current treatment, social support, relationship with employer and insurer and then the administration of the questionnaires. The CS was a sample of convenience of 40 non-injured people including employees from other business units and willing associates. Both groups were onboarded to the service by the first author and their permission to participate was acquired orally for all telephone calls to be recorded “for quality, training and research purposes”. The administration of the questionnaires was explained as part of pre-screening and a convenient time was scheduled for administration of the questionnaires by an intern psychologist. Each CS participant was scheduled under an alias into the workflow of the intern psychologist and instructed on which calls to respond genuinely and when to simulate. Each CS participant responded to the assessment protocol four times, responding twice in the simulated condition (CSS) and twice genuinely (CSG). The order of simulation and genuine responding was counterbalanced to see if there were practice effects. In the simulation condition the community sample participants were instructed to “fake an injury that occurred in the course of your work”.

Measures

The OMPSQ is a 25-item measure that is scored by summing the ratings of 21-items that are a Likert type scales which have various anchoring statements at the endpoints yielding a range of scores 4 to 210 [9] with scores over 105 predicting poor outcomes [35]. The OMPSQ-SF is a subset of 10-items of the parent scale, with scores greater than 50 predicting unfavourable outcomes [23]. These measures were administered as a set of verbal rankings using the anchoring statements and scaled out of 10 to yield totals as in the pen and paper measure. The PSS is a 10-item scale in which statements are rated from “never” to “very often” and it was interpreted dimensionally using the printed norms [11]. The PSS-4 short form is a 4-item scale using items 4, 5, 7 and 8 of the parent scale and it was interpreted dimensionally using the printed norms. The short forms were calculated from the subset of items from the long forms of these scales. Consistent format with the OMPSQ items, participants were asked to rate their financial wellbeing from 0 to 10 anchored from 0 = “very poor” to 10 = “good”. The BFCT was a 10-item list learning task with a free recall trial, a distraction trial of counting backward from 20 to 0, and then the forced choice trial [Appendix 1 lists the target & foil words].

Data Collection

An intern psychologist with honours level qualifications, was trained in the telephonic administration of the measures, with the calls made within a period from November to December 2017 when the program was initiated. The dataset was then compiled and de-identified and provided to the researchers by the manager of the support program.

Statistical Analysis

Statistical analysis was performed using IBM SPSS Statistics Version 26 (64-bit Edition). Shapiro-Wilk Tests and Levene’s Tests were used to assess for normality. Differences between subsets of the community sample were investigated using Mann-Whitney U-test or T-tests depending on normality. Differences between the CS and IW were compared for age using independent groups T-test and for gender using a Bayesian Independent Groups analysis. Given the predicted directional relationship and large effect size, the IW, CSG and CSS were compared using one-way Analysis of Variance, or if marked difference between demographic variables was identified, then Analysis of Covariance (ANCOVA) would be conducted [36]. Given that the CSS and CSG was assessed on two occasions, the test re-test reliability was assessed using a two-way, single measures absolute intraclass correlation for each measure. With a 95% confidence interval and large expected intraclass correlation coefficient for the CSG and CSS a sample size of 40 (n>37) retained power (Power > 0.80) [37]. Receiver operator characteristics (ROC) and area under the curve (AUC) were calculated for the OMPSQ, OMPSQ-SF, PSS, and RSS-4. The Sensitivity (Sn), Specificity (Sp), Positive Predictive Power (PPP) and Negative Predictive Power (NPP) were calculated for viable cut scores.

3. Results

The study was conducted in Sydney Australia with the data deidentified and conducted to the researchers in December 2019 after ethics approval was received on 29th June 2019.

3.1. Demographic Differences

IW suffered non-trivial injuries and consisted of injury 15-upper limb, 13-lower limb, 3-shoulder, 9-back, two lung injuries, and one “internal organs other”; with three of the IW coded with two injuries. Table 1 outlines the demographic data describing the IW and the CS. An independent-measures t-test revealed that the IW were significantly older than the CS (t = 2.253, Sig (2-tailed) = 0.027). Bayesian Independent Group analysis of the gender ratios in the groups revealed the ratio of males to females was statistically significantly different (t = -6.637, Sig. (2-tailed) = 0.000) with more females than males in CS. Table 1 shows the differences between the IW and CS.

Shapiro-Wilks Test for normality in scores comparing males to females showed non-normality in CSG on five of the eight conditions comparing each gender on the PSS, PSS-SF, OMPSQ and OMPSQ-SF. Subsequent Mann-Whitney U-test showed no significant differences between genders and responses on these measures in the CSG. Shapiro-Wilks Test for normality comparing males to females in the CSS, showed normality for each condition and measure, and t-tests showed no significant differences on any of the measures. Shapiro-Wilks test of normality in scores comparing employed to not employed for the CSG showed non-normality in four of the eight conditions on PSS, PSS-4, OMPSQ and OMPSQ-SF. Subsequent Mann-Whitney U-Test showed no significant differences between employed and not employed status in CSG and their test scores. Shapiro-Wilks test of normality in scores comparing employed to not employed in CSS showed non-normality in PSS-4 for not employed, but the other seven conditions had normality. The three t-tests and one Mann-Whitney U-Test showed no significant differences between employed and not employed within CSS and their test scores.

3.2. Analysis of Covariance Between Conditions

Table 2 contains the means for each condition and the results of the Analysis of Covariance with covariates of Gender and Age. The OMPSQ and OMPSQ-SF discriminated between the responses of IW, CSG and CSS. However, the PSS and PSS-4 and Finance Question only discriminated between the CSS and the groups responding genuinely. Gender and Age did not significantly contribute to score on OMPSQ (sig = 0.320 / 0.694), OMPSQ-SF (sig = 0.637 / 0.637), PSS (sig = 0.650 / 0.988), PSS-4 (sig = 0.547 / 0.707), or Forced Choice Total (sig. = 0.115 / 0.444), but gender did contribute to variance in the Finance Rating (sig. = 0.027).

3.3. Short Term Test-Retest Reliability

The test-retest reliability for the CSG and CSS conditions were acceptable, revealing uniform behaviour across the different conditions with telephone administration and the results of this analysis are shown in Table 3

3.4. Receiver Operator Characteristics

“Elbow” points are where the plot line of the ROC curve approaches 90-degrees which indicate that the test score clearly delineates between the two samples. Visual inspection of the ROC curves in Figure 1 shows that the OMPSQ has acceptable “elbow” points for identification of cut scores to differentiate between CSG and the IW and that the OMPSQ SF has some opportunities but tends towards a more dimensional interpretation. The other scales show little opportunity for cut scores. Figure 2 shows the OMPSQ and OMPSQ-SF have shapes that indicate likely cut-points for distinguishing CSS from IW.

Table 4 contains the ROC of the CSG and CSS compared with the IW. Comparing the lower bound of the confidence interval of the estimated AUC as an assessment of the overall model quality, showed that the OMPSQ (0.83) and OMPSQ-SF (0.60) discriminated between the IW and CSG, but the other measures were not better than chance at classifying membership to this group. The PSS (0.68), PSS-4 (0.67) OMPSQ (0.66) and OMPSQ-SF (0.76) classified CSS from IW, with the OMPSQ-SF being the best for this determination.

A cut score for distinguishing CSG and IW on the OMPSQ of 15 had a Sn of 97.5% and SP of 90%, with a PPP of 2.5% and NPP of 100%. A cut score of 51 had 90% Sn, 23.8% Sp, PPP of 92.5%, and NPP of 78.75% which shows the broad range of scores in the IW sample. For discriminating between CSG and IW the OMPSQ-SF at a cut off at 20 had a Sn of 95%, 80% Sp, PPP of 97.5% and NPP of 21.25%.

Table 5 and Table 6 contain the prospective cut-scores for discriminating the IW sample from the CSS with Sn, Sp, NPP and PPP at each of the upper limit of the bands identified by Linton, Nicholas, and MacDonald [23] for the OMPSQ and OMPSQ-SF, respectively. Using a specificity of greater than 95% to minimize false positives, that the OMPSQ using a cut score of 140 identified 28.8% of the CSS and 5% of IW sample as engaging in FSB. The OMPSQ-SF using a cut score of 70, identifying 25% of CSS with a 5% identification of IW as engaging in FSB. Given base rates of FSB [38], it is probable that these are true positive identifications of feigning in this IW sample.

4. Discussion

The present study compared methods for identifying feigning spectrum behaviour from genuine responding on variants of the OMPSQ and PSS, and other possible novel strategies including a forced choice auditory memory procedure and a question about finance delivered telephonically. The OMPSQ had sensitivity to discriminate between non-injured people, people with significant workplace injuries, and non-injured people asked to simulate an injury. The OMPSQ-SF appears to have a “Goldilocks” combination of questions that distinguishes between uninjured community sample members responding genuinely, workplace injured people, and community sample when asked to simulate an injury with the minimum number of questions. While the PSS and PSS-4 did statistically distinguish between the simulation conditions and people injured in the workplace these measures did not have good ROC for cut scores to identify feigning. Therefore, the PSS and PSS-4 scores should not be relied upon solely as indications of FSB, but may contribute in combination with other measures. Similarly, conclusions about high scores on the PSS and PSS-4 cannot be concluded to simply reflect stress, but may represent FSB, and furthermore the construct measured by these tests are falsifiable or able to be feigned without being distinguished from genuine responding.

Using the Linton et al. 2003 [7] and 2011 [23] cut scores of 105 and 90 respectively, 8 and 19 of IW were classified at risk of poor adjustment on the OMPSQ, and 51/80 and 62/80 of the CSS condition were classified as at risk for poor adjustment. The OMPSQ-SF cut-off of 50 classified 7 of the IW and 60/80 of the CSS as at risk for poor adjustment. This shows the differing sensitivities of these cut scores derived from the OMPSQ, with the OMPSQ-SF best resembling the original cut score sensitivity for poor adjustment. Scores above 70 on the OMPSQ-SF, identifying less than 5% of IW, likely reflects feigning spectrum behaviour, and 20 of the CSS would have been identified as simulating and 2 of the IW.

Examining the different demographic variables of the CS statistically showed no differences between variables of age, employment status on the pattern of responding in CSG and CSS conditions. Given the counterbalanced order of administration of test conditions to the CS, the strong Intraclass Correlation, and that there was no significant differences between Trial 1 or Trial to for CSS, this suggests people tend to engage in the same kind of feigning spectrum behaviour regardless of previous exposures to the test material. The data also suggests the OMPSQ and the OMPSQ-SF appear to retain sensitivity to feigning despite repeated administration and exposure to the test materials, and the opportunity to practice simulation. Other longer measures, such as the MMPI-2, have reduced sensitivity to identify feigning spectrum behaviour with coaching or repeated administrations [28,29]. The CSG inherently endorsed financial duress when simulating, and more so after previous exposure to the telephone protocol. This shows within simulated compensation scenarios people naturally equate distress with financial duress despite there being no inherent suggestion of such a relationship. Enquiring about financial wellbeing using a 10-point Likert Scale has some utility in identifying attitude to recovery in compensable samples, with scores below 5 a possible negative indicator. The CSS did not identify memory as a symptom to feign when asked to simulate a workplace injury. This may have been because the task seemed too easy, that the task seemed unrelated to the simulation of a work injury, or that it was easily identified as a possible detection strategy. Alternately, the forced choice task may have failed because the 10-items was not enough items to create the illusion that an injured person would reasonably perform poorly on it, so CSS did not try and fail this measure. At 50-items, the TOMM realistically creates the illusion of being fallible by being beyond free recall and requiring recognition, and at 10-items, the BFCT may not have created such a cognitive illusion.

The goodness of the ROC for identifying CSG on the OMPSQ and OMPSQ-SF was generally better than for the prediction of long-term recovery, with the AUC in studies of recovery ranging from 0.586 to 0.710 [7 & 23]. This likely reflects the increased accuracy of identifying a state rather than a trait in construct measurement, and it is possible that at times of measurement, these studies were confounded by low “trust” states resulting in feigning spectrum behaviour. The compensation status of the participants in these other studies was not identified and it is possible the presence of compensation can distort the predictive accuracy. For example, applying the cut scores identified above to Linton et al. [23] published results would identify 19% of those who had more than 14-days accumulated sick leave and 2.7% of those who did not have time off, as engaging in feigning spectrum behaviour.

Limitations exist in the current study. A confound of the study was that there was only one telephone assessor and the community sample were likely recognizable by voice such that a true blind could not be maintained across the experiment. A further limitation was the statistically significant gender and age differences between the IW and CS, although this did not generate statistically significant differences in responding. The ratio of males to females in the CS was roughly equivalent to the Primary Care Sample in the development of the OMPSQ-SF [23]. Research on gender and feigning spectrum behavior is somewhat limited, however recent research in both clinical and forensic settings found males tended to engage in malingering more than females [39,40] and of those deemed to be engaging in malingering as detected by an interview using the “rare” and “quasi rare” method, the Miller-Forensic Assessment of Symptoms Test, males and females did not differ significantly on their pattern of feigning behavior, except that female’s report of hallucinations was more likely to be considered genuine rather than male’s by expert examiners [39]. In a litigating Australasian sample, the Infrequency (F) scale on the MMPI-2 was used to compare the base rate of malingering in females and males [38], and despite more males comprising of the normative sample (57%) females were as likely to be identified as feigning as males. In the present study ANCOVA showed no significant effect for gender or age on the classification of simulating. The CS was significantly younger than the work injured sample, however, statistically age did not significantly contribute to variance. The research on age and feigning spectrum behavior indicates that young adults are more likely to admit to engaging in feigning spectrum behavior [41] and future research using the cut scores identified above could test “empathic confrontation” type interventions with young adults breaching the cut scores. Some research suggests that age matching and race matching of simulated malingerers to actual malingering populations is not necessary because the differences are not large enough to influence clinical interpretation of the pattern of responding [42] although on performance validity measures doing double duty to detect malingering are improved with age corrected scores [43]. On inspection, there were educational differences between the IW and CS. Research on FBS shows this behavior is detected more often in people with lower levels of education [4], and sophistication which may be assumed from more educated people reduced the effect size of simulation studies [29]. The effects of education status on ability to engage in FBS warrants further examination by well defined studies, and it is possible that this factor reduced the sensitivity of the cut-scores identified.

Future research with larger sample sizes could examine the relative utility of the cut scores developed above with more diverse samples. A further limitation of the current study was the same as in the development of the OMPSQ Short Form, that is, the short forms and statistics were calculated from the administration of the long form, and it is not known how completing only the short form version alone might affect responses and repeating the study with the short forms alone would contribute to the validity of their use. Given the utility and general acceptance of the OMPSQ by workers compensation systems further research into the characteristics of simulated injury is warranted with the present study representing an initiation of this process. Similar research would benefit from use of debriefing interviews to inform on simulator’s strategies and sensitivity to the questions and methods. Similarly, empathic confrontation and debriefing of injured worker’s identified as engaging in feigning spectrum behaviour and analysing the results would be informative about the psychology and approaches of high stakes malingerers within compensation systems. Future research, using these or other cut-offs for feigning spectrum behaviour, could investigate tailoring interventions to address the motivational factors underlying the breaching of such cut-scores. By addressing FSB early in recovery, and assisting forensic examiners in retrospective evaluation of these factors in recovery, these and similar cut-scores can reduce unnecessary costs to society often left unchecked in our compensation systems.

Appendix A.1 Target and Foil Words for Brief Forced Choice Test

Targe Words	Foil
Cat	Rat
Boat	Coat
Rope	Hope
Van	Can
Book	Look
Hotel	Motel
Moose	Goose
Boot	Shoot
Pig	Jig
Tango	Mango