Psychometric Properties of Assessment Instruments for Reactive Attachment Disorder and Disinhibited Social Engagement Disorder in Minors under Protective Measures

1. Introduction

The inclusion of attachment disorders in the DSM-III marked a significant milestone, encompassing manifestations of Reactive Attachment Disorder (RAD) and Disinhibited Social Engagement Disorder (DSED) [1]. In the DSM-5-TR, these challenges are categorized separately [2,3]. Key features of RAD in young children include: (a) absence of attachment behaviors directed toward the primary caregiver, (b) failure to seek and respond to comfort in distressing situations, (c) diminished social and emotional reciprocity, and (d) disruptions in emotional regulation. Notable characteristics of DSED include: (a) little caution in approaching unfamiliar adults, (b) willingness to be with strangers, (c) lack of appropriate social and physical boundaries, evidenced by overly close interaction with unfamiliar adults, and (d) seeking close physical contact [3].

Observing attachment disorders in children involves examining the shallowness and conflict in their relationships, stemming from a lack of trust due to negative experiences with primary caregivers. Children with RAD struggle to trust others, even when encountering kindness. In contrast, those with DSED exhibit indiscriminate trust, placing them at a higher risk of physical, sexual, and emotional harm [4].

RAD and DSED relate to a failure in the normative attachment system that a baby should develop with their primary caregiver, experiencing a lack of access to the protection and security the caregiver should provide [5]. Certain risk factors, particularly adverse childhood experiences, contribute to the development of RAD or DSED. These include psychological maltreatment, sexual abuse, neglect, parental alcoholism, familial drug use, caregiver mental health issues, or the absence of a consistent primary caregiver, potentially due to institutionalization, recurring protective measures changes, parental incarceration, or parental abandonment [6]. Younger children are more susceptible to negligent or abusive behaviors from caregivers, leading to situations of neglect and an increased risk of removal from their biological family and institutionalization [7].

Furthermore, DSED and RAD are frequently diagnosed in children who have been institutionalized, post-institutionalized, or placed in foster homes. These children often display self-soothing behaviors, discomfort in social interactions, and aggression towards peers [8]. Additionally, they commonly exhibit other associated personal, developmental, emotional, social, and behavioral difficulties [9,10]. Despite an initial improvement in symptoms when transferred to a protective institution, this improvement tends to be temporary. Inhibitory behaviors remain stable, while disinhibited behaviors worsen with prolonged stay, suggesting that institutionalization can exacerbate attachment disorders, particularly disinhibited behaviors [11,12,13,14]. Despite the relevance and prevalence of RAD and DAI, there are few designed instruments that assess these disorders.

1.1. Instruments Assessing Attachment Disorders

• Disturbance Attachment Interview (DAI): The DAI, developed by Smyke and Zeanah in 1999 [15], is a semi-structured interview consisting of 12 items administered to a primary caregiver or someone well-acquainted with the child. It aims to assess signs related to disordered attachment and symptoms of both RAD and DSED. The items cover inhibited behaviors for RAD diagnosis, such as the absence of a preferred adult, lack of comfort-seeking, and limited social reciprocity. For DAI diagnosis, disinhibited behaviors are evaluated, including a lack of caution with strangers and a willingness to go with unknown adults. Scoring for the DAI ranges from 0 to 10 for RAD and 0 to 8 for DSED [16].
• The Preschool Age Psychiatric Assessment (PAPA): Seim (2019) [17] developed the PAPA, a caregiver-reported questionnaire designed for preschoolers aged 2-8 years. This assessment evaluates RAD and DSED based on DSM-5 criteria. RAD classification requires meeting RAD criteria A1 and one or more criteria B. For DSED, participants must meet at least two DSED criteria [18].
• Relationship Patterns Questionnaire (RPQ): The RPQ, created by Kurth and Pokorny in 1999 [19], employs a 10-item Likert scale to evaluate RAD symptoms, encompassing both RAD and DSED. Six items describe inhibited behaviors, while four describe disinhibited behaviors [20].
• Reactive Attachment Disorder and Disinhibited Social Engagement Disorder Assessment (RADA): Developed by Lehmann et al. in 2018 [21], the RADA assessment follows DSM-5 criteria. It features 11 items for RAD and 9 for DSED. TAR includes two factors: incapacity to seek/accept comfort and low socioemotional responsiveness/emotional dysregulation, while DSED has one factor related to indiscriminate behaviors [22].
• Development and Well-Being Assessment RAD/DSED (DAWBA RAD/DSED): A section within the DAWBA interview [23] comprises 14 items derived from the Child and Adolescent Psychiatric Assessment for RAD/DSED. These items assess social behaviors of concern to caregivers, scoring from 0 to 10 for TAR and 0 to 18 for DSED [24].
• RAD Questionnaire (Questionnaire Disorder Attachment Reactive): The RAD Questionnaire, developed by Minnis in 2002 [25], consists of 17 items evaluating both reactive and disinhibited attachment disorders concurrently. Scores on this questionnaire range from 0 to 51, with higher scores indicating more severe attachment disorder symptoms [26].

1.2. The current study

The aim of the present study was to assess the psychometric properties of instruments evaluating attachment disorders in samples of minors under protective measures. To achieve this, a systematic review was conducted, adhering to PRISMA standards and utilizing the Consensus based Standards for the selection of health status Measurement Instruments (COSMIN) checklist for systematic reviews of Patient-Reported Outcome Measures (PROMs) [27]. Information was gathered for each analyzed instrument, including authors, sample characteristics, country, design quality of the studies, and measurement properties as indicated in the studies.

2. Materials and Methods

2.1. Search Strategy

A systematic review of the scientific literature related to RAD was conducted according to the guidelines established by the PRISMA statement [28]. The search was carried out in March 2023 in the Web of Science. The names of the assessment instruments were combined using Boolean terms along with words related to the evaluation of psychometric properties and terms referring to the protective care system. The combination of these Boolean terms can be observed in Table 1.

2.2. Eligibility Criteria

Inclusion This review included articles analyzing the psychometric properties of instruments assessing RAD and DSED in minors within the protective care system. To ensure data comprehensiveness, articles meeting the criteria described below were included: 1) Original articles published in English or Spanish, 2) published between 2000 and 2022, 3) with samples of minors in adoption or residential/foster care, and 4) addressing the study of psychometric properties of instruments evaluating RAD or DSED.

Exclusions comprised: 1) Gray literature (doctoral theses, conference communications, or press articles), 2) articles using questionnaires but lacking psychometric information, and 3) systematic reviews of articles.

2.3. Procedure

All records were screened in a blinded by two authors (FG-S and MS) and when there was disagreement, a third reviewer interceded (LL-T). After reviewing the title and abstract to identify articles that analyzed the psychometric properties of instruments, works that did not meet the inclusion criteria were excluded (n = 189).

From the selected articles (n = 25), the full text was consulted, and once again, those that did not address the study of any instrument property were eliminated, resulting in a total of 10 articles. Among these, four focused on the DAI, one on the PAPA, two on the RPQ, one on the RADA, one on the DAWBA RAD/DSED, and two on the RAD Questionnaire. The entire process is outlined in Figure 1.

2.4. COSMIN Checklist for Systematic Reviews of PROMs

In order to thoroughly assess the methodological quality of the studies, the COSMIN checklist for systematic reviews of Patient-Reported Outcome Measures (PROMs) was utilized. This checklist distinguishes between "standards" and "criteria," with the former referring to the requirements set by each research study, indicating the quality of the study itself. Meanwhile, the latter, the "criteria," pertain to what would constitute good measurement (the quality of the PROM) [27]. The objective of this checklist is to evaluate the methodological quality of assessment instruments.

The checklist is divided into three parts and 10 boxes. The first part includes contributions from procedures aimed at conducting a systematic review according to guidelines such as PRISMA [28]. The second part encompasses the criteria used to assess the measurement quality of PROM instruments. The third and final part corresponds to the evaluation of the interpretability and feasibility of the PROM, as well as the formulation and reporting of the systematic review. For optimal use of the COSMIN guide, the authors recommend employing the checklist as a modular tool, filling in the relevant boxes [27].

3. Results

In the following section, the psychometric properties of each of the instruments selected for this study will be presented in detail.

3.1. Characteristics of the samples of the studies analyzed

Four articles analyzed the psychometric properties of the Disturbance Attachment Interview (DAI). These works included samples of between 55 and 853 people, all of whom were children aged between 4 months (Smyke et al., 2002) and 15 years (Elovainio et al., 2015) who were in protective measures. On the other hand, only one work analyzed the psychometric properties of The Preschool Age Psychiatric Assessment (PAPA). It had a sample of 400 adolescent and young-adult participants of both sexes from residential centers. In this case, although the instrument is for younger children, it was applied to adolescents.

On the other hand, an article psychometrically analyzed the Relationship Patterns Questionnaire (RPQ). It had a sample of 135 children with an average age of around 7 years. This work compared normal population, foster children, and hospitalized children (Schröder et al., 2019). The Reactive Attachment Disorder and Disinhibited Social Engagement Disorder Assessment (RADA), on the other hand, had one study that analyzed its properties with sample of over 300 children aged between 11 and 18 in foster care (Lehhmann et al.; 2020).

The Development and Well-Being Assessment RAD/DSED (DAWBA) was evaluated in a study that included 122 children adopted from birth to 10 years of age (Lehhmann et al.; 2015). Finally, the Questionnaire Disorder Attachment Reactive (RAD-Questionnaire) is analyzed in two studies with samples of 55 children (Kliewer Neumann, et al., 2018) and 182 children (Minnis et al., 2002) in protective situations, between 12 months (Kliewer Neumann, et al., 2018) and 16 years (Minnis et al., 2002), all of whom had experienced various changes within the foster care resource, both residential and family. More detailed information on these aspects can be found in Table 2.

3.2. Methodological and measurement quality of the results of the instruments

In general terms, none of the works that evaluate the psychometric properties of the instruments present the eight relevant items indicated in the COSMIN Guide. None of the manuscript’s present cross-cultural validity indices, relation or comparison with the gold standard or responsiveness. The methodological quality of the chosen studies is summarized in Table 3.

• Disturbance Attachment Interview (DAI): In most cases, the focus is on structural validity and internal consistency. In this regard, two out of the four studies conducted structural validity analyses, with both employing confirmatory factor analysis. However, one of them (Kliewer-Neumann, et al., 2015) had a small sample size (< 5 times the number of items). Regarding criterion validity, one study examined it (Kliewer Neumann, et al., 2015). Concerning hypothesis testing and construct validity, various tests were conducted, but without establishing a priori hypotheses.

In terms of reliability (Test-Retest) and measurement error, only one study provided information (Kliewer Neumann, et al., 2015). As for consistency, two out of the four studies estimated Cronbach's alpha and inter-rater reliability (Kliewer Neumann et al., 2018; Smyke et al., 2002). Both studies achieved adequate inter-rater reliability. Regarding the internal consistency of the instrument, Smyke et al. (2002), the authors who developed the interview, obtained satisfactory internal consistency values.

In the case of the study by Kliewer-Neuman et al. (2018), it was not acceptable even after relevant statistical modifications. Thus, after removing item 5, an α= .72 was obtained on the DSED subscale, and on the secure base disturbance scale, α= 0.42 was obtained, which is insufficient. This study also assessed consistency over time with weighted kappa, although it was only performed in 30% of the interviews, yielding substantial results (0.61 to 0.80) (McHugh, 2012). Sufficient data were also provided to calculate the Limits of Agreement (LoA) for measurement error and information on concurrent validity, considering the relationship between DAI and the SDQ questionnaire that assesses emotional, behavioral, and hyperactive symptoms (Kliewer-Neumann et al., 2015).

The study by Kliewer Neumann et al. (2018) assessed the association between DAI and RAD, demonstrating concurrent validity with the inhibition subscale of DAI but not with disinhibition. In the case of the disinhibition scale, an association was found with tests related to the strange situation.

No study evaluated the transcultural validity of the instruments, nor responsiveness (Area under the Curve), and there were not enough studies testing hypotheses to calculate the percentage of agreement as established in the COSMIN criteria.

• The Preschool Age Psychiatric Assesment (PAPA): A study conducted by Seim et al. (2020) assessed the psychometric properties of the PAPA, focusing on criterion validity, measurement error, and discriminant validity. The study yielded favorable results regarding the two-factor factorial structure (inhibited and disinhibited) as well as discriminant validity. It was observed that RAD and DSED were distinct constructs from each other and from other mental health issues.
• Relationship Patterns Questionnaire (RPQ): A study conducted by Schöder et al. (2019) assessed the internal consistency and measurement error of the RPQ. The study reported adequate values for internal consistency (overall scale α = .82; inhibition subscale α = .75; disinhibition subscale α = .81). In terms of criterion validity analysis and responsiveness, calculations were performed in the study by Schröder et al. (2019). Significant Area Under the Curve (AUC) values were obtained, indicating diagnostic accuracy, with lower accuracy observed in boys compared to girls. The study also proposed diagnostic cut-off points.
• Reactive Attachment Disorder and Deshinhibited Social Engagement Disorder Assessment (RADA): One study assessed the psychometric properties of the RADA (Lehmann, Monette et al., 2020). It was focused on analyzing its factorial structure by proposing a three-construct factorial solution (DSED: indiscriminate behaviors with strangers; RAD1: inability to seek/accept comfort; RAD2: withdrawal/hypervigilance). The study also examined internal consistency and criterion and construct validity.
• Developmental and Well-Being Assessment RAD/DSED (DAWBA RAD/DSED): Lehmann et al. (2016) examines structural validity through a confirmatory factor analysis with two factors, consistent with the DSM-5 definition. Regarding construct validity, the study aimed to differentiate difficulties between SDQ, DSED, and RAD.
• RAD Questionnarie: Minnis et al. (2002) assessed structural validity, internal consistency, temporal consistency, measurement error, and criterion validity. Adequate indicators were obtained in all cases. Therefore, satisfactory internal consistency for the tool's use in research settings (α= 0.7) was achieved. The relationship between the questionnaire and the SDQ was tested, yielding very high correlations, which, in general terms, may not necessarily be positive.

On the other hand, the study by Kliewer Neumann et al. (2018) only examined hypothesis testing for construct validity without proposing an a priori hypothesis. Moderate relationships were observed between the RAD Questionnaire and the DAI.

3.3. Criteria measurement: quality of the instruments

The quality of the assessment measures was evaluated taking into account the updated criteria for good measurement properties outlined in the COSMIN manual [27].

Table 4. COSMIN results of the criteria of measurement (quality of the PROM).

Study and Instrument	Structural validity	Internal consistency	Crosscultural validity Measurement invariance	Reliability	Measurement error	Criterion validity	Hypothesis testing for content validity	Responsiveness
Kliewer Neumann, et al. (2018) – DAI	?	?	?	+	?	?	+	?
Kliewer Neumann, et al. (2015) – DAI	?	+	-	?	?	+	?	?
Smyke et al. (2002) - DAI	?	+	?	+	?	?	?	?
Elovainio et al. (2015) - DAI	+	?	?	?	?	+	?	?
Seim et al. (2019) - PAPA	+	+	?	?	?	+	?	?
Schröder et al. (2019) - RPQ		+	?	?	?	+	?	?
Lehhmann et al. (2020) – RADA	+	+	?	?	?	+	?	?
Lehmann et al. (2015) - DAWBA	+	?	?	?	?	?	?	?
Kliewer-Neumann, et al. (2018) – RAD	?	?	?	?	?	?	+	?
Minnis et al. (2002) - RAD	+	?	?	?	?	+	?	?
Note:

Table 4. COSMIN results of the criteria of measurement (quality of the PROM).

Study and Instrument	Structural validity	Internal consistency	Crosscultural validity Measurement invariance	Reliability	Measurement error	Criterion validity	Hypothesis testing for content validity	Responsiveness
Kliewer Neumann, et al. (2018) – DAI	?	?	?	+	?	?	+	?
Kliewer Neumann, et al. (2015) – DAI	?	+	-	?	?	+	?	?
Smyke et al. (2002) - DAI	?	+	?	+	?	?	?	?
Elovainio et al. (2015) - DAI	+	?	?	?	?	+	?	?
Seim et al. (2019) - PAPA	+	+	?	?	?	+	?	?
Schröder et al. (2019) - RPQ		+	?	?	?	+	?	?
Lehhmann et al. (2020) – RADA	+	+	?	?	?	+	?	?
Lehmann et al. (2015) - DAWBA	+	?	?	?	?	?	?	?
Kliewer-Neumann, et al. (2018) – RAD	?	?	?	?	?	?	+	?
Minnis et al. (2002) - RAD	+	?	?	?	?	+	?	?
Note:

3.4. Strength of evidence

Referring to Table 5, it can be observed that none of the studies provided information for all nine criteria in the COSMIN checklist. According to the modified GRADE criteria (Prinsen et al., 2018), most instruments exhibited low to moderate evidence.

4. Discussion

The main objective of this systematic review was to evaluate the psychometric properties of assessment instruments for Reactive Attachment Disorder (RAD) and Disinhibited Social Engagement Disorder (DSED) in minors under protective measures through a systematic review following the COSMIN checklist (Mokkink et al., 2010). As a primary finding, we note that the general trend in the examined studies is to report on internal consistency and structural validity. Additionally, the majority conducted hypothesis testing for construct validity, but without a priori hypotheses. Overall, there is a lack of evidence regarding reliability, cross-cultural validity, measurement error, and responsiveness. Therefore, we conduct an exhaustive analysis of the instruments, though it may be limited due to the nature of the studies.

To date, this is the first study evaluating the psychometric properties of instruments assessing RAD and DSED in under protective measures minors. Consequently, our results are challenging to compare with previous works. However, the study by Wright et al. (2015), which conducted a systematic review and meta-analysis on instruments assessing and/or diagnosing attachment problems, concluded that assessments of the psychometric properties of these instruments often neglect cross-cultural validity or longitudinal reliability. Therefore, more high-quality scientific research is needed to examine the validity of available instruments and provide sufficient evidence for their consideration in clinical diagnosis.

As far as we know, the reliability indices of different instruments, while mostly adequate (alpha between .70 and .82), are not sufficient to use the tools as diagnostic instruments for RAD/DSED. Most studies support the two-factor RAD and DSED factorial structure, but some authors suggest the existence of three factors (DSED: indiscriminate behaviors with strangers; RAD1: inability to seek/accept comfort; RAD2: withdrawal/hypervigilance) (Lehmann, Monette et al., 2020; Lehmann, Breivik et al., 2020), in line with the proposals of the DSM-5-TR and opening avenues for further research into a potential new diagnostic category associated with RAD. However, much more evidence and research are needed in this regard.

Furthermore, it is necessary for studies to conduct cross-cultural investigations testing the functioning of instruments in different countries. Despite studies being conducted in various countries such as Germany (Kliewer Neumann, et al., 2018; Kliewer Neumann, et al., 2015; Schröder et al., 2019), Norway (Seim et al., 2019; Lehhmann et al., 2018; Lehmann et al., 2020; Lehmann et al., 2015), Scotland (Minnis et al., 2002), Romania (Smyke et al., 2002), Belgium (Vervoot et al., 2013), and Finland (Elovainio et al., 2015), there is a lack of evidence for cross-cultural construct validity through multigroup structural equation modeling. Additionally, some studies focus on assessment interviews for these disorders, providing only evidence of inter-rater reliability (Kliewer et al., 2018). Therefore, future research should assess internal consistency over time, content and construct validity, cross-cultural validity, and the structure, efficacy, and efficiency of interviews.

Despite the significant contributions of this work, evaluating key RAD/DSED detection measures in protective minors, it is essential to acknowledge limitations. Strict selection criteria may have limited the scope of the review, as only works published in Spanish and English were considered. The COSMIN checklist (Mokkink et al., 2010), while enhancing the structure and quality of our work, leaves room for subjectivity in data interpretation and quality assessment. Future research should replicate the search in other databases to provide additional information. For example, in criteria where evaluation is based on a time period, it depends on the construct being assessed and what the researcher deems appropriate. Furthermore, even with adequate reliability, a high Cronbach's alpha does not guarantee that the desired construct is being measured or that no essential concepts are missing. Similarly, having high test-retest reliability or responsiveness does not imply that all items are important or that central concepts have not been overlooked. This highlights the possibility of assessing an incomplete or incorrect construct reliably when following the checklist.

On the other hand, research on reactive attachment disorder is limited, resulting in few studies delving into its assessment and even fewer evaluating the psychometric properties of assessment instruments. This limitation is particularly pronounced when considering studies focused on minors under protection, who, as we know, are more susceptible to experiencing RAD/DSED than minors outside the protection system (Talmon-Knuser et al., 2023). Finally, it is important to note that the search was conducted solely in one database, the one that aggregates a greater number of impactful publications. Nevertheless, future studies should replicate this search in other databases to contribute additional information on the subject.

However, this work contributes to expanding knowledge about assessment instruments for reactive attachment disorder in the context of child protection, laying the groundwork for future assessments of the psychometric properties of these instruments. Having identified areas for improvement in assessing the psychometric properties of these instruments, future studies could be designed with a cross-cultural and longitudinal approach to gather more scientific evidence in line with COSMIN checklist criteria, including measurement error, cross-cultural validity, test-retest reliability, responsiveness, criterion validity, and construct validity (Mokkink et al., 2010). In this way, progress can be made in understanding, assessing, and treating RAD/DSED. In conclusion, we note that this is the first study to examine and assess the psychometric properties of six instruments that assess RAD/DSED in a sample of adopted or foster children and adolescents. Regarding the evaluated measures, it can be concluded that all of them exhibit good structural validity, adequate internal consistency, and generally positive results with respect to measurement error.