How Do Earthquake Survivors Evaluate the Psychosocial Support They Receive? Development and Preliminary Validation of a Four-Daomain Scale

1. Introduction

On 6 February 2023, two earthquakes of moment magnitude (Mw) 7.7 and 7.6, centered in the Pazarcık and Elbistan districts of Kahramanmaraş, struck southeastern Türkiye nine hours apart [1]. The mental health effects of disasters begin immediately and may persist for extended periods [2], and the 2023 earthquakes were no exception. Nearly two years later, 31.0% of 642 adult survivors in 11 heavily affected cities screened positive for probable posttraumatic stress disorder [3], and one year after the earthquakes, depressive symptoms were found in 55.8% of 640 healthcare workers at a referral hospital in one of the most severely affected provinces [4]. The response mobilized thousands of mental health professionals alongside medical and humanitarian teams [5], and scalable psychological interventions were later tested with survivors living in temporary housing [6]. Psychosocial support was therefore delivered on a very large scale. Whether the survivors who received it experienced it as well organized and useful is a different question, and it is the question addressed in this article.

Asking the people who received support is more than good practice. The Inter-Agency Standing Committee (IASC) guidelines define mental health and psychosocial support as any type of local or outside support that aims to protect or promote psychosocial well-being and/or prevent or treat mental disorder, and they state that humanitarian action should maximize the participation of local affected populations [7]. From the earliest phase of an emergency, local people should be involved to the greatest extent possible in the assessment, design, implementation, monitoring, and evaluation of assistance [7]. The IASC common monitoring and evaluation framework for mental health and psychosocial support accordingly includes among its outcome indicators the level of satisfaction of people with mental health and psychosocial problems and their families with the care they received [8]. For the early to mid-term stages after disaster and mass violence, Hobfoll et al. identified five empirically supported intervention principles: promoting a sense of safety, calming, a sense of self- and community efficacy, connectedness, and hope [9]. These principles state what support should achieve. Only the people who received the support can report whether they experienced it in this way, and their participation in evaluation requires an instrument through which they can do so.

In practice, evaluation has lagged behind provision. Psychosocial and psychotherapeutic care has increasingly been integrated into mass casualty protocols, yet emergency systems lack structured evaluation mechanisms, and validated, context-specific instruments that capture evaluations such as perceived quality, the helpfulness of psychosocial interventions, and trust in care structures are missing [10]. An expert review of global mental health and psychosocial support programming called for stronger monitoring and evaluation systems to ensure program quality [11]. Lessons drawn from the psychosocial responses to six disasters in five countries likewise included the need to integrate research and evaluation into disaster response planning; program-based research was needed to monitor consumer access, uptake, and outcomes of response services, and a central question was what outcomes these services target and how they should be measured [12]. After the 2023 earthquakes, professionals working in four severely affected provinces reported that short-term staffing, limited coordination, and insufficient infrastructure restricted the effectiveness of the services [13]. Accounts of this kind describe the response from the providers’ side. Service users’ accounts show what recipients themselves attend to. Displaced Syrian service users most often described the importance of time and understanding, dignity, confidentiality, and continuity of care [14]. Survivors of the 2023 earthquakes who were receiving psychosocial support in Hatay described unpredictability, lack of information and control, and discontinuity of care as processes that reactivated or intensified trauma-related distress [15]. In a comparison of three earlier Turkish earthquakes, survivors mentioned a lack of psychosocial support services after the Gölcük earthquake, whereas such services became more evident after the later earthquakes [16]. These qualitative studies show that recipients can judge both how support was delivered and what it brought. They do not give programs a standard way of measuring these judgments and comparing them across services, regions, or groups.

Existing measures do not fill this gap (Table 1). Service satisfaction instruments come mainly from health care. A systematic review identified 28 scales that had been used in at least two studies to assess patient satisfaction with mental health care; they varied substantially in structure, length, focus, and quality, and no gold standard had been established [17]. Information received and perceived care outcomes were among the contents that these scales covered, but the review was restricted to patients treated in adult mental health services [17]. Patient-reported experience measures (PREMs) capture the patient’s perception of what happened during a care encounter and how it happened, whereas satisfaction measures ask patients to evaluate their care [18]. A review of 75 PREMs for mental health care found that their items covered interpersonal relationships, respect and dignity, access and care coordination, drug therapy, information, psychological care, and the care environment [19]. In one such measure for addiction and mental health services, co-designed with service users, treatment effectiveness and staff behavior formed separate subscales [20]. The Client Satisfaction Questionnaire-8 (CSQ-8), developed as a general scale of client and patient satisfaction [21], is the most widely used instrument for measuring patient satisfaction in mental health services [22]. One of its items asks whether the services helped the respondent to deal more effectively with problems, but exploratory factor analysis supported a single factor, so the evaluation is summarized in one dimension [22]. Among Syrian mental health and psychosocial support service users, two CSQ-8 items showed large ceiling effects, and qualitative data indicated that responses to these items were shaped by community stigma, low expectations, and a lack of alternative services rather than by satisfaction [22]. The Verona Service Satisfaction Scale covers seven dimensions, among them professionals’ skills and behavior, information, access, and efficacy, but it was developed for community-based psychiatric services [23]. Studies in this journal have developed or validated satisfaction scales for emergency room nursing [24] and telepsychiatry [25] and have noted that earlier scales for emergency nursing had been validated without considering the perspectives of the healthcare system or the cultural background of the country [24].

Measures of social support and related constructs, several of them developed for disaster and emergency settings, address neighboring constructs. The Multidimensional Scale of Perceived Social Support assesses perceptions of social support adequacy from family, friends, and a significant other [26]. A crisis support scale used with survivors of the Jupiter cruise ship disaster asked about the availability of others, contact with other survivors, confiding, emotional and practical support, negative responses, and satisfaction with support; crisis support decreased over the following year, and higher crisis support was related to better psychological outcome [27]. The Psycho-Social Support Scale was developed as a brief measure of perceived psychological and social support in clinical and emergency contexts [28], and the Perceived Social Support in Disasters Scale captures perceived social support after disaster along source and function axes, with formal institutions as one of the sources [29]. Other instruments assess perceived needs in humanitarian settings [30] or community resilience, including leadership, collective efficacy, preparedness, place attachment, and social trust [31]. None of these instruments asks survivors to evaluate the organized psychosocial support services they received, and none separates the delivery of such services from what the services brought.

The gap has practical consequences. Donabedian distinguished process, what is actually done in giving and receiving care, from outcome, the effects of care on the health status of patients and populations [32]. A single satisfaction score merges the two, and satisfaction scores are frequently negatively skewed, with ceiling effects that overstate patient experiences, mask variation across subgroups, and limit the ability to detect change [22]. Measures of perceived social support, needs, or community resilience, in turn, describe the survivor’s situation rather than the services received. With these instruments, a program cannot tell whether survivors found a service well organized but of little benefit, helpful for themselves but not for their families, or informative but poorly delivered. Decisions about staffing, coordination, and program content depend on exactly these distinctions.

The present study therefore defined the target construct as adult earthquake survivors’ subjective evaluation of the organized psychosocial support services they received and specified four domains before any item was written. Knowledge refers to the informational gain that the survivor attributes to the service. In the protective action decision model, reception, attention, and comprehension of warnings or cues precede all further processing [33]; experience of earthquakes can raise awareness and knowledge relevant to preparedness [34]; and mental health literacy denotes knowledge and beliefs about mental disorders which aid their recognition, management or prevention [35]. Process refers to the perceived quality of service delivery, the process component of Donabedian’s framework [32], and draws on the SERVQUAL dimensions of consumers’ perceptions of service quality, among them reliability, responsiveness, assurance, and empathy [36].

Individual Benefit refers to the gains in coping, self-efficacy, hope, meaning, and daily functioning that the survivor attributes to the service. It follows the intervention principles described above [9], the concept of perceived self-efficacy as “beliefs in one’s capabilities to organize and execute the courses of action required to produce given attainments” [37] (p. 3), and posttraumatic growth, “the experience of positive change that occurs as a result of the struggle with highly challenging life crises” [38] (p. 1). Community Benefit refers to the gains that the survivor perceives in the family, the neighborhood, and the wider community. It draws on community resilience, a process linking a set of networked adaptive capacities to a positive trajectory of functioning and adaptation in constituent populations after a disturbance [39]; on social capital as part of the social infrastructure that affects community resilience [40]; and on the suggestion that postdisaster relief and intervention programs should help survivors maintain their perceptions of being supported and their trust in the benefits of belonging to a valued social group and community [41].

This article reports the development of the Multidimensional Perceived Psychosocial Support Scale for Earthquake Survivors (MPPSS-ES) and a preliminary evaluation of its psychometric properties in adults affected by the 2023 Kahramanmaraş earthquakes. The aims were to document item development, content validity, and response processes; to identify the factor structure in one subsample and test it, together with alternative models, in another; to examine measurement invariance across demographic groups and differential item functioning (DIF); and to estimate reliability and the convergent and discriminant validity of the subscales within the scale. The study makes three contributions. First, it provides, for the disaster setting, a survivor-report measure of organized psychosocial support that separates the evaluation of service delivery from perceived gains in knowledge, in the survivor’s own life, and in the family and community. Second, it documents each development step, including the expert committee decisions that departed from the cognitive interview recommendations, so that users can judge the evidence behind each item. Third, it gives program evaluators and researchers in post-disaster health and social services four subscale scores that can be compared across demographic groups. Because relations with external measures were not examined and the structure was derived and tested within one sample, the evidence is presented as preliminary.

2. Materials and Methods

2.1. Study Design

The scale was developed and evaluated in three stages (Supplementary Figure S0.1). In Stage 1, a construct map with four domains guided the writing of a 68-item pool, which was reduced to 38 items in a content validity review and then refined through cognitive interviews and an expert committee review. In Stage 2, the 38-item research form was administered to adults affected by the 6 February 2023 Kahramanmaraş earthquakes. In Stage 3, the analytic sample was randomly split into a calibration subsample for the exploratory analyses and a validation subsample for the confirmatory analyses; item screening and all remaining analyses used the full sample. Because the structure was derived and tested within one convenience sample, the study is reported as a preliminary validation. Every stage is documented in Supplementary Materials S1–S16, and the analysis scripts and output files are archived on the Open Science Framework (OSF). Reporting follows the APA Journal Article Reporting Standards [42], the Transparency and Openness Promotion guidelines [43], the Standards for Educational and Psychological Testing [44], the reporting guideline of the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) [45], and the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement for cross-sectional studies [46]; the item-by-item mapping is given in Supplementary Materials S17.

2.2. Item Development and Content Validity

The target construct was defined as adult earthquake survivors’ subjective evaluation of the organized psychosocial support services they received after the earthquakes. Psychosocial support was understood in the broad sense of the IASC guidelines, that is, any type of local or outside support that aims to protect or promote psychosocial well-being and/or prevent or treat mental disorder [7]. The construct concerns the recipient’s appraisal of the service and of what it brought to the person and to the people around them; it does not refer to symptom levels or to support from family and friends in general. Four domains were specified: Knowledge, the informational gain that the respondent attributes to the service; Process, the perceived quality of service delivery; Individual Benefit, perceived personal gains; and Community Benefit, perceived gains in the family, the neighborhood, and the wider community. The domains were derived from a targeted narrative review of theoretical frameworks, international guidelines, and neighboring instruments, and the items were derived from the domain descriptions [47]. The resulting 68 items (Knowledge, 12; Process, 15; Individual Benefit, 22; Community Benefit, 19) were written anew in Turkish for a 5-point response format (Supplementary Materials S1).

Eight experts, four psychologists and four sociologists with field experience in disaster-related psychosocial support, rated each item in a single round for relevance, clarity, and representativeness. The content validity ratio (CVR) was computed for each item and criterion [48], and an item was retained only if its CVR reached 0.75, the minimum for a panel of eight, on all three criteria. Thirty-eight items (55.9%) were retained (Supplementary Materials S2).

The 38 items were then tested in face-to-face cognitive interviews with 30 adults who had received psychosocial support after the earthquakes, using think-aloud, paraphrasing, and follow-up probes [49]. The most common problem was uncertainty about the reference period; participants also found some terms too formal and read “people around me” in different ways. Revisions were proposed for 34 items, together with domain-level guiding sentences and a warning about reverse-keyed items (Supplementary Materials S3).

A four-member expert committee then reviewed the draft (Supplementary Materials S4). Where its judgment differed from the interview recommendations, the committee’s judgment was given priority. The committee edited 16 items and did not adopt the guiding sentences or the warning: each item states its own time and context frame, and the reverse-keyed items were intended to detect inattentive responding. It added “after psychosocial support” to six Community Benefit items and the reference group “other people around me who received psychosocial support” to four others, judging from field experience that survivors usually knew who had received support. The resulting 38-item research form was used in the main study; the edited items were not tested again in a second round of cognitive interviews. The scale was administered only in Turkish, and the English item wording in this article is a reporting translation prepared by forward translation, synthesis, and back translation (Supplementary Materials S4 and S16). The expert ratings, cognitive interviews, and committee review were carried out after ethics approval and before the main data collection.

2.3. Participants and Procedure

The target population was adults aged 18 to 65 years who lived in one of the 11 affected provinces at the time of the earthquakes, had been directly affected (for example, through housing loss, bereavement, injury, property loss, temporary shelter, or displacement), had received psychosocial support after the earthquakes, and were Turkish citizens. People receiving acute psychiatric treatment or with cognitive impairment that prevented completion of the survey were not included. Because no sampling frame of support recipients was available, a nonprobability convenience approach was used: trained interviewers recruited participants door to door in their residential areas and through local community contacts, with no more than one eligible adult per household.

Data were collected face to face with paper forms between 15 February and 15 May 2024, 12 to 15 months after the earthquakes, by 32 trained Bayburt University students living in the region. Interviewer training (about 6 h) covered the protocol, eligibility, informed consent, trauma sensitivity, and administration. Participants completed the form themselves; when a participant had difficulty reading or understanding, the interviewer read the items aloud in a neutral manner. Of 748 people approached, 39 were ineligible, and 668 of the 709 eligible people completed the questionnaire (94.2%). The questionnaire took about 10.4 min on average to complete; this estimate is based on completion times that the interviewers noted on the forms, which were not entered into the dataset. Completed forms were entered twice by independent assistants, and discrepancies were resolved against the original forms.

The study was approved by the Gümüşhane University Scientific Research and Publication Ethics Board (approval date: 24 January 2024; document number: E-95674917-108.99-231613) and was conducted in accordance with the Declaration of Helsinki [50]. All participants gave written informed consent.

2.4. Measures

The MPPSS-ES research form contained the 38 items in four domain sections under the title “After receiving psychosocial support”; no further instruction or guiding sentence was given. Responses were given on a 5-point scale (1 = strongly disagree, 2 = disagree, 3 = partly agree, 4 = agree, 5 = strongly agree). Four items (s6, s15, s28, and s38) were reverse-keyed and were reverse-scored before analysis. The items are coded s1–s38 throughout (Supplementary Materials S4, Table S4.4). The items refer to the period after the respondent received psychosocial support, without a fixed date. Because respondents may have received support at different times before data collection and the timing of support was not recorded, the reference period of this retrospective appraisal may differ between respondents. Throughout this article, domain refers to a content area of the construct map, factor to the corresponding latent variable in a factor model, and subscale to the scored set of items. Each subscale was specified as a reflective measure, with its items as indicators of one latent factor.

The survey also recorded gender, age group, marital status, education, and employment status, and included the 42-item Depression Anxiety Stress Scales (DASS-42) [51], which was administered for the research question of the doctoral thesis from which this study derives, the relation between the psychosocial support received and trauma- and stress-related symptoms. The MPPSS-ES evaluates services rather than symptom levels, and relations with other constructs require hypotheses stated in advance (Section 4.6); the DASS-42 was therefore not used as validity evidence in this report and served only in the data quality checks. No measure of socially desirable responding was included, and because the four reverse-keyed items were removed during item reduction, they could not be used to detect inattentive or acquiescent responding in the final form.

2.5. English Reporting Translation

The MPPSS-ES was developed and administered in Turkish; the English version was prepared for reporting and was not administered. Two independent bilingual translators, native Turkish speakers with academic proficiency in English and familiarity with psychological measurement, produced forward translations, and the research team reconciled them into a synthesized version. An independent translator, a native English speaker blind to the source items, translated this version back into Turkish. The research team compared the back-translation with the Turkish source and finalized the English wording; no separate expert committee reviewed the translation. The final wording departs from the synthesized version in four items where the synthesis did not follow the Turkish wording closely (Supplementary Materials S16; Translation document in the OSF repository). Because the English version was not administered, its measurement equivalence with the Turkish form is unknown.

2.6. Data Quality and Missing Data

Two exclusion criteria were applied to all 80 Likert-type items of the survey: more than 10% of the items unanswered, or the same response code given to at least 80% of the 80 items (MPPSS-ES items coded 1–5 and DASS-42 items coded 0–3). Thirty of the 668 questionnaires were excluded, leaving an analytic sample of 638. In this sample, 70 of the 51,040 item responses (0.14%) were missing. An approximate version of Little’s test [52] did not reject the assumption that the MPPSS-ES responses were missing completely at random (χ²(924) = 901.61, p = 0.695), and missingness was not predicted by demographic variables or scale means. Missing responses were replaced by a single expectation–maximization imputation [53] that treated the items as ordinal; no subscale mean of the final form changed by 0.001 or more (Supplementary Materials S5).

2.7. Statistical Analysis

The analytic sample was split into a calibration subsample (n = 322) and a validation subsample (n = 316), stratified by the eight combinations of gender and age group. The subsamples were compared on the demographic variables with χ² tests, Welch’s t tests, and equivalence tests (Supplementary Materials S7). Item distributions and item–total statistics were examined in the full sample before the split; no item was removed on the basis of this screening (Supplementary Materials S8).

Exploratory factor analyses were run in the calibration subsample. Several items had empty or sparse response categories, which made the polychoric matrix nearly singular, so the primary analyses used smoothed Spearman correlations with minimum residual extraction and promax rotation; the final solution was re-estimated from polychoric correlations as a check. The number of factors was examined with parallel analysis [54] (500 simulated data sets, 95th percentile) and eight further criteria. An item was removed if its primary loading was below 0.30 or its communality below 0.10; an item with a primary loading below 0.40, a communality below 0.30, or a difference below 0.15 between its primary and secondary loadings was flagged, and flagged items were removed one at a time. Exploratory graph analysis [55] (EGA; graphical lasso with the extended Bayesian information criterion, γ = 0.50, and walktrap community detection on absolute edge weights) was applied to the retained items as a check that does not assume a factor model (Supplementary Materials S9).

Confirmatory factor analyses were run in the validation subsample with the items treated as ordinal, using weighted least squares mean- and variance-adjusted (WLSMV) estimation with theta parameterization, because the five response categories were used asymmetrically, a condition to which normal theory maximum likelihood is more sensitive [56]. The primary model (M1) specified four correlated factors, with each item loading only on its intended domain. M1 was compared with one-factor, second-order, uncorrelated-factor, bifactor, and two-factor models and with a model of the EGA communities. Fit was judged with the scaled χ², the comparative fit index (CFI), the Tucker–Lewis index (TLI), the root mean square error of approximation (RMSEA) with its 90% confidence interval, and the standardized root mean square residual (SRMR), and nested models were compared with scaled χ² difference tests. Cutoff values close to 0.95 for CFI and TLI, 0.06 for RMSEA, and 0.08 for SRMR [57] were used as rough guides only, because they were derived for maximum likelihood estimation. A post hoc model (M1a) freed four within-domain residual covariances selected from the modification indices of M1 in the validation subsample; because M1a was specified after inspecting these data, M1 remains the confirmatory test. Sensitivity analyses examined the residual correlations of M1, the removal of single items, robust maximum likelihood (MLR) estimation with the items treated as continuous, and a four-factor exploratory structural equation model (ESEM) [58] with geomin rotation (Supplementary Materials S10).

In the full sample, measurement invariance of M1a was tested across age group, gender, marital status, employment status, and education (grouped into three levels). Identification constraints for ordinal indicators followed Wu and Estabrook [59], and constraints were added in nested steps from configural to strict invariance; each step was compared with the previous one by the scaled χ² difference test [60] and the changes in CFI and RMSEA, which were interpreted against the criteria of Chen [61]. Where a response category was empty in a group, it was merged with the adjacent category in all groups. DIF was examined with ordinal logistic regression, using the rest score of the item’s subscale as the matching variable; an item was flagged if the change in McFadden’s pseudo-R² reached 0.010, and p values were Holm-adjusted; for DIF, employment status was dichotomized as employed versus not employed (Supplementary Materials S11).

Reliability of each subscale was estimated in the full sample with Cronbach’s α, McDonald’s ω from M1a, which counts the shared variance of the residual covariance pairs as error, and coefficient H; 95% confidence intervals for α and ω were bias-corrected and accelerated bootstrap intervals (500 resamples). The standard error of measurement of each subscale mean was computed as SD × √(1 − ω) (Supplementary Materials S12). Within-subscale convergence was summarized by the average variance extracted (AVE). Discriminant validity was examined with the factor correlations, the heterotrait–monotrait ratio (HTMT) [62], computed from absolute polychoric correlations because the Knowledge items correlate negatively with the other items, and the Fornell–Larcker criterion [63] (Supplementary Materials S13). No external measure was analyzed.

Because all data were self-reports from one source at one time point, common method variance [64] was examined with Harman’s single-factor test, a comparison of M1a with a one-factor model, and models adding an unmeasured latent method factor to M1a (Supplementary Materials S14). Subscale scores were computed as item means (range 1–5); floor and ceiling effects were defined as the percentages of respondents with the lowest and highest possible subscale scores (Supplementary Materials S15). Analyses were conducted in R version 4.5.2 with the psych, EFAtools, qgraph, igraph, lavaan [65], semTools, ordinal, caret, and Amelia packages, with fixed random seeds for all resampling. During manuscript preparation, the authors used large language model (LLM) tools for language editing and statistical-code drafting; all LLM-assisted output was reviewed and approved by the authors, who accept full responsibility for the content. No identifiable participant information was entered into any LLM tool.

3. Results

3.1. Sample Characteristics

The analytic sample comprised 638 adults, 323 (50.6%) of them women (Table 2). The largest age group was 51–65 years (32.9%), 67.6% of the participants were married, 72.4% had completed primary to high school, and 53.6% were not employed. The calibration (n = 322) and validation (n = 316) subsamples did not differ in gender, age group, marital status, or employment status. The calibration subsample had a slightly higher education level (Hedges’ g = 0.24, p = 0.002), and the two subsamples could not be regarded as equivalent in education (Supplementary Materials S6 and S7).

3.2. Item Analyses and Exploratory Structure

In the full sample, item means ranged from 2.38 to 4.14, most items were negatively skewed, and 12 items had more than 25% of responses in the highest category (Supplementary Materials S8). In the calibration subsample, eight items had at least one empty response category, all in the disagreement range, and the sampling adequacy of the 38-item pool was borderline (Kaiser–Meyer–Olkin index = 0.597 with Pearson and 0.598 with Spearman correlations; Supplementary Materials S7). Treated as a single scale, the 38 items had low internal consistency (α = 0.635; mean inter-item correlation = 0.053), and five of the six Knowledge items correlated negatively with the total of the pool. Because the pool was written to cover four domains, item removal was left to the factor analyses, and no item was removed on the basis of this screening.

With all 38 items, parallel analysis suggested 10 factors and the other criteria between 3 and 25, so the four-factor solution specified by the construct map was used as the starting point. Eleven items with a primary loading below 0.30 or a communality below 0.10 were removed first, followed by s19 and s8. For the remaining 25 items, parallel analysis suggested five factors. The fifth factor consisted of five Community Benefit items (s30, s32, s33, s35, and s38) with mixed content that did not correspond to a domain of the construct map; in the four-factor solution these items were flagged and removed one at a time. In total, 18 items were removed, including all four reverse-keyed items (Supplementary Materials S9, Table S9.1).

For the final 20 items, four factors were supported by parallel analysis, the eigenvalue rule, very simple structure (complexity 2), and the Hull CFI and RMSEA indices, whereas Velicer’s minimum average partial test and the Hull method based on the common part accounted for suggested five, very simple structure (complexity 1) suggested seven, and the Bayesian information criterion was still decreasing at eight factors (Supplementary Materials S9, Table S9.2). In the four-factor solution (Table 3), every item loaded on the factor of its intended domain, with primary loadings from 0.676 to 0.999, no secondary loading above 0.17 in absolute value, and communalities from 0.52 to 0.98; the four factors explained 77.4% of the variance. The Knowledge factor correlated negatively with the other three factors (−0.25 to −0.17), and the other factor correlations ranged from −0.04 to 0.26. The descriptive fit of this solution was poor (TLI = 0.689, RMSEA = 0.201, root mean square residual = 0.040), and some item pairs were answered almost identically (s7 and s10, r = 0.944). Re-estimation from polychoric correlations gave the same item–factor assignment, with Tucker congruence coefficients of 0.998 or higher.

EGA identified three communities rather than four. The Individual Benefit items formed one community, the Process items grouped with four Community Benefit items (s29, s34, s36, and s37), and the remaining Community Benefit item (s31) grouped with the Knowledge items (Supplementary Materials S9, Table S9.7). EGA thus separated Individual Benefit and Knowledge from the other items but did not separate Process from Community Benefit.

3.3. Confirmatory Factor Analysis

In the validation subsample, M1 had high CFI (0.976) and TLI (0.972) values, but its RMSEA (0.103, 90% CI [0.096, 0.111]) and SRMR (0.086) were above the conventional guides (Table 4). The one-factor, two-factor, and EGA-community models fitted clearly worse (CFI = 0.832–0.933; RMSEA = 0.171–0.268), and M1 fitted significantly better than the one-factor model (Δχ²(6) = 770.13, p < 0.001) and the uncorrelated-factor model (Δχ²(6) = 49.28, p < 0.001). The second-order model did not fit significantly worse than M1 (Δχ²(2) = 3.43, p = 0.180), but the Knowledge factor loaded negatively on its general factor (−0.509), whereas the other three factors loaded positively (0.286 to 0.601). In the bifactor model, the general factor accounted for little of the total score variance (omega hierarchical = 0.116), and its loadings were positive for the Knowledge items but mostly negative for the other items. Neither model represented one coherent overall construct, and M1 was retained (Supplementary Materials S10).

All standardized loadings in M1 were significant and ranged from 0.640 (s31) to 0.969 (s25) (Table 3; Figure 1). The Knowledge factor correlated negatively with Process (r = −0.23), Individual Benefit (r = −0.15), and Community Benefit (r = −0.29), as in the calibration subsample. Process and Individual Benefit were practically uncorrelated (r = 0.06, p = 0.355), and the remaining two correlations were small and positive (0.14 and 0.25).

The largest modification indices of M1 concerned four within-domain residual covariances (s16–s20, s7–s10, s9–s12, and s1–s2). Freeing them in the post hoc model M1a improved fit (Δχ²(4) = 345.73, p < 0.001; RMSEA = 0.088, 90% CI [0.080, 0.096]; SRMR = 0.083), and the factor correlations hardly changed. Because these pairs were selected in the validation data, M1a is not an independent test; in the calibration subsample, two of the pairs (s16–s20 and s1–s2) ranked only 26th and 30th among the within-domain modification indices. M1a was used in the subsequent analyses so that the dependence within these pairs was not counted as common factor variance.

No single item accounted for the misfit: with one item removed at a time, the scaled RMSEA of M1 ranged from 0.100 to 0.109 (the scaled indices could not be computed without s3). With MLR estimation and the items treated as continuous, M1a fitted clearly worse (CFI = 0.783, RMSEA = 0.160). Of the 190 residual correlations in M1, 43 exceeded 0.10 in absolute value, and 41 of these involved items from different domains, most often Knowledge and Community Benefit items. In the ESEM, every item had its largest loading on the intended factor (0.650 to 0.990), and only two cross-loadings reached 0.20 in absolute value (s20 on Process, 0.26; s34 on Knowledge, −0.31). In the full sample, the polychoric correlation between s7 and s10 was estimated as nearly 1, and lavaan warned that the covariance matrix of the parameter estimates of M1a was not positive definite, so standard errors and tests involving these items should be read with caution (Supplementary Materials S10).

3.4. Measurement Invariance and Differential Item Functioning

Before the multigroup analyses, adjacent response categories had to be merged for 6 to 14 of the 20 items, depending on the grouping variable, and gender invariance was tested with 19 items because the configural model with all 20 items did not converge (Supplementary Materials S11). For all five grouping variables, equal thresholds, loadings, and intercepts did not significantly worsen the fit of M1a (all p ≥ 0.224), CFI changed by 0.001 at most, and RMSEA decreased at every step (Table 5). By these criteria, scalar invariance was supported. Equal residual variances produced a significant χ² difference for every variable (p < 0.001), although CFI changed by no more than 0.004, so strict invariance was not supported by the χ² test. The employment models with equal thresholds, loadings, or intercepts contained standardized loadings above 1 (up to 1.046) and negative residual variances, and in several models some item pairs had within-group polychoric correlations close to 1. Scalar invariance by employment status was therefore not established. No item showed DIF for any grouping variable; the largest change in McFadden’s pseudo-R² was 0.005, and no Holm-adjusted p value was below 0.05.

3.5. Reliability and Convergent and Discriminant Validity

Cronbach’s α ranged from 0.858 to 0.953, ω from 0.828 to 0.960, and coefficient H from 0.935 to 0.970 (Table 6). For Knowledge and Process, ω was lower than α (0.891 vs. 0.926 and 0.828 vs. 0.858), because ω from M1a treats the shared residual variance of the paired items as error. The mean inter-item correlations of Knowledge (0.76) and Individual Benefit (0.77) were high for subscales of four and six items. The standard error of measurement of the subscale mean scores ranged from 0.18 to 0.28 points, and the corrected item–rest correlations ranged from 0.50 (s14) to 0.92 (s1) (Supplementary Materials S12).

The AVE ranged from 0.656 (Community Benefit) to 0.833 (Individual Benefit). In M1a fitted to the full sample, the factor correlations ranged from −0.282 to 0.280, and no confidence limit exceeded 0.369 in absolute value (Table 7). The Knowledge factor again correlated negatively with the other three factors, and Process and Individual Benefit were uncorrelated (φ = −0.001, 95% CI [−0.091, 0.090]). HTMT values ranged from 0.091 to 0.290, and the Fornell–Larcker criterion was met for all six pairs. At the item level, correlations were strong within the subscales and weak, partly negative, between them (Figure 2). No external measure was analyzed, so these results concern the internal structure of the scale only (Supplementary Materials S13).

3.6. Common Method Variance

In Harman’s test, the first unrotated factor accounted for 26.3% of the variance, and a one-factor confirmatory model fitted poorly (RMSEA = 0.288, SRMR = 0.414). A latent method factor with equal loadings was estimated at practically zero, and the model was equivalent to M1a (Δχ²(1) = 0.00, p = 1.000). With free loadings, the method factor improved fit (Δχ²(20) = 326.34, p < 0.001), but its loadings ranged from −0.556 to 0.405, positive for all Knowledge items and negative for all Process items. This pattern is closer to the cross-domain associations described above than to a uniform response tendency. Adding the method factor changed the substantive loadings by 0.039 on average, with the largest change for s14 (0.101) (Supplementary Materials S14).

3.7. Subscale Score Distributions

Subscale means ranged from 3.66 (Individual Benefit) to 4.01 (Process), and all four distributions were slightly negatively skewed (Table 6; Supplementary Materials S15). The highest possible score was obtained by 24.5% of the respondents on Knowledge, 22.9% on Process, 17.7% on Individual Benefit, and 12.2% on Community Benefit, whereas the lowest possible score was rare (at most 0.9%).

4. Discussion

The MPPSS-ES provides preliminary evidence for a 20-item, four-domain measure of how adult earthquake survivors evaluate the organized psychosocial support they received. The four-factor structure specified before item writing was the starting point of the exploratory analyses, in which 18 of the 38 items were removed, and it was reproduced in the validation subsample, where every item loaded on its intended factor in both the confirmatory model and the ESEM, although RMSEA and SRMR did not indicate close fit. Scalar invariance was supported across age group, gender, marital status, and education, no item showed DIF, and internal consistency was high. The evidence concerns the internal structure of the scale; whether the subscale scores relate to other constructs in the expected way has not yet been tested.

4.1. Interpretation of the Four-Domain Structure

The four domains were only weakly related. In the confirmatory model, the factor correlations ranged from −0.29 to 0.25, and neither a second-order nor a bifactor model yielded a general factor that could be read as one overall evaluation. In these data, what survivors reported learning, how they rated the delivery of the service, and what they felt it changed for them and around them varied largely independently, although the layout of the form, with visible domain headings, may have contributed to this separation (Section 4.5). This is the main difference between the MPPSS-ES and global satisfaction measures such as the CSQ-8, for which exploratory factor analysis supported a single factor [22], and it is the reason why the MPPSS-ES is scored as a profile of four subscales rather than as a total. A review of mental health satisfaction scales suggested that subscales covering different aspects of care are likely to be correlated and that one global score should usually be measured when satisfaction is a trial outcome [17]. In the present data the subscales were weakly correlated, so for survivors’ evaluations of organized psychosocial support they appear to carry separate information. In the terms of the distinction between experience and satisfaction measures [18], the MPPSS-ES is an evaluation measure: it asks survivors to judge the services and what they brought rather than to report what happened during an encounter. Its subscales were also far less related than those of a patient experience measure for addiction and mental health services, whose five subscales correlated from 0.47 to 0.71, with 0.47 between treatment effectiveness and staff behavior [20]. The disaster setting, the diversity of the services received, or the layout of the form may account for this difference, which should be examined in new samples.

The near-zero correlation between Process and Individual Benefit (r = 0.06) is informative in its own right. Donabedian separated process, what is actually done in giving and receiving care, from outcome, the effects of care on the health status of patients and populations [32]. The present data suggest that survivors made a similar distinction: how favorably they rated the delivery of the service was practically unrelated to how much they felt it had changed their own coping or outlook. Evaluations that rely on satisfaction with delivery alone may therefore say little about perceived benefit. The contrast is not a pure one between process and outcome, however: the Process subscale also includes an item on whether the support helped to solve problems (s10) and one on feeling supported by the community during the support process (s9).

The negative correlations of the Knowledge factor with the other three factors (−0.29 to −0.15) were not expected. The coding of the Knowledge items was checked, and the same pattern appeared in both subsamples. Several explanations are possible, and the data do not allow them to be separated. Survivors may have received different kinds of support, for example information-oriented psychoeducation rather than individual counseling, and the type of support received was not recorded. Survivors who reported greater knowledge gains may also have judged the delivery and effects of services more critically. A response style related to the layout of the form, in which the Knowledge items formed the first domain section, cannot be ruled out either. In addition, the Knowledge items ask about knowledge of disasters and emergencies and of where to seek help, and do not name the service; their link to the support received rests on the title of the form (“After receiving psychosocial support”). Whichever explanation holds, the finding argues against summing the subscales, and future studies should record the type of support received so that this pattern can be examined directly.

Exploratory graph analysis did not separate Process from Community Benefit, and four Community Benefit items grouped with the Process items. The distinction between these two domains therefore rests mainly on the factor analyses. The two domains differ in content, one describing the service and the other the changes observed among other people, but both refer to what the respondent observed outside themselves, which may explain why they formed one community in the network model. The distinction should be tested again in new samples. Within the domains, some items were answered almost identically; in the calibration subsample, s7 and s10 correlated at 0.944. The mean inter-item correlations of Knowledge and Individual Benefit were also high (0.76 and 0.77), which points to narrow content as much as to precision.

The MPPSS-ES should also be distinguished from measures of perceived social support. The Perceived Social Support in Disasters Scale, developed in survivors of the same earthquakes, captures perceived social support along source and function axes [29], and the Multidimensional Scale of Perceived Social Support assesses support adequacy from family, friends, and a significant other [26]. The MPPSS-ES asks instead about organized services that the respondent has received. Its Community Benefit subscale refers to changes among family, neighbors, and others after psychosocial support, not to the support that these people provide.

4.2. Measurement Invariance and Practical Comparability

Scalar invariance across age group, gender, marital status, and education means that latent mean differences between these groups can be interpreted as differences in evaluation rather than in the meaning of the items, and no item showed DIF. Three qualifications apply. Strict invariance was not supported, so comparisons of observed subscale means assume equal measurement precision across groups that was not fully shown. Response categories had to be merged for many items, and gender invariance was tested with 19 items. The employment models contained improper solutions, so scalar invariance by employment status was not established, and comparisons by employment status should await further evidence. Most importantly for service evaluation, invariance was not tested across the characteristics of the support itself, such as the type of service, the provider, or the time since support was received. These are the comparisons that program evaluators are most likely to make.

4.3. Methodological Tensions

The confirmatory fit indices pointed in different directions. CFI and TLI were high, whereas RMSEA (0.103 for M1 and 0.088 for M1a) exceeded the values usually taken to indicate close fit. The cutoffs proposed by Hu and Bentler were derived for maximum likelihood estimation [57], and indices from WLSMV estimation with ordinal items do not translate directly into these guides. When the items were treated as continuous, fit was clearly worse (CFI = 0.783, RMSEA = 0.160). The misfit was concentrated in associations between items of different domains: of the 43 residual correlations above 0.10 in absolute value, 41 involved items from different domains, most often Knowledge and Community Benefit items. The results therefore support the assignment of items to domains more clearly than they support close fit of the model. M1a, which freed four residual covariances selected in the validation subsample, was used in the later analyses so that the dependence within these pairs was not counted as common variance; its post hoc status applies to all results that depend on it.

Two further decisions shaped the results. First, sparse response categories made the polychoric correlation matrix nearly singular, so the exploratory analyses used Spearman correlations. The factor pattern was the same with polychoric correlations, but the fit indices were not, so the exploratory fit indices depend on the type of correlation and are descriptive only. Second, with 25 items, parallel analysis suggested five factors, and the fifth consisted of five Community Benefit items on kinship relations, children’s joy and return to school, solidarity, and unmet needs. These items were removed because the group did not correspond to a domain of the construct map and its content was mixed. They may nevertheless reflect an aspect of community benefit that the construct map did not specify. All four reverse-keyed items were also removed during item reduction, so the final form is positively keyed throughout.

4.4. Practical Implications

The four-subscale profile supports uses that a single satisfaction score cannot. A program whose recipients rate Process highly but Individual Benefit low is delivering services that are well received but not experienced as helpful, and the near-zero correlation between these two subscales among respondents suggests that such profiles can occur; whether they distinguish programs requires data collected across programs. A program with low Knowledge scores may need to strengthen its informational components, and low Community Benefit scores may indicate that services reached individuals without being seen to change family or community life. Because the Knowledge factor correlated negatively with the other factors, a high score on one subscale should not be read as implying a high score on the others.

Several cautions follow from the results. The MPPSS-ES is not a diagnostic instrument, and no norms or cut-off scores are available. The standard error of measurement of the subscale means (0.18 to 0.28 points) supports comparisons of groups rather than of individuals, and test–retest reliability and responsiveness have not been examined. Between 12.2% and 24.5% of respondents obtained the highest possible subscale score, so differences among respondents who evaluate services very favorably are hard to detect. Satisfaction measures used in humanitarian settings show the same problem, in which ceiling effects overstate patient experiences, mask variation across subgroups, and limit the ability to detect change [22]. Perceptions of support also change over time after a disaster [27,41], so users should record when support was received and when the scale was completed. Finally, the form was tested with its title, domain headings, item order, and response labels, and users who wish to compare their results with this study should keep that layout.

In Türkiye, psychosocial support after the 2023 earthquakes was delivered by many institutions under difficult conditions, and professionals reported short-term staffing, limited coordination, and insufficient infrastructure [13]. Provider accounts of this kind and recipients’ evaluations answer different questions. A recipient-based measure can show where services were experienced as poorly organized or of little benefit, and it can do so in the same terms across providers and regions.

4.5. Limitations and Future Directions

The limitations of this work concern the items and the form, the sample, and the scope of the evidence. They are set out in that order and are not repeated in the Conclusions.

First, the expert committee edited 16 items after the cognitive interviews, and these items were not tested again in a second round of interviews before the main study. The committee also added the phrase “other people around me who received psychosocial support” to four Community Benefit items, although the cognitive interviews had shown that wording which requires knowledge of other people’s service use could not always be answered reliably. The committee judged, from field experience, that survivors usually knew who had received support; two of these items (s36 and s37) remained in the final form. Their functioning in the main study was adequate, but respondents who do not know who received support may find them hard to answer, and cognitive interviews on the final wording are needed. The items on children and on other people who received support (s34, s36, and s37) also had no “not applicable” option.

Second, the form showed the domain headings to respondents and used “partly agree” as its midpoint. Visible headings may have strengthened the grouping of items by domain, and the midpoint may have been used both for partial agreement and for uncertainty. A randomized comparison of layouts with and without headings would show whether the structure depends on the layout. Third, the items refer to the period after support was received, without a fixed date, and the timing of support was not recorded. Because data were collected 12 to 15 months after the earthquakes, respondents may have been evaluating services received at different times, and recall may have affected their judgments. Receipt of support was self-reported, and the type of support and the provider were not recorded, so respondents may have evaluated different kinds of services.

Fourth, the sample was a nonprobability convenience sample of adults aged 18 to 65 years who had received psychosocial support after one disaster in one country. The calibration and validation subsamples were drawn from the same sample, so their agreement shows internal replicability rather than generalizability. The two subsamples also differed slightly in education, and no a priori sample size or power calculation was made. Survivors who had left the region, older adults, and people who did not receive support were not included. The structure should be tested in independent samples, after other disasters, and in other populations before the scale is used outside the present setting.

Fifth, the evidence is limited to content, response processes, internal structure, invariance, and internal consistency. No external measure was analyzed, so convergent and discriminant validity were examined only among the subscales, and the relations of the subscales to service satisfaction, perceived social support, distress, or functioning are unknown. Test–retest reliability and responsiveness were not assessed; in a review of 88 PREMs, these properties were also among the least often tested, with responsiveness not tested for 82 instruments and measurement error for 86 [66]. The English item wording is a reporting translation that was not administered. The final form contains no reverse-keyed items, and no measure of socially desirable responding was included, so acquiescence and social desirability could not be separated from substantive responding. Because all data came from the same respondents at one time point, and obtaining measures from the same source is a major cause of common method variance [64], such variance cannot be ruled out. Finally, the study was not preregistered, and the post hoc residual covariances of M1a make the fit and reliability estimates based on this model somewhat optimistic.

4.6. A Validation Roadmap for Prospective Users

The limitations translate into an ordered sequence of studies (Table 8). The first priority is a confirmatory replication in an independent sample, with M1 and the four residual covariances of M1a specified in advance, together with a test of relations to other variables. Hypotheses should be stated before data collection: for example, Process should correlate more strongly with a service satisfaction measure than with a measure of perceived social support, and Individual Benefit should be inversely related to psychological distress. The second priority concerns the items and the form: cognitive interviews on the final wording, a comparison of layouts with and without domain headings, and a test–retest study. Later steps address invariance across characteristics of the support received, responsiveness in longitudinal designs, revision of near-redundant items, and adaptation to other languages and disaster types.

5. Conclusions

Survivors of the 2023 earthquakes have lacked a validated instrument for evaluating the organized psychosocial support they received. The MPPSS-ES gives adult earthquake survivors a structured way to evaluate the organized psychosocial support they received, in four domains: what they learned, how the service was delivered, what it changed for them, and what it changed in their family and community. In the present sample, the four domains were distinct, their items functioned similarly at the scalar level across age group, gender, marital status, and education, and their scores were internally consistent. The domains were only weakly related, and knowledge gains were inversely related to the other domains, so the scale should be used as a profile rather than as a total score. The evidence is preliminary and concerns internal structure only; relations with external measures, stability over time, and sensitivity to change remain to be shown. With these limits in view, the MPPSS-ES can help program evaluators and researchers hear from the people who received psychosocial support after disaster, alongside the accounts of those who provided it.