Validity and Internal Consistency of a Rubric for Cervical Collar Placement in Nursing Students

1. Introduction

Nursing education has undergone significant changes in recent decades, gradually shifting from models focused on the transmission of theoretical content to educational approaches geared toward the integrated development of knowledge, technical skills, and complex clinical competencies. This change responds to the need to prepare future professionals for increasingly demanding healthcare environments, where decision-making, patient safety, and clinical reasoning play a fundamental role [1,2].

In this context, clinical simulation has established itself as an effective pedagogical strategy in healthcare training, allowing for the training of technical and non-technical skills in controlled and safe environments. Several studies have shown that simulation allows for the integration of theory and practice, promotes the progressive acquisition of skills, improves clinical performance, and increases the safety of real patients, both for students and professionals in continuing education [3,4,5,6,7].

Among the essential maneuvers in the initial care of polytrauma patients is the placement of a cervical collar, a key procedure for preventing secondary spinal cord injuries and ensuring adequate stabilization of the cervical spine. Despite being a widely protocolized technique, the literature describes considerable variability in its execution, especially in training contexts, which can compromise clinical safety and quality of care [8,9]. This variability highlights the importance of having not only clear protocols, but also assessment tools that allow for objective evaluation of the level of competence achieved.

In education, rubric-based assessment has been proposed as a structured and transparent strategy for evaluating clinical performance, breaking down complex procedures into observable criteria and establishing progressive levels of mastery. Rubrics contribute to improving the objectivity of assessment, facilitate formative feedback, and promote self-regulation of learning, especially when self-assessment and peer assessment processes are integrated [10,11,12]. However, despite the existence of well-defined technical protocols for cervical collar placement, the literature shows a shortage of validated instruments specifically designed to assess this skill in clinical simulation settings, particularly in the context of nursing education. This lack of psychometrically sound tools limits the standardization of assessment and makes it difficult to compare results across training cohorts and teaching contexts. [3,7,13,14]

Within this framework, the objective of the present study was to analyze the psychometric properties—interrater reliability, internal consistency, and factorial structure—of a rubric designed to assess the technique of cervical collar placement in nursing students in a clinical simulation context.

2. Materials and Methods

2.1. Design

A quasi-experimental, cross-sectional study was conducted to validate a rubric designed to assess the technique of placing a cervical collar in a clinical simulation setting. The study was conducted at a public university in Spain, as part of a compulsory subject in the Nursing Degree programme. The cross-sectional design allowed for a snapshot of the performance of participants in different academic cohorts and an assessment of the applicability, consistency, and relevance of the rubric. The study considered four fundamental aspects: 1) content validity (assessed by experts); 2) inter-rater reliability (determined using the Kappa index); 3) internal consistency (estimated using Cronbach’s alpha coefficient); and 4) internal structure using factorial analysis.

The data were collected in February of 2022, 2023, and 2024 in five 2-hour sessions each academic year.

2.2. Population and Sample

The sample consisted of nursing degree students enrolled in the course ‘Fundamentals of Health Care.’ Special Care”, corresponding to the second year of the curriculum, in the academic years 2021-2022, 2022-2023 and 2023-2024. Convenience sampling was used, with a total of 186 students divided into 62 groups of 3 people each. Participation was voluntary and anonymous, and verbal informed consent was obtained from the students. The sample size was determined using the QuestionPRO sample size calculator [15], considering a 95% confidence level, a 5% margin of error and an expected proportion of 50%, which yielded a minimum requirement of 170 participants.

2.3. Instruments

An ad hoc rubric was used to evaluate the cervical collar placement procedure. This tool was developed based on the judgement of three experts with clinical experience and teaching experience in emergency medicine and clinical simulation.

The rubric used consists of eight items covering the steps of selecting, measuring, and placing the collar. Each item is scored on an ordinal rating scale from 1 to 4, where 1 represents incorrect or poor performance, and 4 represents autonomous and correct performance. The maximum score is 32 points and the minimum is 8 points. This structure allows for the assessment of both technical and behavioural aspects of clinical performance in simulation.

To reduce information bias, the scale administration procedure was standardised. It was delivered in paper format to each group before the procedure was performed, using uniform instructions and the supervision of three trained instructors, who were responsible for all administrations to ensure the homogeneity of the process over the three years. The items were scored by the students themselves, according to the levels of mastery established in the rubric, which allowed for the application of uniform criteria and reduced variability in the interpretation of the items, minimising the risk of evaluator bias.

2.4. Procedure

The intervention was carried out in a clinical simulation classroom during a two-hour practical session, in which a scenario involving the fitting of a cervical collar was developed. In order to include all enrolled students, the session was repeated four times in each academic year, over a period of three years.

Each session involved five teams made up of two groups of three students each, encouraging active participation and collaborative learning.

-Start (Briefing): the instructor presented the task to be performed—fitting a cervical collar—and gave a brief theoretical explanation of the technique, followed by a practical demonstration and presentation of the assessment rubric, including its structure, the levels of proficiency it comprises, and the criteria for assigning scores. The students organised themselves into groups of three, taking on the roles of victim, responder 1 and responder 2. Each group was assigned a number and paired with another group to carry out the activity.

-Development of the scenario: the students placed the cervical collar under the direct supervision of the instructor, ensuring uniform application of the rubric and promoting adherence to clinical practice standards. The process was carried out in two phases determined by the mode of assessment of the scenario used: peer assessment or self-assessment (Figure 1)

• First phase: Group 1 performed the procedure and Group 2 observed. Upon completion, group 2 evaluated group 1 (peer evaluation), while group 1 performed its self-evaluation.
• Second phase: Group 2 performed the procedure and group 1 observed. Upon completion, group 1 evaluated group 2 (peer evaluation), while group 2 performed its Self-evaluation.

The data were analyses using IBM SPSS Statistics version 26.0 (IBM Corp., Armonk, NY, USA). A cleaning and verification process was performed beforehand to identify missing values, duplicates, and inconsistencies.

2.5. Variables

The independent variables are the group, the academic year, and the assessment method (self-assessment or peer assessment). The dependent variables correspond to the mastery indicators and performance levels of the rubric items.

2.6. Analysis

Measures of central tendency and dispersion (mean, mode, standard deviation) were obtained for the dependent variables, and a one-factor ANOVA was performed to analyze the differences between groups. A statistical significance level of p < 0.05 was established.

Although no predefined confounding variables were included, the academic year was considered a potential influencing factor and was analyzed specifically to minimize a possible cohort effect. In accordance with STROBE guidelines, analytical subgroups were explored, comparing scores between the three academic years (2021–2022, 2022–2023, 2023–2024) and between the two assessment methods. Likewise, the possible interaction between course and method was examined exploratorily by comparing means in each combination using descriptive analyses and ANOVA.

To analyze inter-rater reliability, Cohen’s Kappa index, suitable for nominal variables, was used, calculated for each item and considering the two groups as evaluators: self-assessment and peer assessment, in order to determine the degree of agreement between the self-assessment and peer assessment scores. The Kappa values were interpreted according to the classification proposed by Landis and Koch (1977), according to which values < 0 indicate agreement worse than chance; 0.00–0.20 slight agreement; 0.21–0.40 acceptable agreement; 0.41–0.60 moderate agreement; 0.61–0.80 substantial (good) agreement; and 0.81–1.00 almost perfect agreement.

The internal consistency of the instrument was analyzed using Cronbach’s alpha coefficient, both for the rubric as a whole and for the different subgroups of items that comprise it, with the aim of evaluating the degree of internal homogeneity and the correlation between items, as well as their contribution to the common construct. Cronbach’s alpha assumes that the items measure the same dimension and that the responses have a logical order. The interpretation of Cronbach’s alpha values was carried out following the classification proposed by Tavakol and Dennick (2011), which is widely accepted in the fields of health sciences and education. According to this classification, values of α ≥ 0.90 indicate excellent consistency; between 0.80 and 0.89, very good; between 0.70 and 0.79, good; between 0.60 and 0.69, moderate; between 0.50 and 0.59, low; and values below 0.50, very low.

The internal structure of the rubric and its structural validity were assessed using exploratory factor analysis (EFA). The suitability of the data for exploratory factor analysis was evaluated using the Kaiser–Meyer–Olkin (KMO) sample adequacy index and Bartlett’s sphericity test. KMO values ≥ 0.60 were considered acceptable, while a statistically significant Bartlett test (p < 0.05) was interpreted as indicative of sufficient correlations between items for the application of factor analysis. Factor retention was based on the criterion of eigenvalues greater than 1 (Kaiser criterion) and on inspection of the scree plot. A factor solution that explained at least 50% of the total variance was considered adequate. To facilitate the interpretation of the factors, a Varimax orthogonal rotation with Kaiser normalisation was applied, with the aim of maximising the variance explained by each factor. For the interpretation of the factorial structure, a minimum factor loading 0.40 in a single factor was established as the criterion for retaining items. Items with loadings below this value or with significant cross-saturations were reviewed for possible elimination or conceptual reassignment.

2.7. Ethical Considerations

The study was conducted in accordance with the principles of the Declaration of Helsinki and current regulations on educational research. Student participation was voluntary, the objectives of the study were explained in advance, and informed consent was obtained. The confidentiality of the information and the impossibility of identifying individual participants were always guaranteed.

3. Results

The experts reviewed and evaluated the initial tool in terms of clarity, relevance, and appropriateness to ensure the validity of the content before its use. The initial tool included 10 domain indicators as performance indicators, following recognised standards and clinical guidelines. Following their evaluation, the experts proposed a final tool that eliminated two items related to device selection, in line with the lack of implementation of these items in the proposed scenario, and improved the wording of two of the eight remaining items. Thus, the final rubric used consists of eight domain indicators and four levels of performance or achievement (Table 1).

The normality of the dependent variables (items) was analysed using the K-S (Kolmogorov–Smirnov) test (p=0.001) and the homogeneity of variances using Levene’s test (p=0.001).

In the 2021-2022 academic year, 20 teams were completed; in 2022-2023, 17 teams were completed; and in 2023-2024, 25 teams were completed. The total number of students participating was 186, organized into 62 groups. One record was removed from a team that did not complete the scoring of all items (either in self-assessment or peer assessment) for the 2022/2023 academic year. Thus, the sample consisted of 183 participants in 61 groups.

The measures of central tendency and dispersion obtained by the groups in the rubric are shown in Table 2, according to the mode of assessment (self-assessment and peer assement) and the academic year of the participants.

The scores obtained through the rubric by the assessment and peer assessment methods show no significant differences either in the assessments of the 61 groups in total (p = 0.34) or by academic year: (2021–2022: p = 0.25; 2022–2023: p = 0.33; 2023–2024: p = 0.45). A comparison of the scores obtained in each assessment method in each academic year shows that the 2021–2022 academic year has the lowest scores in the rubrics for both self-assessment and peer assessment.

In relation to the score obtained in the rubric using each assessment method in each academic year, significant differences are detected in the values obtained through self-assessment (p<0.001) and peer assessment (p<0.001), indicating variability in performance according to the academic cohort, with lower scores in the 2021-2022 academic year.

The frequency distribution of the scores (domain measures: 1-2-3 and 4) given by students through the two assessment methods (self-assessment and peer assessment) for each of the items in the rubric is shown in Table 3.

There is a tendency to give items higher scores (levels 3 and 4), especially in items 1 (collar selection), 4 (position of medical equipment and instructions) and 5 (neck measurement). In the distribution of frequencies attributed to each item in peer assessment, a greater dispersion of scores was observed, with values at lower levels in items 2, 3, 6, 7 and 8.

3.1. Inter-Rater Reliability

Table 4 shows the Kappa index values obtained for each item in the rubric, calculated from the comparison between the score assigned by the performer themselves and that awarded by their peers.

Substantial agreement was observed in three of the eight items evaluated (items 1, 4, and 6), indicating a high degree of concordance among evaluators. The remaining five items (items 2, 3, 5, 7, and 8) showed moderate agreement, reflecting adequate concordance. No item had an agreement lower than 0.40 or higher than 0.80, which shows consistent and acceptable inter-rater reliability for the rubric used.

3.2. Internal Consistency

The total Cronbach’s alpha obtained for the rubric is 0.753, indicating good internal consistency. Table 5 shows the Cronbach’s alpha value for each item, as well as the associated degree of consistency, allowing for analysis of the individual contribution of each element of the instrument.

Of the eight items in the rubric, two showed moderate consistency (items 3 and 7), two achieved acceptable consistency (items 5 and 6), three showed very good consistency (items 1, 2, and 8), and one item showed excellent consistency (item 4). These results indicate that most of the items in the rubric contribute strongly to the overall construct, reinforcing the internal reliability of the instrument.

3.3. Factor Analysis

Given that the rubric assesses the same competency through the same items, the CFA was performed jointly on the set of assessments, considering each record as an independent observation of the type of assessment performed. Table 6 shows the matrix of components of the domain indicators that make up the rubric.

Exploratory factor analysis identified a two-factor structure with eigenvalues greater than 1, which together explained 55.7% of the total variance.

Factor 1 explained 37.7% of the variance and grouped items related to the positioning and immobilisation of the victim (items 3, 4, 7, and 8), suggesting a procedural dimension.

Factor 2 explained 55.7% of the variance and grouped items related to the selection, handling, measurement, and placement of the collar (items 1, 2, 5, and 6), forming a distinct dimension with a greater component of clinical judgement and decision-making.

The resulting structure showed a balanced distribution of items between both factors, with four items in each dimension, reinforcing the internal consistency and applicability of the rubric in the evaluation of the procedure.

The rotated component matrix (Varimax) showed factor loadings between 0.42 and 0.85 in the first factor and between 0.48 and 0.79 in the second. Item 8 showed slight cross-saturation, with a lower secondary loading in the second factor, so it was kept in the procedural dimension in view of the conceptual and clinical consistency of the instrument.

4. Discussion

The aim of this study was to analyse the psychometric performance of a rubric designed to assess the placement of cervical collars in nursing students. The results obtained allow us to interpret the variability observed between academic cohorts, as well as to assess the inter-rater reliability, internal consistency and factorial structure of the instrument in a clinical simulation context.

One of the most relevant findings was the greater intragroup variability observed in the 2022–2023 academic year, both in self-assessment and peer assessment. Given that the assignment of students to groups, the timing of the activity, and the teaching conditions were homogeneous in the three courses, this variability does not appear to be attributable to organisational factors. This pattern suggests that variables related to the overall performance of the group or the previous level of competence acquisition may be involved, highlighting the ability of rubrics to detect differences between training cohorts, as described in previous studies on competence assessment in higher education [16].

In relation to the distribution of scores per item, a general trend towards high performance levels was observed in those items associated with more objectifiable and easily observable actions. Conversely, items involving greater technical precision, coordination, or postural control showed a greater dispersion of scores. This pattern is consistent with the greater technical and cognitive complexity of certain clinical manoeuvres and has been previously described in the assessment of competencies using rubrics in healthcare contexts [7,14,17].

From the point of view of inter-rater reliability, the Kappa index values indicated moderate to substantial levels of agreement for most items. The lowest values were concentrated in those items with more interpretative descriptors, especially those related to the neutral position of the neck and bimanual immobilisation. The absence of objective or quantifiable criteria and the lack of discrimination between certain levels of performance may favour heterogeneous interpretations among assessors, a phenomenon widely described in the literature on clinical assessment with rubrics [11,14,16,17]. The inter-rater variability observed in this study is consistent with that described in other training contexts, where discrepancies in the interpretation of procedural items by different assessors are common, especially when they involve interpretative components. This phenomenon is characteristic of standardised clinical procedures, such as those included in teaching guides on the placement of cervical collars, including the institutional guide of the University of Murcia [18].

However, this degree of variability is common in instruments used for training purposes, where assessment is conceived as a dynamic process that includes feedback and evaluative dialogue. In this sense, moderate levels of inter-rater reliability do not necessarily compromise the pedagogical usefulness of the instrument when structured feedback processes are integrated [19,20].

The internal consistency analysis showed an adequate overall Cronbach’s alpha value. In the field of health sciences education, alpha values around 0.70–0.80 are considered acceptable when assessing complex clinical skills that integrate technical, cognitive, and procedural components [21,22]. The items with the highest interpretative load, specifically items 2, 3, 5, and 6, showed lower reliability values compared to the more technical and observable items. This pattern has been described in recent studies on the validation of clinical instruments, where elements that require greater evaluative judgement tend to show lower inter-rater agreement [7,13,14,17].

From a structural perspective, the exploratory factor analysis performed jointly provided additional evidence of the internal validity of the rubric by identifying a two-dimensional structure consistent with its theoretical framework. A first factor mainly grouped items related to the technical-procedural execution of cervical collar placement, while a second factor integrated items with a greater interpretative load, associated with the measurement and correct adjustment of the device. This differentiation reflects the coexistence of technical components and clinical judgement in performance assessment, as described in recent clinical assessment instruments [7,22].

Likewise, the presence of some items with moderate loadings on both factors suggests a mixed nature typical of complex clinical procedures, in which technical execution and clinical assessment are closely interrelated. Overall, these results reinforce the competency-based approach of the rubric and support its usefulness in training contexts, in line with studies that indicate that reliability does not need to be maximum to fulfil an effective educational function when well-designed instruments are used and accompanied by structured feedback [23,24].

Limitations and Future Directions

One of the main limitations of the study is that the sample consisted exclusively of undergraduate nursing students, which may restrict the generalisation of the results to other groups of health science students and professionals. Given that the placement of a cervical collar is a technique shared by different healthcare professionals, it would be advisable to extend the validation of the instrument to other degrees with responsibility for this procedure, as well as to different educational and training levels.

Furthermore, future studies could explore the application of the rubric in other teaching contexts and simulation scenarios, as well as analyse the impact of possible adjustments to the item descriptors on their psychometric performance, in order to optimise the reliability and validity of the instrument.

5. Conclusions

The proposed rubric has demonstrated adequate psychometric properties in terms of item analysis, showing an acceptable level of agreement and overall internal consistency, indicating that the rubric has a solid and homogeneous structure. The results of the study show that the rubric can be considered a reliable and consistent tool for assessing the technique of placing cervical collars in nursing students.

The agreement between assessors was adequate for most items, which supports its use in contexts involving self-assessment and peer assessment.

Furthermore, the internal structure identified in two factors confirms that the tool integrates both procedural skills and elements of clinical judgement, which are essential components in the development of advanced clinical competencies.

In the field of health sciences education, this rubric provides a structured framework that can improve the quality of feedback and promote more objective and standardised assessment. Its application in clinical simulation scenarios contributes to enhancing patient safety and promoting meaningful learning, making it a valuable resource for critical care and emergency training programmes.