An Automated, Web‐Based Data Analysis Tool for Assessing Overtriage and Undertriage of Trauma Team Activations in the Emergency Department

1. Introduction

Healthcare providers around the globe face the daunting task of improving patient outcomes while keeping costs manageable [1]. Advanced analytical technology, driven by the digital revolution, holds promise for enhancing patient care and data-driven decision-making, which promotes a culture of continuous improvement within healthcare organizations [2,3,4,5]. Industries such as manufacturing, finance, and logistics have shown that the successful implementation of workflow automation is critical in optimizing operations [6]. Automation techniques can be tailored to address the unique needs and complexities of the healthcare industry, especially when existing methods rely on manual data analysis [1,4,6,7,8]. This customization can significantly enhance efficiency and accuracy in handling healthcare data.

Trauma triage systems play a crucial role in trauma care in the Emergency Department (ED) [9,10,11]. Research has consistently shown that appropriate trauma triage improves patient care and outcomes, enhances ED efficiency, and reduces operating costs [12,13]. Critical resources like blood products, hospital staff, and inpatient beds are limited, making it a complex challenge to balance the need for rapid and effective trauma care with the scarcity of available resources [14,15]. Trauma team activations (TTA) at the anticipated level of need are essential to ensuring that trauma patients with severe injuries receive timely and adequate care, even in the face of limited resources [11,16,17,18]. In the United States, trauma centers have established TTA protocols to respond promptly to patients with varying degrees of injuries [19]. These protocols rely on criteria of physiologic findings, mechanism of injury, and anatomic damage to approximate injury severity, morbidity, and mortality risk [17,18,20].

Though structured, TTA criteria do not consistently anticipate every patient’s needs, often leading to inappropriate triage decisions [10,11,21,22,23]. Overtriage occurs when patients are triaged to a higher level of care than their injury severity warrants, leading to increased healthcare costs [22,24,25]. In contrast, undertriage occurs when patients with significant injury burdens do not receive the full or appropriate level of TTA, which can lead to adverse outcomes by preventing or delaying the delivery of definitive care [14,19,24,26]. The American College of Surgeons Committee on Trauma (ACS COT) aims for triage error rates of less than 5% for undertriage and between 25% and 50% for overtriage [15,23,26]. Accurate reporting of overtriage and undertriage rates in trauma activations is imperative for optimizing trauma resource utilization and improving trauma care [23,24,26].

The Standardized Triage Assessment Tool (STAT) is used to evaluate the appropriateness of trauma triage and TTA [14]. It evaluates whether a trauma patient was misclassified in terms of injury severity using the Cribari Matrix Method (CMM) and determines if the patient required major trauma intervention based on Need for Trauma Intervention (NFTI) criteria [13,14,20,27]. CMM calculates the overtriage and undertriage using the Injury Severity Score (ISS) and the level of TTA, while the NFTI criteria assess resource utilization and level of TTA, as outlined in Figure 1 [12,13,25,28].

The STAT is a combination of the CMM and the NFTI tool [14,28]. CMM calculates the overtriage and undertriage using the ISS and the level of TTA—in undertriage, a patient is severely injured with an ISS greater than 15 but with a TTA other than full; in overtriage, a patient with a minor to moderate ISS (less than or equal to 15) receives operating room within 90 minutes; (3) discharge from the ED to interventional radiology; (4) discharge from the highest level of TTA [12,28,29]. The NFTI criteria assess resource utilization and trauma-related interventions; a positive NFTI (NFTI+) requires at least one of the following: (1) transfusion of packed red blood cells within four hours of ED arrival; (2) discharge from the ED to the the ED to the intensive care unit (ICU) with ICU length of stay of three or more days; (5) use of non-procedural mechanical ventilation within 72 hours; and, (6) mortality within 60 hours of arrival [20,30,31,32].

Using insights from successful workflow automation in other industries, the objective of this project was to develop an automated, web-based tool for calculating overtriage and undertriage rates of TTA using the STAT [6,33,34]. To our knowledge, this is the first tool specifically designed for this purpose: automating the calculation of triage accuracy and optimizing trauma care decision-making.

2. Materials and Methods

2.1. Data Collection and Analysis

We collected de-identified data sets of 1,578 TTAs from the trauma registry of a single ACS COT-verified Level 1 Trauma Center from November 1, 2022, to October 31, 2023, inclusive. At this institution, Level 1 or ‘Red’ traumas require a full trauma team activation, which includes immediate mobilization of both the ED and Trauma Surgery teams, comprising the Trauma Surgery attending and resident teams, as well as the ED attending and resident teams. Level 2 or ‘Yellow’ traumas require activation of the ED team and the resident Trauma Surgery team, but do not necessitate immediate involvement of the Trauma Surgery attending. Level 3 or ‘Green’ traumas are managed solely by the ED team, who may consult the Trauma Surgery team if they deem it necessary. Trauma overtriage and undertriage for each TTA were determined according to the criteria of the CMM and NFTI used in the STAT, as shown in Figure 1. All statistical analyses were done using Python version 2022.3.1. The web tool was designed using the website development programming languages Hypertext Markup Language (HTML) and Cascading Style Sheets (CSS).

2.2. Application Prototyping and Development

Figure 2 illustrates the prototyping methodology, which involves an iterative process of planning, designing, building, testing, and debugging the application [35]. During the development stage, a graphical user interface (GUI) was constructed, allowing end-users to interact with the application and perform relevant calculations. The application was designed to receive TTA data through a data upload feature that supports the Microsoft Excel spreadsheet format. The application was encrypted and secured behind a login requiring authorized credentials for access. Screenshots of the final application are shown in Figure 3.

2.3. Application Performance Testing

Preliminary testing of the application was performed by testing the tool on a mock data set of 15 TTAs and comparing the results with the manual data analysis process conducted by the programmer (author AA). Intermediate testing used a limited dataset of 104 de-identified TTAs spanning a one-month subset of the TTAs collected from the hospital. Final testing was conducted on the completed TTA dataset collected from the hospital, which included 1,578 de-identified TTAs over one year. For both datasets, the Accuracy and Process Efficiency of the automated and manual analysis methods were evaluated to assess the tool’s performance relative to the manual process. Accuracy was determined by comparing the triage classifications (appropriate, overtriage, or undertriage) from the manual calculations with those from the automated tool. It was anticipated that, once provided with complete data in Excel format and correctly programmed to apply the STAT method, the tool would achieve 100% accuracy. Process Efficiency Improvement was calculated as:

$$\begin{gathered} ProcessEfficiencyImprovement \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\left(1-{\displaystyle \frac{AutomatedProcessTime}{ManualProcessTime}}\right)\ast 100\% \end{gathered}$$

(1)

2.4. Application Evaluation

The application and its functionality were presented to a focus group of trauma providers from a level 1 trauma center. The providers were first trained on performing manual calculations using the STAT method, followed by instructions on how to use our tool with a sample dataset. After, the participants were asked to complete an evaluation survey using a 5-point Likert scale, evaluating the tool’s performance, and their experience with and perception of the tool. Mean ratings for each question, on a scale of 1 to 5, were calculated and reported for all participants.

3. Results

3.1. Performance Testing

As expected, the automated data analysis process had 100% accuracy in determining overtriage and undertriage for the mock and actual datasets. The time to complete the analysis of the full dataset was 26 minutes and 19 seconds (1579 seconds) using the manual process, whereas it took 23 seconds using our automated tool (Table 1). The mean process efficiency of the automated data analysis process was 98.0%. This represents a significant time-saving advantage.

3.2. Survey Findings

Ten participants completed the online survey to evaluate the tool: two Trauma Program Directors, two Performance Improvement/Clinical Data Analysts, a Trauma Medical Director, an Associate Trauma Medical Director, a Trauma Surgeon, a Trauma Liaison, a Trauma Research Associate, and a consultant. The majority of participants had a significant amount of experience in caring for trauma patients and in trauma center leadership roles: six had over six years of experience in trauma center leadership, three had between three and six years of experience, and one had one to two years of experience.

Table 2 shows the results of the evaluation survey. All participants said the application was easy to use, giving the program an average rating of 4.6. They all thought the application significantly reduced data analysis time, with an average rating of 4.9. They also all agreed that the application assisted in identifying mistreated trauma patients during the performance improvement process (mean rating of 4.7) and noted that the tool effectively automated both overtriage and undertriage analyses, each receiving a mean rating of 4.8. The reliability of the tool was deemed to be high, with an average rating of 4.6. All participants indicated they would consider incorporating this tool into their institution/workflow.

3.3. Open-Ended Participant Feedback

Participants expressed enthusiasm about the tool’s ability to enhance workflow efficiency, particularly by reducing the time required for data analysis. One participant highlighted its potential value for trauma research. Several others noted that the tool holds great promise for future advancements in data analysis technology for trauma center leadership. A suggested improvement for future development was to add a feature for identifying outliers in the overtriage and undertriage of TTA.

4. Discussion

We outlined the development and evaluation of an automated, web-based data analysis tool that calculates the mistriage rates of a dataset of TTAs at a trauma center. Findings indicated that our tool represents an accurate and highly time-efficient method for analyzing mistriage. Feedback from experienced trauma providers was overwhelmingly positive, highlighting the tool's perceived efficiency and reliability in streamlining processes. Overall, participants felt the application was easy to use, accurate in calculating overtriage and undertriage rates, and beneficial for performance improvement activities, with all respondents agreeing or strongly agreeing with the evaluated metrics.

From a quality improvement perspective, having a reliable tool that can accurately report on rates and trends of overtriage and undertriage of trauma activations while reducing data analysis processing time allows trauma centers to focus on critical aspects of utilization management [16,23,34]. This includes developing actionable plans to enhance the quality of patient care delivery and improving overall trauma center operations.

Limitations

One notable limitation of the project was that there was limited end-user interaction with the tool, as the institution had high-level security and permission requirements for any application to be installed on local computers. Additionally, the evaluation of the data analysis tool was conducted with a small sample size of participants, which may affect the generalizability of the results. To enhance the reliability of our evaluation, it is recommended to conduct a larger-scale participant study to minimize potential biases and improve the robustness of the findings.

Future Considerations

Further development and refinement of our tool will be necessary before broader institutional and external deployment. Future versions of the data analysis tool will include features for summarizing trends in overtriage and undertriage and identifying outliers, as suggested. Additional functionalities will incorporate an export feature for downloading detailed overtriage and undertriage determinations for each analyzed case and tools for handling incomplete data. Ultimately, we plan to provide access to this tool to all trauma providers, starting with our institution and eventually extending to others. This will involve the engagement of the information technology infrastructure and organizational leadership teams at our trauma center to ensure adequate security permissions and approvals before deployment. Successful deployment will enable the tool to be utilized for quality improvement initiatives and research projects, enhancing its impact on trauma care and outcomes.