Standardizing and Classifying Anterior Cruciate Ligament Injuries: International Multicenter Study Using a Mobile Application

1. Introduction

Knee ligament injuries can occur due to a sudden axial load on the knee, combined with valgus stress and rotational force around the tibia [1,2]. An injury to the ligament complex results in a loss of stability; therefore, an early clinical diagnosis of instability is imperative. Clinical evaluation includes tests such as the Lachman test, anterior drawer test, and Pivot-Shift (PS) test, considered the "gold standard" for assessing knee rotational function after an injury. There are also scales based on questionnaires from the International Knee Documentation Committee (IKDC) and the Tegner Activity Scale (TAS). However, these are subjective measurements [2]. Systems like the KT1000 and KT2000 arthrometers (MED metric Corp., San Diego, CA, USA) quantitatively document sagittal laxity, while rotational laxity can be measured using ambulatory devices, instrumented boots, magnetic resonance imaging, electromagnetic sensors, robotic technology, or navigation systems. Despite significant technological advancements, these tests may not be suitable in a clinical setting, considering issues related to feasibility, affordability, and comfort. For this reason, the PS test is the only clinical test that correlates with knee function after ACL rupture, assigning a grade to the pivot shift subjectively, leading to efforts to quantify bone movements and correlate them with the degree of injury. However, the dynamic and unrestricted nature of the maneuver introduces significant kinematic variability [3].

Considering the prevalence and importance of accurate and numerical assessment of knee rotational instability, a smartphone application called "Pivot-Shift Meter (PSM)" has been developed, using the gyroscopes and accelerometers integrated into mobile phones. In initial tests of the application, the results were compared with those of the KT-1000 arthrometer, demonstrating that 95% of cases with a lesion were classifiable, and the instrument is valid for rotational measurement of the ACL, with an inter-observer ICC of 70% in velocity amplitude and 95% in time. This proves that a quantitative analysis of the maneuver can be conducted through the application [4,5]. Compared to a widely accepted arthrometer, the "Pivot-Shift Meter" can differentiate between an injury and ligament hyper-laxity, offering a grading of the injury. However, results of initial studies showed a low correlation between intra-observer analysis regarding the amplitude of the pivot shift effect graphed in the application due to the variables presented by each evaluator during the maneuver. This led to the initiation of the current multicenter and international case study in which we aim to analyze, classify, and follow more than 400 tests performed by specialist physicians from five different countries, including Bolivia, Chile, Colombia, Ecuador, and Mexico, using the mobile application PSM on patients with ACL injuries. The purpose is to optimize the software's adaptability for each user and the motivation behind conducting this study lies in the importance of diagnosing and postoperative monitoring of ACL injuries due to the potential repercussions such as early joint degeneration and knee instability. Therefore, digital arthrometry appears to be the path toward developing precise and numerical evaluation methods for rotational instability, with all the advantages that early diagnosis and classification can provide before any surgical treatment.

2. Materials and Methods

A prospective cohort design was employed to conduct this multicenter study. The primary objective is to evaluate the effectiveness of the PSM application in the detection and classification of ACL injuries. A sample of patients with ACL injuries will be recruited, selected by trained Latin American medical specialists using the trial version of the PSM application. Included patients are aged 18 and above, regardless of gender, and meet the selection criteria that will be mentioned below.

The trial version of the PSM application will serve as the measurement instrument. Using inertial sensors to detect ligament injuries in the knee.

2.1. Procedure

2.1.1. Part 1: Specialized orthopedic surgeons from different countries in Latin America, treating a significant number of patients with ACL injuries and experienced in arthroscopic reconstruction, were contacted. A total of 8 surgeons from Latin American, including Bolivia, Chile, Colombia, Ecuador, and Mexico, agreed to participate in the study and use the PSM application on patients with ligament injuries. Additionally, they were given the option to include clinical observations, pre- and post-operative notes, results of diagnostic studies, arthroscopy images, or other relevant patient-specific information2.1.2. Part 2: Each surgeon was provided with a guided tutorial to familiarize themselves with the application, and an adjustable elastic band to secure their mobile phones, which must have an accelerometer and gyroscope, with either Android or iOS operating systems. Evaluators used their personal mobile devices to perform tests with the application

2.2. Sample

Patients diagnosed with ACL knee injuries, assessed by Latin American orthopedic specialists trained in using the trial version of the PSM application.

2.2.1. Inclusion Criteria

• Individuals aged 18 years or older.
• Individuals with clinical data indicating at least one injured ACL bundle, diagnosed by a specialist in traumatology and orthopedics.
• Individuals scheduled for surgical ACL reconstruction.
• Individuals with a history of arthroscopic surgical intervention for ACL reconstruction in the affected knee.

2.2.2. Exclusion Criteria

• Individuals under 18 years of age.
• Individuals with neurovascular injuries or pathologies that may affect study results.
• Individuals with fractures, neurological, or muscular problems that impede the performance of the standardized pivot maneuver.

3. Results

A total of 47 healthcare professionals responded to the survey for evaluator recruitment. For their selection, specialists in Orthopedics and Traumatology were sought, resulting in the recruitment of 8 evaluators from 5 countries, including Bolivia, Chile, Colombia, Ecuador, and Mexico, with an average of 100-120 ACL reconstruction surgeries performed per year.

The evaluators accepted the terms and conditions of the "PSM" application and underwent training for its proper use, including a standardized pivot technique, and precise instructions were provided for validating the obtained results. They were also given a guided tutorial on registering patients in the study. Patient data was recorded while preserving their privacy, and the results of the maneuvers performed were automatically included in an online database. Each patient's record included information such as gender, age, height, weight, and the examiner had the facility to add observations, clinical classification according to IKDC criteria, results of digital arthroscopy (KT-1000), results of diagnostic studies, arthroscopy images, and notes with relevant clinical case information. According to the examiners, 60% of ACL injuries were classified as Grade I, 33% as Grade II, and 7% as Grade III, based on the IKDC criteria. Subsequently, the information from the database was downloaded and organized into a comma-separated values (.csv) file using a Python-generated code for further analysis.

In total, 437 tests were captured using the standardized pivot method, with 233 tests on the right knee and 204 on the left knee. Among these, 136 tests were conducted in a medical office due to clinical evidence of ACL injury during physical evaluation, 180 tests were performed during preoperative evaluation in patients diagnosed with ACL injury scheduled for arthroscopic reconstruction, 62 tests were conducted during immediate postoperative evaluation in postoperative patients, and 20 tests were performed in patients with a history of arthroscopic reconstruction to track their progress. Additionally, 30 registered tests were conducted on patients diagnosed with ACL injury without specifying the clinical evolution period. Results in all three axes of movement (X, Y, and Z) were obtained using the gyroscopes integrated into the evaluator's smartphone (Figure 1).

The decision was made to analyze the X-axis of the gyroscope since it reflected the flexion and extension of the knee during the pivot-shift maneuver. Features were extracted from the raw X-axis data vector of all evaluators and subsequently processed using a program designed in MATLAB.

Out of the 437 registered tests, 38 were discarded as they did not meet the standards for analysis. These included tests with technical failures during execution, incorrect placement of the mobile device, or incorrect execution of the standardized pivot maneuver, leaving a total of 399 standardized tests.

Once the signal was acquired, the X-axis gyroscope data was divided into three sections. Mathematical, morphological, and statistical characteristics of four types of data representation were then analyzed: the original values, normalized values, Fourier transform of the original values, and Fourier transform of the normalized values (Figure 2).

After examining and evaluating the different features and data representations, it was concluded that segment 3 was the most relevant and representative for capturing the amplitude or magnitude of the knee movement response in the X-axis during the pivot-shift maneuver. Consequently, this segment was used to categorize the tests into different classes. The following mathematical characteristics of segment 3 were taken into account for this categorization: standard deviation (S3n-STD), range (S3n-R), range plus the average of the segment (S3n-Rx), the ordinate of the linear regression (S3n-O), and the slope of the linear regression (S3TF-P).

Following the exclusion of tests with failures and obtaining the characteristics of standardized tests, the analysis for their reclassification was performed. The results of the multicenter study, consisting of 399 tests, were distributed into 8 classes.

Results of 224 control tests obtained using the first version of the PSM application, before the implementation of standardization in the pivot technique, were also included. These tests were distributed into their respective classes (Table 1)

Statistical analysis results revealed that the calculated value for the χ^2 test was 62.17213413. When compared to the critical value of 14.067, with a significance level of 0.05, a p-value of .00001 was obtained, indicating a significant association between control tests and standardized tests. These findings support the effectiveness of standardizing the pivot technique, as it has had a notable impact on the distribution of the classes.

With standardized tests and control tests reclassified into 8 classes, the obtained characteristics described above were used to create a neural network for class groups. The network was trained with the results recorded in the application, and a general performance evaluation of the model in the classification task was performed (Figure 3; Table 2).

The performance analysis results of the created model revealed highly favorable performance in terms of its ability to classify maneuvers into the desired class. Accuracy, precision, and sensitivity indicators exceeded 90%, demonstrating the model's high capability to accurately classify the maneuver class. Specifically, the F1 score of the model reached a result of 97.27%, confirming its overall accuracy and sensitivity in classifying the classes. These results support the effectiveness and reliability of the model in the objective identification and classification of standardized maneuvers.

Next, additional analysis was conducted to evaluate the ability of each class to assess the degree of ACL injury in four levels: grades 0, 1, 2, and 3.

For this, a confusion matrix specific to the classes was created to evaluate the classification function of the injury according to its corresponding grade. The following mathematical, statistical, and morphological characteristics of significant signals for grade assignment by class were analyzed: maximum value of segments S1 and S3 (S1-Max, S3-Max), ranges of segments S2 and S3 (S2-R, S3-R), standard deviation of S3 (S3-STD), Shannon entropy of S3 (S3-We), and the slope of the linear regression of segment 3 (S3TF-P). During the analysis, it was observed that classes 1 and 8 represented maneuvers furthest from the standardized method, so their classification results by grade were discarded. The following results were reported for classes 2 to 7 (Figure 4; Table 3).

The class-specific analysis yielded highly favorable results in terms of neural network performance indicators. In all analyzed classes, accuracy, precision, and sensitivity exceeded 90%, demonstrating the model's ability to accurately classify ACL injuries into different grades, according to the PSM maneuver class. The F1 score was calculated, and notable results were obtained. In Class 2, an F1 score of 98% was achieved, indicating the model's high ability to classify injury grade, particularly for this specific category of PSM maneuver. In Class 3, an F1 score of 97% was achieved, and in Classes 4, 5, 6, and 7, F1 scores of 96%, 94%, 95%, and 96%, respectively, were recorded.

These results confirm the software's ability to classify with high accuracy and reliability the grade of ACL injury by analyzing the amplitude or magnitude of the knee movement response during the pivot-shift maneuver, as represented in segment 3 of the X-axis of the gyroscope data.

The evaluation of the performance of the neural network model in classifying standardized maneuvers and in determining the degree of ACL injury according to grade clearly demonstrates its high efficiency, sensitivity, and reliability. The analysis of variance was carried out using the Shapiro-Wilk test, which indicated that the sample was not distributed normally. This was confirmed by the Bartlett test, which rejected the hypothesis of homoscedasticity. Due to these findings, non-parametric tests were chosen for the analysis of the data.

To determine the statistical significance of the differences between the results obtained for standardized tests and control tests, the Mann-Whitney U test was applied, which confirmed the presence of statistically significant differences in the results obtained between the groups.

To evaluate the accuracy of the device in diagnosing ACL injuries, the ROC curve was analyzed using the results of standardized tests and control tests, which enabled the calculation of the AUC. The ROC curve reflects the ability of the device to differentiate between ACL injuries and non-injuries, with an AUC of 0.80 indicating a high accuracy rate.

4. Discussion

While the pivot-shift test is considered a cornerstone in assessing knee laxity, its clinical application comes with several significant limitations. Performing the test and evaluating the results are subjective tasks prone to errors [6,7]. This subjectivity is a relevant factor contributing to uncertainty among observers and intra-observers, limiting the reliability of the obtained data. The standardized maneuver proposed in this study statistically demonstrated more consistent and reliable results in classifying tests compared to the control group. This, in turn, can enhance the accuracy and reliability of diagnosing ACL injuries.

The accelerometers and gyroscopes of the cell phone used during the study to measure the pivot maneuver have shown promising results in evaluating and classifying the pivot mechanism non-invasively [8,9,10]. The PSM application allows the capture of 500 samples in 5 or 6 seconds of the angular velocity in three axes (X, Y, Z). The results are immediately stored in a cloud database for analysis within a trained neural network, providing a classification of the ACL injury grade based on the performed maneuver class. This application's ability to offer objective data on the maneuver and the recorded tibial acceleration is a significant advantage compared to the subjective assessment provided by an evaluator [11,12,13,14,15,16].

The classification of ACL injuries into Grade I, II, and III is particularly relevant due to the discrepancies reported in the literature regarding surgical outcomes between single-bundle and double-bundle reconstruction. While many studies claim no significant difference between the two approaches, our findings suggest that proper classification into Grade I, II, and III injuries may account for these inconsistencies [2,3,4,5,6,7,8,16,17]. The accurate classification achieved through the PSM application may help identify cases where surgical techniques should be tailored based on the severity of the injury. The data generated by the PSM application have the advantage of being a continuous variable, contrasting with studies that generally report categorical data corresponding to the IKDC grade. This clinical categorization is associated with the risk of not accounting for variation within each group of evaluators during the physical examination and incorrect characterization of results on either side of cutoff points. This highlights the subjectivity in the described scoring systems for quantifying ligament injuries, and despite several scoring systems being proposed, none have been widely accepted due to the challenge of assigning numerical values to non-quantifiable factors [17].

Regarding the limitations of other methods for measuring tibial axis rotation, such as Rottometer, Vermont knee laxity, and Lars rotatory laximeter, as well as electromagnetic navigation through skin sensors, all of which measure tibial axis rotation non-invasively, they are either bulky for orthopedic clinic use or lack precision. Intraoperative optical navigation systems using bone markers are precise but invasive and unsuitable for clinical settings [18,19,20].

In summary, this study demonstrates the potential of using a mobile application to detect the pivot effect in patients with ACL injuries, providing physicians and patients with a more objective and continuous measurement compared to current rating systems.

5. Conclusions

The development of tools and the use of technology in the clinical field to optimize diagnosis, prognosis, and medical treatment have evolved the concept of healthcare. Inertial sensors like accelerometers and gyroscopes in smartphones are a novel, convenient, accessible, and comparatively more affordable option that can be used to diagnose and quantitatively assess ACL-injured knees.

Standardizing the pivot technique has shown a significant influence on the distribution of maneuver classes, which is crucial for classifying ACL injury grades. The neural network developed for the PSM application, when using the standardized maneuver, achieved a high degree of confidence in its predictive values, allowing for precise detection and classification of various ACL injury grades. Thus, the hypothesis posed in this study is confirmed.

The PSM software trained with this new neural network has the potential to be a supportive tool when considering diagnostic and follow-up options for ACL injury patients. It enables the treating physician to more quickly select the appropriate technique for surgical correction and assess the degree of progress after applying the correct treatment. All of this contributes to greater patient well-being by providing a more accurate diagnosis and timely treatment selection.

It is emphasized that standardizing the pivot maneuver is crucial to ensure the reliability of the results obtained through this tool. The data from a physician should be interpretable and classifiable, so it is necessary for specialist doctors using this application to perform the correct movement to generate reliable results.

In general, the application of technology like inertial sensors in smartphones and neural networks can be a valuable tool to enhance the accuracy and objectivity of clinical assessment of ACL injuries and other joint conditions. It is important to continue researching and refining these tools to ensure their accuracy and utility in clinical practice.