Evaluation of Feature Selection Methods and Machine Learning Models for Identifying Collided Positions of Containers Equipped with an Accelerometer

1. Introduction

The trade and logistics industry, heavily reliant on container usage, faces significant challenges due to frequent container collisions and these collisions are one of the primary sources of container damage [1]. A 70% of rail and road transport vehicles have defects in container securing and packaging [2], increasing the risk of damage to containers or their contents in collision incidents. Studies have shown that improper container securing, and poor cargo packaging are the main causes of commercial goods damage [3].

Although container ships handle over 80% of international cargo transport, with more than 90% of goods transported through containers [4], traditional methods for inspecting container damage and collided positions often require trained personnel to perform visual inspections based on container management standards [12,13,14]. However, this method is not only time-consuming and labor-intensive but also requires training and managing the inspectors, greatly impacting work efficiency, and increasing inspection costs. To alleviate these challenges, some studies have proposed using image recognition technology to improve work efficiency and reduce management costs [15,16,17,18]. However, image recognition could only inspect damage on the surface of containers and lacked attention to the state of the contents within the containers.

Although physical damage is the main cause of damage to container cargo, accounting for 25% [19], certain collisions may leave the container’s surface unscathed but cause internal impact or cargo displacement, resulting in damage. This issue is particularly important for fragile items, such as glass products, electronic components, precision instruments, and similar delicate goods. This gives rise to the following issues: (1) if the surface of the container remained intact after a collision, any damage to the goods inside could have been easily overlooked (2) it is difficult to determine whether the damage to the goods was caused by a collision. Therefore, accurately identifying the collided positions in container accidents has become imperative. Identifying the collided positions not only could enhance the effectiveness of goods safety supervision but also could provide a reliable basis for the determination of accident liability, holding significant theoretical and practical significance.

Although previous studies have utilized sensors and machine learning to implement various container-related applications such as locating containers during transportation to ensure safety throughout the transit process [20,21,22], measuring the fill status of liquid cargo within containers [23], automatic loading and transporting of containers [24] and detecting physical collisions between ship vertical sections and containers [25], no study has been made to identify the collided positions of containers. We propose a machine learning-based approach using the data from the accelerometer attached to the door of a container.

The dataset collected from an accelerometer often contains high-dimensional data filled with irrelevant features and noise. Existing studies have focused solely on model performance, overlooking the computational expense incurred when processing high-dimensional data. Reference [26,27,28] has highlighted the issues of increased computational resource consumption and reduced model performance due to high-dimensional data. In this study, we propose a machine learning-based framework to identify the collided positions of containers using the accelerometer sensor data with a reduced dimensionality of the data using a feature selection technique.

The remainder of this paper is organized as follows: Section 2 elaborates on the framework of identifying collided positions including the process of data collection and interpretation, the application of feature selection method, and the machine learning-based prediction of collided positions. Section 3 presents and explains the experiment results. Section 4 summarizes the main findings and offers concluding remarks.

2. Machine Learning-Based Identification of Collided Positions

Figure 1 shows the process flow of machine learning-based identification of collided positions. It consists of three processes: the first process to collect data the accelerometer, the second process to find the most helpful feature subset for detecting collided positions, and the final process to use machine learning to identify collided positions.

2.1. Data Acquisition

In this study, we used an IMU (Inertial Measurement Unit) sensor to collect three-axis acceleration data (X, Y, Z) during collision events on the container. The IMU is a sensor mainly used to detect and measure acceleration and rotational motion. As illustrated in Figure 2, the X and Y axes of this accelerometer record horizontal data, while the Z axis records data from the vertical orientation. For the sake of data collection convenience and to simplify the installation of the sensor, we vertically mounted it on the inside of the rear door of a container, subsequently closing the door. This arrangement allowed us to detect collisions of varying degrees in both the vertical and horizontal directions on the container, yielding three-axis acceleration data (X, Y, Z) when the container experienced a collision.

After installing the IMU sensor, we conducted collision tests on a container in various positions, including top, bottom, left, and right side. The sensor recorded raw acceleration data in the XYZ axes at a rate of 100 Hz. As shown in Figure 3, the original data collected for each collision event was observed to have a very short duration, typically about a second.

To avoid retaining excessive irrelevant data during the collision process, we got the most pronounced samples at 30 Hz for each collision from the XYZ-axis acceleration data recorded by the sensor. As depicted in Table 1, the sample data were arranged in rows of the format [x1, y1, z1, x2, y2, z2, ..., x30, y30, z30], with each row containing 90 data points, representing a single collision event. The collision events were then labeled by 0, 1, 2, and 3 corresponding to the top, bottom, left, and right position, respectively. After labeling the column as “Class”, the entire dataset was saved in a CSV file as the original dataset.

We obtained a dataset consisting of 661 container collision events occurring at different positions; a total of 159 collision events were recorded at the top position, 176 at the bottom, 168 at the left, and 158 at the right.

2.2. Feature Selection

As high-dimensional data can bring significant challenges to the classification algorithms [29], Feature Selection (FS) is needed. It was shown that FS could improve model performance, save computing resources, and enhance model interpretability in the field of network security attack detection [30], human activity recognition [31], and medical detection [32]. Finding the optimal subset of features without reducing model performance is not easy. The complete number of subsets is $2^{\mathit{n}}-1$ where n is the total number of features. Since the search time increases exponentially as n increases, we need a more efficient FS method.

At present, the main methods of FS can be divided into filtering method, wrapper method and embedded method [33]. In order to find the most suitable FS method to reduce the data dimension for container collision problem, we tested a total of five FS methods: two filtering methods such as Pearson correlation coefficient (PCC) and mutual information (MI) [34]; two packaging methods such as sequential forward selection (SFS) and sequential backward selection (SBS) [34]; and an embedding method such as extra tree (ET) [35]. These five methods are popular and representative enough, with simple calculation logic and easy implementation.

2.3. Classification Models

In this study, we tested such machine learning (ML) models to identify the collided positions on a container as Decision Tree (DT) [36], K-Nearest Neighbors (KNN) [37], Support Vector Machine (SVM) [38], Random Forest (RF) [39], and Extreme Gradient Boosting (XGBoost) [40]. The purpose is to compare the performances of the features selected using different FS’s under different classification models and find the best combination of FS method and ML model. Considering that the amount of data in our study is small, 10-fold cross-validation is used for training and validation to efficiently use limited data.

2.4. Classification Systems

To design the most effective system for identifying container collided positions, we not only evaluated all possible combinations of feature selection methods and machine learning models, but also compared two systems: a single multi-class classification system that identifies one of the four collided positions (top, bottom, left, or right), and four separate binary classification systems, each dedicated to identifying whether a collision occurred at a specific position (top, bottom, left, or right).

3. Experiment Results

In this study, we used such three indicators to evaluate the classification systems as accuracy, execution time and CPU utilization. Accuracy refers to the ratio of correctly classified samples to the total number of samples. Execution time refers to the total time taken by the model, from training to evaluation through 10-fold cross-validation. CPU utilization refers to the proportion of CPU resources consumed during the model execution. We experimented on the computer equipped with an Intel(R) Core (TM) i5-12400F processor, 32.0 GB of memory, an NVIDIA GeForce RTX 4060 GPU, and 1 TB of SSD storage, running the Windows 10 operating system.

3.1. Results for a Single Multi-Class Classification System

For selecting the optimal subset of features, we set the parameter settings for each FS method as follows. The filtering threshold for PCC and MI is set to 0.2 since, if the threshold is higher than 0.2, no features are not selected. As the classifier model for SFS and SBS, we used the Gaussian Naive Bayes (GaussianNB) [41] model since it requires low computational resources. We configured the parameters for ET using the default initialization settings in the sklearn library.

Figure 4 shows the number of features selected by each FS method. All FS methods effectively reduced the feature dimension compared with the original data. Even the SBS method, which selected the largest number of features, only selected 23 features compared to the 90 features of the original data, while the SFS method selected the least number of features, only 9.

We evaluated the accuracies of the 5 ML models using the features selected by the 5 FS methods, respectively. Table 2 shows the results including using the original data. From the table, we can see that all FS methods show a similar level of accuracy as the original data. The accuracies of some FS’s are even better than that of original data. For example, the accuracy of DT is 0.9304 for the original data, while it is 0.9440 for the SFS method with only 9 features. The accuracy of KNN for the original data is 0.9288, while it is improved to 0.9712 for the ET method with 11 features. The accuracy of SVM for the original data is 0.9485, while it is improved to 0.9713 for the SBS method with 19 features.

Comparing all the accuracies of ML model and FS method combinations, the best-performing combination is XGBoost and SBS. This combination achieved an accuracy of 0.9849. Although the result was the same as that using the original data, the number of features was reduced from 90 to 19, significant reduction of feature dimensionality.

Table 3 and Table 4 show the execution time and CPU utilization of each ML model for different FS methods. We can see that FS has a significant effect on reducing the time and CPU utilization. For example, the time of RF for the original data is up to 2.3920s, while the time of RF using the features selected by ET is reduced to 1.0990s. The CPU utilization of XGBoost for the original data is up to 25.40%, while it is reduced to 14.75% when using SFS. Among all the combinations, the combination with the least time is KNN and SFS, which takes only 0.029 s. In terms of CPU utilization, the combination of KNN and ET is relatively good, consuming only 0.10% of CPU.

Looking at the results, we found that the combinations with the best performance in terms of accuracy, execution time and CPU utilization are different. By observing the values of the three indicators, we determined the following two criteria for evaluating the best model:

(1) The minimum accuracy must be maintained above 0.95.

(2) Choose models with high accuracy, low execution time, and low CPU utilization as much as possible.

According to the criteria, the best combination is KNN and ET. The accuracy is 0.9712, which exceeds the criteria of 0.95. Although the accuracy is not the highest, it has the lowest execution time and CPU utilization of 0.300s and 0.10% respectively.

3.2. Results for Separate Binary Classification Systems

In this experiment, we evaluated the 4 binary classification systems where each selected class such as top, bottom, left, or right is distinguished from all other classes. The selected class is labeled as a positive (1), and all other classes are combined as a negative (0). Figure 5 shows the number of features selected by the 5 FS methods for each classification system. In this study, we call each classification system Class 0, Class 1, Class 2, and Class 3 according to the top, bottom, left, and right position, respectively. Specifically, the minimum number of features for Class 0 is given by MI, which is only 1, while those for Class 1, Class 2 and Class 3 are given by SFS, which are 3, 4 and 2, respectively. In contrast to Figure 4 where the minimum number of feature was 9 for SFS. The binary classification systems can reduce the feature dimensions more significantly, obtaining a smaller subset of features.

Table 5, Table 6 and Table 7 show the accuracy, execution time and CPU utilization for Class 0. Overall, the best combination is KNN and PCC. First, its accuracy 0.9576 is higher than the criteria of 0.95. Secondly, the execution time 0.310s is the least among all the combinations with an accuracy greater than 0.95. Finally, the CPU utilization is 0.20%, the minimum among all combinations.

Table 8, Table 9 and Table 10 show the accuracy, execution time and CPU utilization for Class 1. Overall, KNN and SBS is the best combination with the accuracy 0.9788, the execution time 0.0300s, and the CPU utilization 0.15%. The accuracy is higher than the criteria of 0.95. Although the execution time is greater than that of the KNN and SFS combination, the accuracy is higher. Furthermore, the CPU utilization is the least, only 0.15%.

Table 11, Table 12 and Table 13 show the accuracy, execution time and CPU utilization for Class 2. The best combination is DT and SFS. Its accuracy is 0.9909, which is higher than the criteria of 0.95. In addition, it has the lowest execution time and CPU utilization, only 0.0130s and 0.25% respectively.

Table 14, Table 15 and Table 16 show the accuracy, execution time and CPU utilization for Class 3. The best combination is DT and ET with an accuracy of 0.9803 above the criteria of 0.95. In terms of execution time, although DT and MI, DT and SFS, and DT and SBS take a little less execution time than DT and ET with 0.0130s, 0.0110s, and 0.0150s, respectively, their accuracy is 0.9773, 0.9562, and 0.9689 below that of DT and ET. Furthermore, DT and ET has the lowest CPU utilization, only 0.10%.

Table 17 summarizes the best combination for each classification system and its performance. We can see that the performances are different from each other. The best performance is for Class 2, the highest accuracy 0.9909, the lowest execution time 0.0130s and the lowest CPU utilization 0.05% respectively. The performance of Class 0 is the lowest with accuracy 0.9576, execution time 0.0310s and CPU utilization 0.20% respectively. When comparing with the best performance of the single multi-class classification system (accuracy 0.9712, execution time 0.300s and CPU utilization 0.10%), we know that using the single multi-class classification system produces more stable results than using a different binary classification system according to the target position.

Figure 6 shows the accuracy heat map for the 4 classes. The vertical axis represents the ML model, and the horizontal axis represents the FS method. Each number is the accuracy of the corresponding combination, the closer the color is to yellow the higher the accuracy, and the closer the color is to blue the lower the accuracy. We see that the overall color for Class 2 is closest to yellow, followed by Class 3 and Class 1. In contrast, Class 0 shows that the overall color is closest to dark green. This indicates that all combinations for Class 0 exhibit relatively poor performance. According to Table 5, the combination with the worst classification performance for Class 0 is DT and MI, with accuracy 0.8020, while the worst performances for Class 1, Class 2, and Class 3 do not fall below 0.94.

Figure 7 shows the distribution of data for the 4 class systems after applying two dimensional Principal Component Analysis (PCA) [43]. We found that the data distribution for Class 0 is particularly not concentrated compared with other classes, which is the reason why the classification performance of Class 0 is poor. The collisions at different positions will produce different natures of data. During data collection, collisions were conducted on different positions of the container. When collisions occurred, the container usually produced a slight offset along the collision direction, so the data generated was more concentrated, making it easier for the ML model to identify and classify. For example, in Figure 7, Class 2 and Class 3 represent the data distribution for the left and right position respectively. Their data distributions are relatively concentrated. The accuracy of Class 2, even the worst combination of DT and MI, achieved an accuracy 0.9864, the best combination of RF and PCC an accuracy 0.9985.

In contrast, when the collision occurred at the top position of container, the displacement caused by the collision is hard to occur since the container is usually placed on the ground. The container can only vibrate at its original position, producing data that is more widely distributed than other positions, correspondingly more difficult to classify. As we can see the results in Table 17 and Figure 6, the performance for Class 2 is high, which means that Class 2 is easier to classify, while the performance for Class 0 is low, which means that Class 0 is most difficult to classify. This shows that the container collision data collected by sensor data have different properties.

In conclusion, the results show that it is rather reasonable to use the single multi-class classification system with a combination of KNN and ET than separate binary classification systems. Because the results of feature selection and classification are more easily affected by the nature of the data, one system to select the optimal feature subset and classify all classes is most stable and produces the best performance. Furthermore, one system is operationally simpler since separate systems demand a lot of additional workloads and more complicated workflow.

4. Discussion

In this study, we successfully collected acceleration data through an accelerometer and applied feature selection and machine learning techniques to accurately identify the collided position of container. The contribution of this study is to verify the importance of feature selection technology in processing high-dimensional data and reducing computing costs. Accurately identifying the collided position of container, the efficiency of cargo safety supervision can be much improved, the operational efficiency of the logistics and trade industries can be improved, and human productivity can be liberated. In addition, it is possible to provide a reliable evidence for accident liability, which is of great significance to reducing container and cargo losses and reducing claim costs.

While this study enabled an accurate identification of container collided position with saved computing resources, this study also has some limitations and future research directions. The amount of data in this study is relatively small, and the investigation with different types and sizes of data on the effectiveness of feature selection methods should be studied in the future. Considering that the container collision environments in the real world may be more complex, future research should collect more diverse data through various sensors to verify and expand the methods proposed in this study.

5. Conclusions

We developed a system using acceleration data collected through an accelerometer and applying feature selection and machine learning to accurately identify the collided position of containers with reduced cost. This system offers an optimal solution for applications with limited computing resources or where cost-saving is a priority such as a mobile device like a smart container. This study not only boosted computing efficiency but also provided new technical support for the safe transportation in maritime logistics where containers are the primary mode of transportation.