Intelligent Vehicle Classification System Based on Deep Learning and Multi-Sensor Fusion

I. Introduction

II. Related Works

III. Method

A. DeepVehiSense

In DeepVehiSense, the convolution operation can be expressed as:

$$F_{\mathrm{conv}}=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}\left(I,K\right)$$

(1)

Figure 1. DeepVehiSense.

Figure 1. DeepVehiSense.

Where $I$ is the image input of the model, $F_{\mathrm{conv}}$ is the convolution feature map, and $K$ is the convolution kernel.

The vehicle classification model we proposed is based on convolutional neural networks (CNNs) [20]. The architecture of the model consists of several main parts: input, convolution, pooling, fully connected, and output layer.

The input layer accepts preprocessed vehicle images. The convolution layer uses the ReLU activation function in order to extract the image features [21]. The pooling layer employs max pooling in order to reduce the spatial dimension of features while increasing invariance to image displacement. The fully connected layer, after convolution and pooling layers, performs high-level feature processing for classification. A fully connected layer with a Softmax activation function for multi-class classification makes up the output layer.

We make improvements on the original network using adaptive network depth technology:

$$\mathrm{L}=\mathit{arg}\underset{l\in \{1,\dots ,L_{\max }\}}{\mathit{min}}\mathcal{L}\left(f^{\left(l\right)}\left(I\right)\right)$$

(2)

Where $L$ is optimal depth of the network, $L_{\max }$ is the maximum possible depth, $\mathcal{L}$ is the loss function. While the output of the input I is $f^{\left(l\right)}\left(I\right)$.

C. IntelliFeatureExtractor

In IntelliFeatureExtractor, DeepVehiSense uses the self-attention to enhance the representation ability of features. By calculating the attention weights at each position, the network can adaptively focus on the most important features for the classification task.

Figure 2. CohereSenseAggregator.

Figure 2. CohereSenseAggregator.

DeepVehiSense exploits multi-scale feature fusion and the attention mechanism to extract the vehicles’ shape features. The following is a detailed description of our innovative approach:

• Multi-scale feature fusion

The article proposed DeepVehiSense to process the original image and its downsampled version simultaneously to capture features at different scales:

$$F^{\left(s\right)}=\mathrm{Conv}\left(I^{\left(s\right)}\right)$$

(4)

Where ${\mathrm{I}}^{\left(\mathrm{s}\right)}$ is the input image of scale $\mathrm{s}$ and ${\mathrm{F}}^{\left(\mathrm{s}\right)}$ is the corresponding feature map.

• Attention mechanism

We introduce a self-attention module that enables the network to adaptively concentrate on the most useful regions in the figure for classification. The attention weight A can be calculated by the following formula:

$$A=\mathrm{softmax}\left({\displaystyle \frac{Q\cdot K^T}{\sqrt{d_k}}}+V\right)$$

(5)

Where Q, K and V represent the query, key, and value matrices, and ${\mathrm{d}}_{\mathrm{k}}$ denoting the dimensionality of the key.

The pseudo code is as follows:

E. Model Fusion with Gradient Boosting

We adopt the Gradient Boosting method to fuse features extracted from different scales and attention mechanisms to further improve the classification accuracy.

Figure 3. Training Strategy.

Figure 3. Training Strategy.

Figure 4. Experiment Result 1.

Figure 4. Experiment Result 1.

Figure 5. Experiment Result 2.

Figure 5. Experiment Result 2.

We innovatively modified the parameters of this multi-task learning model as follows:

$${\mathsf{\theta }}_{\mathrm{MTL}}=\arg \underset{\mathsf{\theta }}{\min }{\sum }_{j=1}^J{\mathsf{\lambda }}_j{\mathcal{L}}_{\mathcal{j}}\left(f_{\mathsf{\theta }}\left(x\right),y_j\right)$$

(7)

Where ${\mathsf{\theta }}_{\mathrm{MTL}}$ is the parameter of the multi-task learning model, J as tasks number, ${\mathsf{\lambda }}_{\mathrm{j}}$ as the j-th task’s weight, ${\mathcal{L}}_{\mathcal{j}}$ as the j-th task’s loss function.

IV. Experiments

C. Experimental Results

The experimental results show that DeepVehiSense has achieved significant performance improvements on the test set. Accuracy reached 95.02%, which is a significant improvement over the existing system of 91.77%. The F1 score reached 0.92, indicating that the model has achieved a good balance between recall and precision.

In accuracy analysis, as Table 1, our proposed method has achieved significant improvement in accuracy, reaching an average accuracy of 95%, compared with optical sensor-based methods (85%) and other traditional machine learning methods (75%) have significantly improved. This result shows that through deep learning and multi-sensor data fusion, our model can effectively capture the characteristics of vehicles and achieve high-precision classification in complex traffic scenes.

The F1 score (Table 1) gives us a measure of the balance between model precision and recall. Our method exhibits an average F1 score of 0.92, which further confirms the model's superior performance in reducing misclassification and improving classification comprehensiveness. Compared with other methods, our F1 score is significantly improved, which demonstrates the better generalization ability of our model in the complex vehicle classification task.

Table 1. Performance comparison of different vehicle classification methods.

Table 1. Performance comparison of different vehicle classification methods.

Method	Accuracy (%)		F1 score		Recall rate (%)
	Mean	MinMax	Mean	MinMax	Mean	MinMax
Optical sensor-based methods	85	80-90	0.75	0.70-0.80	80	75-85
Traditional machine learning methods	75	70-80	0.65	0.60-0.70	70	65-75
Deep learning method (CNN)	90	85-95	0.85	0.80-0.90	85	80-90
Multi-sensor fusion method	92	85-95	0.88	0.85-0.90	90	85-95
Our proposed method	95	90-100	0.92	0.90-0.95	95	90-100

In Recall Rate, our method also performs well, with an average recall of 95%, which shows that our model is able to identify the vast majority of vehicle instances while minimizing missed detections. This result is particularly important for application scenarios such as automated toll collection systems, where a high recall rate ensures that nearly all passing vehicles are correctly classified and billed.

As the comparison with state-of-the-art, by comparing different methods, we can see that although existing deep learning-based methods (such as CNN) have achieved good results, we further improve the performance of the model by introducing multi-sensor fusion and attention mechanisms. This shows that combining multi-source data and advanced feature extraction method can significantly enhance the accuracy and robustness for vehicle classification.

In error analysis, although our method achieves excellent results in performance, in some cases, such as extreme weather conditions or severe occlusions, the model's performance may degrade. Future work will include further analysis of these special cases and explore strategies to improve model robustness, such as by integrating more sensor data or employing more sophisticated data augmentation techniques.

V. Discussion

VI. Conclusions

References

1. Zhao, P.; Qi, C.; Liu, D. Resource-constrained hierarchical task net-work planning under uncontrollable durations for emergency decision-making. Journal of Intelligent & Fuzzy Systems 2017, 33, 3819–3834. [Google Scholar]
2. Lin, Z.; Wang, Z.; Zhu, Y.; Li, Z.; Qin, H. Text sentiment detection and classification based on integrated learning algorithm. Applied Science and Engineering Journal for Advanced Research 2024, 3, 27–33. [Google Scholar]
3. Li, M.; Tong, L.; Zhou, Y.; Li, Y.; Yang, X. Scattering from fractal surfaces based on decomposition and reconstruction theorem. IEEE Transactions on Geoscience and Remote Sensing 2021, 60, 1–12. [Google Scholar] [CrossRef]
4. Lyu, W.; Zheng, S.; Ling, H.; Chen, C. Backdoor attacks against transformers with attention enhancement. In Proceedings of the ICLR 2023 Workshop on Backdoor Attacks and Defenses in Machine Learning; 2023. [Google Scholar]
5. Yabo, A.; Arroyo, S.I.; Safar, F.G.; Oliva, D. Vehicle classification and speed estimation using computer vision techniques. In Proceedings of the XXV Congreso Argentino de Control Autom ́atico (AADECA 2016) (Buenos Aires, 2016); 2016. [Google Scholar]
6. Jahan, N.; Islam, S.; Foysal, M.F.A. Real-time vehicle classification using cnn. In Proceedings of the 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT); IEEE, 2020; pp. 1–6. [Google Scholar]
7. Chen, J.; Xu, W.; Wang, J. Prediction of car purchase amount based on genetic algorithm optimised bp neural network regression algorithm. Preprints, June 2024. [Google Scholar]
8. Asvadi, A.; Garrote, L.; Premebida, C.; Peixoto, P.; Nunes, U.J. Multimodal vehicle detection: fusing 3d-lidar and color camera data. Pattern Recognition Letters 2018, 115, 20–29. [Google Scholar] [CrossRef]
9. Feng, D.; Haase-Schutz, C.; Rosenbaum, L.; Hertlein, H.; Glaeser, C.; Timm, F.; Wiesbeck, W.; Dietmayer, K. Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Transactions on Intelligent Transporta-tion Systems 2020, 22, 1341–1360. [Google Scholar] [CrossRef]
10. Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv arXiv:1409.1556.
11. Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition; 2015; pp. 1–9. [Google Scholar]
12. Lin, Z.; Wang, C.; Li, Z.; Wang, Z.; Liu, X.; Zhu, Y. Neural radiance fields convert 2d to 3d texture. Applied Science and Biotechnology Journal for Advanced Research 2024, 3, 40–44. [Google Scholar]
13. Tan, C.; Wang, C.; Lin, Z.; He, S.; Li, C. Editable neural radiance fields convert 2d to 3d furniture texture. International Journal of Engineering and Management Research 2024, 14, 62–65. [Google Scholar]
14. Wang, J.; Zheng, H.; Huang, Y.; Ding, X. Vehicle type recognition in surveillance images from labeled web-nature data using deep transfer learning. IEEE Transactions on Intelligent Transportation Systems 2017, 19, 2913–2922. [Google Scholar] [CrossRef]
15. Zhuang, F.; Qi, Z.; Duan, K.; Xi, D.; Zhu, Y.; Zhu, H.; Xiong, H.; He, Q. A comprehensive survey on transfer learning. IEEE, 2020; Volume 109, no. 1, pp. 43–76. [Google Scholar]
16. Haghighat, A.K.; Ravichandra-Mouli, V.; Chakraborty, P.; Esfandiari, Y.; Arabi, S.; Sharma, A. Applications of deep learning in intelligent transportation systems. Journal of Big Data Analytics in Transportation 2020, 2, 115–145. [Google Scholar] [CrossRef]
17. Engel, E.; Engel, N. A review on machine learning applications for solar plants. Sensors 2022, 22, 9060. [Google Scholar] [CrossRef] [PubMed]
18. Wang, T.; Zhu, Z. Multimodal and multi-task audio-visual vehicle detection and classification. In Proceedings of the 2012 IEEE Ninth International Conference on Advanced Video and Signal-Based Surveillance; IEEE, 2012; pp. 440–446. [Google Scholar]
19. Zhou, C.; Zhao, Y.; Cao, J.; Shen, Y.; Cui, X.; Cheng, C. Optimizing search advertising strategies: Integrating reinforcement learning with generalized second-price auctions for enhanced ad ranking and bidding. 2024.
20. Dang, B.; Zhao, W.; Li, Y.; Ma, D.; Yu, Q.; Zhu, E.Y. Real-time pill identification for the visually impaired using deep learning. 2024.
21. Yan, C.; Qiu, Y.; Zhu, Y. Predict oil production with lstm neural network. Unknown Journal 2021, 357–364. [Google Scholar]
22. Li, L.; Li, Z.; Guo, F.; Yang, H.; Wei, J.; Yang, Z. Prototype comparison convolutional networks for one-shot segmentation. IEEE Access 2024, 12, 54978–54990. [Google Scholar] [CrossRef]
23. Zhao, G.; Li, P.; Zhang, Z.; Guo, F.; Huang, X.; Xu, W.; Wang, J.; Chen, J. Towards sar automatic target recognition multicategory sar image classification based on light weight vision transformer. 2024.
24. Geiger, A.; Lenz, P.; Urtasun, R. Are we ready for autonomous driving? the kitti vision benchmark suite. In Proceedings of the 2012 IEEE conference on computer vision and pattern recognition; IEEE, 2012; pp. 3354–3361. [Google Scholar]
25. Wang, S.; Liu, Z.; Peng, B. A self-training framework for automated medical report generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing; 2023; pp. 16443–16449. [Google Scholar]