Robust Detection of Cracked Eggs Using a Multi-Domain Training Method with NSFE-MMD for Practical Egg Production

1. Introduction

Eggs, as a high-protein and nutritionally rich food, are one of the most commonly consumed dietary items among the general population[1]. Due to the physical properties of eggshells and their internal structural features[2], they are susceptible to damage during various processes such as farming, production, and transportation[3]. The presence of damaged eggs can adversely affect egg quality, leading to a reduced shelf life. This not only increases the risk of bacterial contamination in human consumption[4] but also diminishes the operational efficiency of businesses.

In general, cracked egg detection relies on percussion vibration[5,6,7] and machine vision detectionn[8]. Percussion vibration primarily utilizes sound signals and vibration sensor data to analyze whether the tested samples exhibit surface cracks. Botta et al.[9], utilizing the theory of resonant inspection (RI), extracted the acoustic features of eggs and achieved the detection of cracked eggs by combining support vector machines (SVM). However, Percussion vibration systems are affected by ambient sounds, and their accuracy decreases in actual industrial environments.

In recent years, with the rapid advancement of computer vision, notably represented by Convolution Neural Network (CNN) and Visual Transformers(ViT) [10,11,12], machine vision-based methods for crack egg detection have made significant breakthroughs. Many research papers often utilize image data collected in laboratory settings or single industrial environments for model training, achieving remarkable results on the in-domain egg data[13,14]. Xiong et al. [15] utilized various feature parameters and Support Vector Machines (SVM) for classifying cracked and non-cracked eggs. They also employed brightness reduction and region labeling techniques to preprocess egg surface images. Experimental results showed that the model achieved an accuracy of 97.5%. Bao et al.[8] employed a negative LOG (Laplace-Gaussian) operator and hysteresis threshold algorithm to enhance crack features under backlighting conditions. Experimental results indicated a recognition rate of 92.5%. Huang et al.[16] improved the YOLOv5 model by incorporating CA, integrating BiFPN and GSConv with Neck, to enable the detection and tracking of damaged, unwashed eggs on the production line.

Traditional cracked egg detection methods based on deep learning have achieved awesome progress in single in-domain egg data. However, the deployment environment for visual systems and the varieties and origins of egg samples are often changed in practical egg production. As shown in Figure 1, the performance of trained detect models often deteriorates on unknown egg detection scenarios. Many studies have shown that supervised data-driven deep learning models experience significant accuracy degradation when confronted with domain shifts between the training and real-world deployment scenarios[17,18,19]. For instance, Mohanty et al.[20] achieved a 99.34% accuracy rate in classifying crop diseases using the GoogLeNet model but faced significant challenges when handling out-of-distribution data, resulting in accuracies of 31.4% and 31.69% in two target domain datasets. In the field of egg detection, several cracked egg datasets can be acquisition during the research phase, yet the actual production process faces a large quantity of unknown egg data. The performance of deep learning models trained to assume that the training egg data and target egg data share similar features will inevitably decrease.

To address this problem, researchers attempt to improve the generalization of deep learning model from two aspects. Many works focus on improving the network architectures [21,22], such as utilizing attention mechanisms, regularization, or building a deeper model. However, the efforts are almost meaningless on unknown testing egg data in practical production. The complicated network structure design and parameter tuning usually cause the model to overfit the training data. On the other hand, the researchers start with training data. More diverse annotated data can help facilitate the model’s extraction of domain-invariant representations. However, the cost of time and samples prevents the model training data from increasing indefinitely. Some researchers also try to reduce the sample acquisition cost by generating more training samples through the generative model[23,24,25], but it still cannot reduce the multi-object labeling cost in the detection scene of the production line. Nevertheless, blindly adding multi-domain datasets is costly and may lead to feature confusion. It is necessary to judiciously utilize limited multi-domain egg data for training to enhance the model’s generalization ability in unknown cracked egg detection.

Representation learning-based methods[19,26,27] are the widely used algorithms for addressing domain shifts. Features learned by the model exhibit domain invariance, which demonstrates excellent domain generalization and can be transferred to other domains. Inspired by this, a multi-domain training method with NSFE-MDD is proposed. This approach effectively facilitates the extraction of domain-invariant features in limited egg data, thereby enhancing the performance of the defective egg detector on egg data from unknown testing domains.

In this study, egg datasets are captured from two prevalent sample acquisition scenarios (multi-channel production lines and static light box), and diverse egg samples from different origins. The multi-domain training approach with NSFE-MMD is proposed to realize robust detection of unknown egg data coming from practical production. This method is integrated into the original YOLOv5 and YOLOv8 models, which do not need to be tuned or parameter adjusted. Subsequently, the performance of the model is evaluated by cracked egg detection on distinct egg data from different origins and acquisition environments. The manuscript is organized as follows: Section 2 introduces the establishment of the data collection system, sample acquisition, and the proposed methodology. Section 3 outlines the experimentation conducted in diverse egg data of unknown testing domains. Section 4 and 5 discuss the research findings and conclusion, respectively.

2. Materials and Method

2.1. Machine Vision System

In practical production, egg data usually come from different places of origin and acquisition environments. To ensure the dataset’s relevance to the industrial production scene, a significant quantity of eggs was acquired from Wuhan, Qingdao, and Nanyang, China. The image acquisition process was carried out in the production line and the static acquisition box at the College of Engineering, Huazhong Agricultural University.

The dynamic production line, illustrated in Figure 2a, comprises a three-channel conveyor belt, camera, and light source. This setup has widely demonstrated significant advantages in capturing images depicting various surfaces of eggs[28]. Notably, it enables the transportation of up to 15,000 eggs per hour. Each channel is equipped with numerous rollers driven by motors. As a result, images captured by the production line typically feature 9-12 eggs in each frame. As the Figure 2b shows the second device is a static acquisition box typically used for collecting a small number of static samples. It is equipped with an industrial camera connected to a computer, providing static sample collection scenarios. The static acquisition device usually only captures a single egg in each image.

Figure 2. Image acquisition device

Figure 2. Image acquisition device

Figure 3. The Architectures of robust cracked egg detector with NSFE-MMD

Figure 3. The Architectures of robust cracked egg detector with NSFE-MMD

2.2. Multiple Domain Egg Samples Preparation

As shown in Table 1, a total of 800 eggs were used in this study from three provinces in China: Wuhan of Hubei, Qingdao of Shandong, and Nanyang of Henan. Shendan, Zhengda, and Wenshi are the brands of eggs. The eggs were acquired on different origins, devices, and environments, and the five egg data have distinct domain shifts to each other which can be known in Figure 4. During the data collection process on the production line, hours of video are captured using cameras, and a substantial number of training samples through frame extraction at fixed intervals.

Moreover, substantial differences can be observed among egg samples obtained from different origins, environments, and devices. Domain1, Domain2, and Domain3 serve as the known training data for the model. Domain 1 consists of washed eggs from Wuhan, collected in a static acquisition box in a laboratory environment. Domain 2 comprises unwashed eggs from Wuhan, collected through a dynamic production line. Domain 3 comprises washed eggs from Wuhan, also collected through a dynamic production line. Compared to Domain1, the dynamic acquisition in Domain2 and Domain3 introduces motion blur, and there is a significant difference in the number of samples in a single image. Domain 2, consisting of unwashed eggs, introduces additional interference from dirt compared to Domain 1 and Domain 3. Domain 4 and Domain 5 are not involved in model training; they are designated as unknown testing domains. Domain 4 and Domain 5 came from different locations with the training data, showing differences in egg varieties and shell transparency.

2.3. The architectures of multi-domain training method with NSFE-MMD

In practical scenarios demanding a balance between detection efficiency and accuracy, single-stage object detectors are widely favored for cracked egg detection[16,29]. Currently, the crack egg detection method, built upon an improved object detector, has shown promising results[30]. However, an increase in computational complexity and the number of parameters may impact the recognition performance of the object detector under the egg data with unknown distributions. Excessive parameters are prone to causing model overfitting, and the design of a redundant network structure can also impede detection efficiency. To boost the recognition performance of the model under the egg data of unknown testing domain, a strategy is proposed without requiring alterations to the original model.

The diagram of multi-domain training method with NSFE-MMD is illustrated in Figure 3. Specifically, in the training phase, egg data from different origins are input into the Domain displacement layer to perform optimal domain matching using NSFE-MMD. This is then fed into a general deep learning model (object detector) for training. In the testing phase, facing the egg data of unknown testing domains, the model can achieve good detection results without tuning or parameter adjustment.

2.4. Proposed NSFE-MMD

2.4.1. Problem statement

Compared to the detection methods currently employed in laboratory environments or single scenarios, actual egg production detection often involves dynamic multi-channel processes and variations in factors such as origin and batch. Domain shift between the testing domain egg dataset and the known domain dataset used for training leads to the degradation of model performance. It is assumed that the image domain can be expressed as:

$$D=\{X,P(X\left)\right\}$$

(1)

Where X is the feature space of i.i.d. samples from D, and $P\left(X\right)$ is the marginal probability distribution. The labeled known egg data distribution and the unknown target egg data distribution can be defined as:

$$S={\left\{({\mathrm{x}}_i;({\mathrm{y}}_i,{\mathrm{B}}_i))\right\}}_{i=1}^n\sim {\left({\mathrm{D}}_S^X\right)}^n$$

(2)

$$T={\left\{{\mathbf{x}}_{i\prime }\right\}}_{i=1}^{n^{\prime }}\sim {\left({\mathrm{D}}_T^X\right)}^{n^{\prime }}$$

(3)

Where S and T represent the data distributions of the egg data of the known training domain and the egg data of the unknown testing domain, respectively. n is the number of samples in the known egg data domain, and $n^{\prime }$ is the number of samples in the egg data of the unknown testing domain. $({\mathrm{y}}_i,{\mathrm{B}}_i)$ are the label information. ${\mathrm{D}}_S$ and ${\mathrm{D}}_T$ are the probability distributions of the known egg data domain and target egg data domain, respectively.

2.4.2. Algorithm principle of NSFE-MMD

NSFE-MMD method is proposed to utilize the known egg data domain to extract as many domain-invariant features as possible to enhance the model’s performance on the unknown testing egg domain. Computing the domain distance is essential because it identifies the domains with the greatest dissimilarity, which can be exploited as training data to enhance the model’s ability to extract domain-invariant features. By training on the domain pair with the maximum deviation, while maintaining semantic consistency, the model is better equipped to generalize to the dataset with unknown distribution, thus improving its performance on the unknown target egg data domain. The Maximum Mean Discrepancy (MMD)[31] is a method used to measure the difference between two distinct data distributions by mapping the data into a Reproducing Kernel Hilbert Space (RKHS). Here, it is assumed that there are two data distributions, representing the known and target domains, respectively. The sample sets for these distributions are denoted as and Q . Using H to represent RKHS, the mapping function, maps X to H. Thus, MMD can be expressed as follows (4).

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathrm{MMD}}^2(X_s,X_t)=& \frac{1}{n(n-1)}\sum _{i=1}^n\sum _{j=1}^{n}k(\varphi \left(x_i^s\right),\varphi \left(x_j^s\right))\hfill \\ \hfill & -\frac{2}{mn}\sum _{i=1}^n\sum _{j=1}^{m}k(\varphi \left(x_i^s\right),\varphi \left(x_j^t\right))\hfill \\ \hfill & +\frac{1}{m(m-1)}\sum _{i=1}^m\sum _{j=1}^{m}k(\varphi \left(x_i^t\right),\varphi \left(x_j^t\right))\hfill \end{array}\end{array}$$

(4)

- n is the number of samples in the first domain.

- m is the number of samples in the second domain.

- i represents the i-th sample in the first domain.

- j represents the j-th sample in the second domain. $k(\dot{)}$is the kernel function.

Due to the relatively small domain differences inherent in eggs from various origins, common feature spaces[32] often struggle to estimate the domain discrepancies among samples. A simple nonlinear kernel function has been designed to map samples into the feature space, as shown in Equation (5).

$$k(x_i,y_i)={(x_i·y_i)}^2$$

(5)

To ensure the comparability of the MMD measures across different data or feature spaces, the calculated data is normalized. The Normalized Squared Feature Estimation-MMD(NSFE-MMD) can be represented as:

$${\text{NSFE-MMD}}^2(P,Q)=\frac{MM{D}^2(P,Q)}{{max}_{\varphi \sim \mathcal{H}}MM{D}^2(\varphi ,\varphi )}$$

(6)

Here, P and Q are two probability distributions, $\mathcal{H}$ is the considered feature space, $\varphi$ is a test function within that space, $MM{D}^2(P,Q)$ is the maximum mean discrepancy between the two distributions, and ${max}_{\varphi \sim \mathcal{H}}MM{D}^2(\varphi ,\varphi )$ is the maximum value of the MMD squared over all possible test functions in the given feature space. The previously mentioned multi-origin egg datasets have their pairwise domain distances calculated based on NSFE-MMD.

Table 2 shows the domain distance predicted by NSFE-MMD between the collected egg domains in Section 2. The difference between Domain1 and Domain3 is significantly higher than that of other combinations. The optimal match domain pair can be obtained by maximum distance can maintain semantic consistency.

3. Experiment and Discussion

3.1. Model Metrics

The training results of the YOLOv5s and YOLOv8 models are assessed using mAP (mean average precision), representing the average value of all AP (average precision). AP is the mean of the PR (precision-recall) curve, reflecting the model’s accuracy in identifying cottonseeds. The relevant terms are defined as follows:

$$AP=\sum _0^{1}p\left(r\right)dr$$

(7)

$$Precision=\frac{TP}{TP+FP}$$

(8)

$$Recall=\frac{TP}{TP+FN}$$

(9)

$$mAP=\frac{\sum AP}{N}$$

(10)

$Precision$ indicating the model’s accuracy in identifying positive samples;

$Recall$ reflecting the model’s ability to correctly classify all positive samples;

$TP$ ‒ true positive samples;

$FP$ ‒ false positive samples;

$FN$ ‒ false negative samples;

N ‒ the number of samples.

3.2. Implementation Details

The training configuration is outlined in Table 3. A batch size of 16 is adopted for training the network. The optimization of the network model parameters is achieved through Stochastic Gradient Descent (SGD). The starting values for the learning rate, momentum, weight decay, and additional parameters are in line with the standard settings found in YOLOv5 and YOLOv8. Adjustments to the learning rate are conducted via the cosine annealing method. This training is carried out on a GPU, extending over 100 epochs.

In these experiments, to evaluate the effectiveness of the proposed method, The data are selected from Domain 1 (Wuhan, washed, static), Domain 2 (Wuhan, Unwashed, Dynamic), and Domain 3 (Wuhan, washed, dynamic)for training. To ensure the fairness of the experiment, Domain 4 (Qingdao, Unwashed, Dynamic) and Domain 5 (Nanyang, Unwashed, Dynamic) are designated, encompassing samples from different origins and acquisition devices, as the testing domains for the model. Following 100 epochs of training on the training dataset, the model’s final weights (last.pt) are utilized for testing on Domain 4 and Domain 5.

3.3. Training curve analysis

The cracked egg detection experiments are conducted on different egg data domains. The iterative curve of YOLOv5 is shown in Figure 5. Similar to usual in-domain detection, the training and testing egg data are both domain 1. The results demonstrate that classical YOLOv5 exhibits strong performance when trained and tested on in-domain egg data.

Figure 6 illustrates the iterative performance curve when testing on unknown domain 4(Qingdao, Unwased, Dynamic), which uses single domain 1 (Wuhan, Washed, Static) as the training data. In contrast to the results shown in Figure 5, unacceptable accuracy losses are observed when testing on unknown testing egg data in different origins and detect environments. This observation reinforces our conclusion that the model’s performance significantly deteriorates when presented with practical egg data in different varieties, origins, or environments.

Figure 7 showcases training on Domain 1 (Wuhan, Wased, Static) and Domain 2 (Wuhan, Unwashed, Dynamic), where the loss function stabilizes and rapidly converges during training. Similarly, there is an oscillation in performance on the egg data of the unknown testing domain, but it eventually recovers to a good level.

Figure 8 illustrates the curves for training on Domain 2 (Wuhan, Unwashed, Dynamic) and Domain 3 (Wuhan, Washed, Dynamic) as the training data. The loss function consistently decreases and converges on both the training set and the unknown testing data of Domain 4 (Qingdao, Unwashed, Dynamic). However, there is some oscillation in performance on the egg data of the unknown testing domain, which takes a longer duration to recover to satisfactory levels.

Figure 9 displays the curves when using Domain 1 (Wuhan, Washed, Static) and Domain 3 (Wuhan, Washed, Dynamic) as the training set. Upon analyzing various multi-domain training curves, it becomes evident that utilizing domains with the maximum domain distance as training data (Domain 1 and Domain 3) results in the model achieving the highest mean Average Precision (mAP) on the egg data of unknown testing domain (Domain 4). Larger domain distances facilitate quicker overcoming of temporary accuracy drops in the mid-training phase of multi-domain training, leading to faster convergence to higher overall performance (mAP). However, in the mid to late stages of training, the model acquires superior domain-invariant representations from multi-domain data, resulting in enhanced overall performance.

Figure 10 shows the curves when using all known domains as the training set, and it can be observed from the training curves that the model has a limited effect in improving the accuracy of the unknown data of the testing domain (domain 4). It even leads to a slight decrease in accuracy. In cases where data with large inter-domain distances are already included, adding data with smaller inter-domain distances degrades the accuracy. This suggests that an influx of too much data from different sources exacerbates the complexity of training and is not conducive to robust representation extraction. This observation aligns seamlessly with the expectations derived from the introduced multi-domain training method with NSFE-MMD.

Figure 7, Figure 8, Figure 9 and Figure 10 visually presents the iterative performance curve testing on unknown egg data, showcasing the adaptability and generalization prowess of the multi-domain training method. The experiments offer strong evidence that the use of NSFE-MMD for matching domain pairs with greater disparities within the training dataset significantly enhances the model’s ability to generalize. This method proves more effective than the random selection of training data for mitigating domain shift, thereby highlighting the practical utility of the proposed approach.

3.4. Quantitative Experimental Analysis

As illustrated in Table 4 the performance metrics of YOLOv5 when trained using different strategies across various domains. The use of a checkmark () in the "Domain1," "Domain2," and "Domain3" columns indicates the inclusion or exclusion of specific domains in the training process. "SDT" refers to single-domain training, where the model is trained on one domain only, "MDT" stands for multi-domain training involving multiple domains, and "ADT" refers to all-domain training. Specifically, the YOLOv5s model trained by the multi-domain trained method based on NSFE-MMD has a detection mAP of 86.6% on the egg data of the unknown testing domain 4, which is an increase of 86%,63.7%, and 8% compared to models trained on a single domain, and an increase of 4.7% compared to models trained on all egg data of training domains. Compared with the single-domain and common multi-domain training methods, the proposed method shows the best detection performance (mAP) in unknown domain 4. The proposed method also exhibits stronger generalization performance using less training data when facing ’ADT’ with more training data.

To further verify the method’s effectiveness, another unknown domain 5 is tested and the results are shown in Table 5. The experimental results also support the view of the paper. To be specific, the YOLOV5s model trained by the multi-domain trained method based NSFE-MMD has a detection mAP of 87.9% on the egg data of the unknown testing domain 5, which is an increase of 87.9%,3.3%, and 7.9% compared to models trained on a single domain, and an increase of 50.6% compared to models trained on all egg data of training domains.

The simplicity of the proposed method makes it possible to integrate it into other object detection models. The typical object detection model YOLOv8s was also utilized to verify the proposed method’s effectiveness.

As illustrated in Table 6, the YOLOv8s model trained by the proposed multi-domain trained method has a detection mAP of 88.8% on the egg data of unknown testing domain 4, which is an increase of 67%,5.7%, and 4.4% compared to models trained on a single domain, and an increase of 3.7% compared to models trained on all domains. The experimental results further elucidate the efficiency of the proposed method in improving performance on unknown egg data detection of practical production.

4. Conclusion

Traditional cracked egg detection generally assumes that the distributions of training egg data and testing egg data are the same. However, the performance often deteriorates due to the domain shift, such as the egg origins and varieties are different in practicality, and the detection devices are changed. This study proposes a multi-domain training strategy with NSFE-MMD to improve the performance of the original deep learning model on practical egg production detection. The performance of an original object detector trained by the proposed multi-domain training method with NSFE-MMD was tested on the unknown testing egg data. The contributions are mainly:

1)

The multi-domain training approach with NSFE-MMD is proposed. The optimal match domain pair is obtained to promote to extract maximum domain invariant characteristics. The multi-domain training approach is integrated into the original YOLOv5 and YOLOv8 models. It can be simply and efficiently applied in practical large-quantity egg production for cracked egg detection, which does not need too complex network structure improving or parameters adjusting. Hence, the complexity and application difficulty of deep learning model applied in egg production is totally reduced.

2)

The validation experiment and analysis of the proposed method and existing detect model on the unknown egg domain are conducted. The original YOLOV5 model trained by proposed multi-domain training method achieves a detection mAP of 86.6% on the unknown testing domain 4, surpassing the best performance of models trained individually on single domains. Similarly, the YOLOV8 model achieves a detection mAP of 88.8% on the unknown testing domain 4, demonstrating improvements of 8% and 4.4% respectively. Moreover, the mAP of our proposed method is 4.7% and 3.7% higher compared to models trained on all training domains. It demonstrated that the proposed multi-domain training method is robust and efficient in improving the performance of unknown cracked egg detection, which is common in practical large-quantity egg production.

3)

A multi-domain dataset utilizing egg samples from different origins and different devices was constructed. It can be provided to other studies as a multi-domain training sample and as an unknown domain to test the robustness of the model.

Future research endeavors will focus on the utilization of more robust strategies to enhance the model’s detection accuracy in egg data of unknown testing domains and extend its applicability to detecting other defect categories within eggs.

Funding: This research was funded by the National Key Research and Development Program of China (Program No. 2022YFD1300705) and the Fundamental Research Funds for the Central Universities (Program No. 2662023GXPY005).