Evaluating Active Learning and Classifiers on Laying Hens’ Motion Data of 27-Behavioral Classes

1. Introduction

1.1. Regulations and Policies on Animal Welfare

Innovations on animals have been continuously striking domain interest in clinical medicine [1], livestock [2], emergence rescue [3], and disable assistance [4]. Accordingly, to protect animals widely from domestic pets, livestock, wildlife, laboratory-using, etc., policies and regulations have been established and triggered a decent amount of controversial inquiries of animal welfare [5,6]. Ideally, animals shall be provided with conditions as defined: (i) animals living without pain, (ii) control of the species-adequate living environment, and (iii) positively stimulated activities and social interactions of animals, both with humans and other animals [7].

1.2. Metabolic Signs and Hormones as Indicators of Animal Welfare

Despite these ideals, fulfilling each requirement is difficult, especially given the challenge of assessing animal welfare. From a veterinary perspective, welfare assessment often relies on metabolic and hormonal indicators [8,9], including body temperature, perspiration rate, heart rate, serotonin, urinary cortisol, cortisol, and epinephrine. These parameters are considered effective for accurately understanding animal well-being. However, these procedures can be time-consuming or labor-intensive and risk disrupting the animals’ subsequent behavior, particularly in group settings [10].

1.3. Behavior Patterns as Indicators of Animal Welfare

In contrast to detailed biochemical assessments, animal behaviors offer a more scalable means of welfare evaluation, particularly when managing large flocks or herds. Behaviors often convey critical psychological information—such as fear, freedom, or aggression [11]. Therefore, employing behavior patterns as a tool to interpret welfare is common, provided non-intrusive and non-manipulative research protocols are followed [12]. For individual animals, behavior markers such as water intake frequency [13,14] can signify levels of activity. Lameness is also a concerning issue in poultry, involving biomechanical damage. Previous research combined sensor technology with direct observation of birds’ gait parameters—including speed, cadence, step length, step width, step angle, and walking gait scores—to assess welfare [15]. As the broilers walked, varying pressure on their feet yielded different light-scatter patterns, captured by a mirror beneath the walking surface and processed with image analysis software. However, due to the development limitation of that age, it was difficult to calibrate uniformly among devices. At the group level, thermal stress can serve as an important indicator of environmental comfort, evaluated by measuring distances between broilers in their activity space [16]. Besides behavior, egg production is also a critical welfare indicator in poultry practice [17]; declines in egg quality can signal disease or infection [18] or nutrient deficiencies [19]. These group indicators can be interrelated, as poultry exposed to high thermal stress show reduced fertility and egg production [20].

1.4. Smart Computing and Smart Sensing for Animal Welfare

Developments in smart sensing and computing have produced compact, cost-effective sensors with larger storage capacity and higher data throughput. Such advancements enable the use of machine learning models to analyze animal behavior [21,22,23]. For example, wearable devices containing accelerometers and magnetometers can identify multiple behaviors (e.g., standing, walking, eating, lying down) [24,25,26]. GPS allows for monitoring large-scale grazing movements [27], and radio-frequency identification (RFID) systems localize individuals within high-density enclosures [28].

1.5. Annotation Issue in Animal Behavior Analysis

Studies on animal behavior recognition generally pose the issue of lakc of well-annotated data [29,30], and to the result, there possibly existing variations in experimental design and individual animal differences can introduce biases at multiple stages, including: (1)data acquisition, (2) data preprocessing, (3) model training, (4) model prediction, and (5) model application.

During data acquisition, biases may stem from different operators, animal subjects, environmental factors (e.g., seasonal changes, diseases), and device limitations (e.g., humidity, battery power, occlusions). During preprocessing and model training, annotators may mislabel data or apply inconsistent labels owing to differences in expertise. Previous research notes that observer bias often exists in behavior experiments dependent on extensive observation [31].

Because of these factors, real-world scenarios generally demand time-intensive annotation to maintain model adaptability. As shown in Figure 1, dataset drift can occur in large-scale commercial or industrial settings, complicating animal welfare implementations. An approach that builds lightweight models and queries less biased data is thus encouraged to create efficient annotation environments that facilitate research on behavioral pattern detection, quantification, and analysis. This paper’s contributions are as follows:

• We introduce a 27-class behavior recognition framework for laying hens, which surpasses the classification scope of many existing animal behavior studies. The high complexity and class imbalance inherent in this dataset provide a unique opportunity to examine the impact of imbalanced class distributions on machine learning model performance and active learning strategies.
• We conduct a systematic evaluation of machine learning classifiers and active learning strategies, identifying distinct model convergence patterns and revealing performance degradation in boosting classifiers, a phenomenon rarely addressed in the literature.
• We integrate uncertainty-based sampling with recent advancements in active learning, incorporating density, representativeness, and diversity metrics, and assess their effectiveness in annotating both majority and minority behavior classes.
• We analyze the effect of different query batch sizes to annotation efficiency and model performance, demonstrating that larger batch sizes significantly enhance annotation efficiency while maintaining only a marginal reduction in predictive accuracy.

The remaining parts are structured. Section 2 introduces mainstream methodologies from machine learning, active learning, and feature extraction techniques. Section 3 presents experiment design of broiler 27-classes active learning. Section 4 demonstrate the results of multiple active learning informativeness driven by various algorithmic classifiers. Section 5 discusses the effectiveness of our annotation framework, limitations, and future aspects and conclusion in Section 6.

2. Methodology

We introduce the pertinent machine learning and active learning, followed by the adopted feature extraction techniques based on our precedential studies [32,33], and our proposed generalized annotation framework.

2.1. Machine Learning

We focus on the classifiers used in previous studies of animal behavior recognition. However, as they are regarded as the base for the latter active learning, we have to consider the efficiency of fitting time. There is a set of classifiers covering the mainstream categories, including tree, linear, ensemble, and neural network.

• Support Vector Machine (SVM): SVM [34] is excellent in generating decision boundaries among varying dimensional space, to fine-tune its model, and avoiding issues of under-fitting or over-fitting; its regularization adjusts through shrinking or expanding distance between two types of margin (e.g. sort margin and hard margin), enabling capturing a variety of relationships of data such as the radial, linear or nonlinear. In earlier studies, it is used in cow behavior recognition on a 9-features low-dimensional accelerometer dataset with respect to standing, lying, ruminating, feeding, walking normaly, walking abnormally, however, it is unable to recognize the lying down of cows with zero sensitivity and zero precision [35].
• K-nearest-neighboughrs (KNN): KNN is well-known in predicting house prices, since its mechanism is superior to learning the class-centroid samples and their neighbor samples, which adhering with the house prices correlated in regionally. However, its performance largely depends on whether the data has clustering characteristics to discriminate easily. In previously, it is proposed in GPS telemetry or dead-reckoning of animal position recognition and outperforming other complex models [36].
• Logistic Regression (LR): LR as one of linear models learning features linearly, and then, using sigmoid or softmax to transform the numerical values, enabling multi-classification, optimized in the direction of maximum likelihood function, it can yield probability for classifications so that can be used in animal movement analyses [37].
• Gaussian Naive Bayes (GNB): GNB is based on the Bayes theorem based on an assumption of features that are independent and follow Gaussian distribution, therefore, it is fast to handle high-dimensional data. In earlier study of dog behavioral classification with respect to standing, walking, running, sitting, lying down, and resting, it outperforms Decision Tree, KNN, and SVM [38].
• Decision Tree (DT): DT constructed with tree nodes and splitted branch hierarchically, samples will be separated into subsets driving by infomration gain. It can handle features containing missing values based on the directional mechanism. In animal behavior recognition, it especially takes priority in classifying the behavior at different activity intensity levels, such as sports momentum at different levels, still and walking. Also, the transition state between standing and lying down between animal durations [39].
• Random Forest Classifiers (RF): RF is based on an ensemble of trees by consolidating a multitude of tree’s decisions on classifications. It selects the best split at each node by considering a subset of features and using bootstrapped samples which is part of the whole training dataset. Compared with DT, which takes advantages at learning data with higher generalizability and robustness, thus avoiding over-fitting. It was applied in accelerometer-based free-grazing cows behavior recognition of feeding, lying, standing and walking [40].
• Bagging Classifier (BAG): BAG is an ensemble of classifers at meta level, in this study we adopted theDTas its basic component to benchmark.
• Extra Trees (ET): ET as ensemble classifier which is similar to RF, however, it introduces with more randomness into threshold for each feature and using the whole dataset (e.g. No Bootstrapping) for training each tree, therefore, it competing to RF with higher generalizability and speed, also previous study claims that it can be used for a wide variety of animal species, when recognizing 13-behaviors of cattle based on with accelerometer-based motion data [41].
• Light Gradient Boosting Machine (LGBM): LGBM as a boosting algorithm is efficient to training on large-scale data since it used histogram-based feature which are discretized from origin features. Its optimization relying on an ensemble of trees conducted in a leaf-wise way [42].
• Histogram Based Gradient Boosting Classifier (HGBC): HGBC is faster than LGBM in handling large-scale dataset, which is as the same boosting-based and histogram feature-based components as LGBM, where its difference optimized in a level-wise way. In previous study of accelerometer-based cattle behavior recogntion, it outperforms RF, SVM, and KNN [43].
• Multi layers perceptrons (MLP): MLP as deep learning consisted of layers of interconnected neurons and corresponding weights to pass through the activation function, input values from features therefore can transformed into output values for prediction. With a variety of loss functions, it enables learning non-linear relationship from features, however, it usually requires large amount of data to support its generalizability for multi-classifcation [44].

2.2. Active Learning

Active learning as a sub-domain of machine learning is used for efficient annotation or alleviating the issue that data are scarce or insufficient [45]. In contrast to annotating randomly or passively, active learning selectively querying the annotator with samples that are decision boundary-nearby, therefore facilitating the machine learning models to converge or optimize faster without requiring the majority of samples that are decision boundary-distant and redundant.

Active learning is emerged across various domains, including nature language process [46], medical image process [47], website data mining [48], genetics [49]. Recently, its emergences with identifying patterns or relationships based on their special or novel intricacies so that can be integrated into animal behavior analyses, including home range estimation [50] and wildlife camera trap images [51], have stressed its novel insights in exploring the animal world of finding the machine-learnable characteristics of patterns.

Those commonly used active learning strategies are: Least Confidence [52], Entropy [53], Margin [54], Model Expected Change [55], and hybrid approaches that consider density [56], cluster [57], or distance [58]. Because their mechanisms are orchestrated in multiple ways, it is necessary to validate their effectiveness and robustness for efficiently annotating animal data.

2.3. Query Strategies

2.3.1. Least Confidence (UNCERTAINTY)

Assuming that the data space $\left\{\mathcal{D}\right|L\bigcup U\}$ consists of labeled L and the other unlabeled U, where the latter is much larger than the former in the volumes. After model training based on L; For an unlabeled $x\in U$, there is a $c_{max}$ takes the largest possiblility in-between $[0,1]$. The Least Confidence strategy querying the most informative sample accordingly:

$$arg\underset{x\in U}{min}\widehat{P}(c_{max}\mid x)$$

(1)

or

$$arg\underset{x\in U}{max}1-\widehat{P}(c_{max}\mid x)$$

(2)

Note that the $c_{max}$ belong to n-classes $\left\{\mathbb{C}\right|c_1,c_2,..,c_n\}$.

2.3.2. Entropy

Assuming that $x_i\in U$ has the posterior probabilities $\widehat{P}\left(c_i\right|x)$, the Entroy can be denoted as ${\displaystyle \sum _{c_i}^n}\widehat{P}\left(c_i\right|x)log\widehat{P}\left(c_i\right|x)$, which can be seen as equal to the best sample’s informativeness:

$$arg\underset{x\in U}{max}\sum _{c_i}^n-\widehat{P}\left(c_i\right|x)log\widehat{P}\left(c_i\right|x)$$

(3)

The most-possible class $c_i$ belongs to a set of k-classes $\{c_1,c_2,..,c_k\}$, similar to Least Confidence, which uses posterior probabilities generated by the model.

2.3.3. Margin

Assuming that $x\in U$, $\widehat{P}\left(c_{max}\right|x)$ is the largest possibility predicted by the model, followed by the secondary possible class $\widehat{P}\left(c_{max-1}\right|x)$, where the samples closing to the decision boundaries are informative to provide informativeness, must have a relatively lower margin, the queried samples can be targeted as:

$$arg\underset{x\in U}{min}\widehat{P}\left(c_{max}\right|x)-\widehat{P}\left(c_{max-1}\right|x)$$

(4)

The most-possible class $c_i$ and the second-possible class $c_{m}ax-1$ belong to a set of k-classes $c_1,c_2,..,c_k$. Margin refers to the difference between the most possible class and the secondary possible class predicted by the model. Theoretically, they mutually provide information about the decision boundary; thus, the smaller it is, the more effective it is in optimization.

2.3.4. Disagreement by Query-By-Committee

Assuming that there is a committe that contains classifiers and classes with numbers of m and k, respectively. On a sample $x_i\in U$, If $x_i$ get the most votes for number of h, then the disagreement can be denoted as ${\displaystyle \frac{m-h}{m}}$, function that can search informative sample therefore:

$$arg\underset{x\in U}{max}{\displaystyle \frac{m-h}{m}}$$

(5)

Instead of relying on individual model outputs, Query-By-Committee (QBC), constituting its membership from a set of candidate models, leverages each model’s outputs to contribute to the final decision, where the informative samples mainly possess the disagreements. However, QBC driven by multiple classifiers is inappropriate for large-size, high-dimensional, and multi-classification in our study. Heuristically, we may look at some ensemble models driven by UNCERTAINTY, which consist of weak classifiers performing the same as theoretical QBC. Therefore, we excluded them.

2.3.5. Expected Maximized Change (EMC)

Assuming that $x\in U$ with its probabilities across two consecutive time-steps ${\widehat{P}}_{t-1}\left(x\right)$ and ${\widehat{P}}_t\left(x\right)$, the difference between the intersection of deciding the change, we use the Euclidean distance norm-2 to measure the changes based on posterior probabilities, where the informative samples can be querying:

$$arg\underset{x\in U}{max}\parallel {\widehat{P}}_{t-1}\left(x\right)-{\widehat{P}}_t{\left(x\right)\parallel }_2$$

(6)

EMC is to trace the most prominent change between two continuous time steps of annotating before and after according to the model behavior. Since model optimization is a process that leads to fine-tuning, as well as expanding the labeled L through annotating, therefore, querying unlabeled U led to the largest change toward the final optimization in theory.

2.3.6. Core-Set-Score (CSS)

Assuming that m-samples of $x_i\in U$ (unlabeled data) and n-samples of $x_j\in L$ (labeled data), CSS process can be decomposed into 2-steps:

• For each $x_i$, there could a sample $x_j$ taking the minimal difference represented as core-score.
• Querying the sample with the largest core score among $x_i$.

At step-1, the core-score is row-wise min of norm-1 Euclidean distance ${\parallel ·\parallel }_2$:

$$css\left(x_i\right)=\underset{j\in \{1,2,...,n\}}{min}{\parallel x_i-x_j\parallel }_2$$

(7)

At step-2, the informative sample at each iteration:

$$arg\underset{x_i\in U}{max}css\left(x_i\right)$$

(8)

CSS focuses on the diversity of samples, which leverages the unsupervised knowledge of samples’ feature vectors. The core-set-score denotes the diversity of $x_i\in U$ to labeled L, which is the same as thinking how far we use our hand to touch the surface of an objective, but in CSS, using the unlabeled U to touch labeled L and viewing the most-distant sample $x_i\in U$ as informative with diversity.

2.3.7. Density Weighted Uncertainty (DWU)

Assuming that a sample $x_i\in U$ and $neighbors\left(x_i\right)\in U$ denote its surrounded samples, DWU also can be decomposed into 2-steps:

• Calculating the average difference between $x_i\in U$ and samples belonging to its predefined k-neighbors as density.
• Synthesized the density and the UNCERTAINTY as informativeness score, querying based on the samples that contribute max value.

At step-1, we attain the density based on ${\parallel ·\parallel }_1$, the norm-1 Manhattan distance:

$$density\left(x_i\right)={\displaystyle \frac{\parallel x_i-neighbors\left(x_i\right){\parallel }_1}{k}}$$

(9)

At step-2, we query the synthesized informative samples:

$$arg\underset{x_i\in U}{max}(1-\widehat{p}\left(c_{max}\right|x_i))\times density\left(x_i\right)$$

(10)

Where the $(1-\widehat{p}\left(c_{max}\right|x_i))$ is the Least Confidence and Density Weighted Uncertainty (DWU) further accounting for the representativeness of the sample in unlabeled U.

The above mechanism-driven active learning query strategies were used for efficiency in annotation. However, no strategy can always take priority when applied in a broader context of real-world datasets. The reasons that beside the data knowledge shapes vary, individual model’s interpretability for linear, non-linear, or radial varying also. Except for the animal races and manual experiments creating the data purpose usage and structures, respectively; For the group-living animals, possibly the known phenomenon, "pecking order" in the group life. Such contexts vary for frequencies, food intake, gaits, postures, or social behaviors. For an animal individual, potentially, at different time phases, would demonstrate a behavior transition, implicating response to environmental stress. The bias seems unavoidable. However, it is also a key to animal research interest [59].

2.4. High-Dimentional Linearity of Feature Extraction

We here propose our generalized annotation framework shown in Figure 2, enabling a workflow of annotation algorithm design that is systematic and efficient. The approach includes feature extraction, selection, and behavior classification, and is designed to be transferable among various species. Although we examined in this experiment were the laying hens, there is flexibility follows from the fact that motion-based features extracted from sensor data are universally applicable; measurements of acceleration, angular velocity, and statistical time derivatives have proved useful in behavior classification for a variety of species, including sheep, dogs, and some types of wildlife [60,61,62]. The framework’s feature extraction methods capture basic movement behavior, and its labeling of behavior can be changed without altering the underlying methodology in order to suit different animals. The time-series data is extracted into high dimensional feature space with multiple statistical knowledge, including mean, standard deviation, variances, ENTROPY, moving average, and Pearson relationships combined linearly to provide the deeper informativeness to active learning. These features are detailed in Table 1.

Assuming that there are universally applicable motion-based features which can be fetched from sensor data (e.g., acceleration and angular velocity along the X-Y-Z axes) provides the generalizability across species. These traits of motion are not species-specific; they don’t represent unique biomechanical properties — they’re applicable to many animals. Whereas traditional methods are based upon species-specific behavior patterns, we explore feature abstraction, which ensures that the model learns general motion signatures instead of species-dependent features.

Additionally, active learning strategies utilized in this framework work in a species-independent manner, selecting informative samples based on probability distributions, instead of relying on prior knowledge of branch-specific behavior. So, the annotation process is data-driven uncertainty measures guided, which means that we will have the input of the most valuable and representative samples for performing the annotations, which gives efficiency while handling with accuracy. This decoupling of species-specific traits means the framework is scalable, flexible, and widely applicable to animal behavior research, decreasing the annotation burden for other animal studies, including precision livestock farming and wildlife surveillance.

3. Experiment Design

Data from sixteen laying hens were collected over four days—March 4, June 8, July 13, and July 20, 2020—between 10:00 a.m. and 4:00 p.m. Each day, a group of four individuals (weighing approximately 2112 g and aged between 61 and 75 weeks) was selected from each pen. We chose these four days specifically because laying hens among these days behaved higher mobility with more number of behavioral patterns never used in previous studies.

The primary device that accesses motion data is via a wearable device of a wireless inertial measurement unit. It contains a 3-axis accelerometer (19.62 m/s2 [2 G]) and 3-axis angular velocity sensor (500 degree/s) (i.e., gyroscope), marked as "TSDN151, 282 ATR-Promotions".

Another device is a camera placed on the inside box-top area, used to record video streams of broilers living in 100cm X 76cm cages, They are synchronized so that the annotator can thoroughly examine the animal’s behavior through a moment of videoframes and decide to annotate the sequences of sensor recordings of 1.28 sec.

The remaining environment-enrichment equipment, including a nest-box and a perch bridged at a higher position of the nest-box, a food pool, and a water pool placed aligned to the cage outward, they can be used to collect and distinguish data of various contexts, including pecking box, skipping to box from the ground, skipping from box to stretch, skipping down, eating with a lower posture, and drinking with a higher posture.

We stress that our annotation framework is designed for standardized and generalized annotation-efficient work, the annotation scenarios may vary in behavioral contexts introduced by animal-self and experimental operations. Therefore, we use 27-classes that, beyond other studies [63,64,65], where introduce more similar contexts (e.g. rest and standing), abnormal contexts (e.g., pecking others, pecking sensors, pecked by others), which is smaller in frequency but rigorously according to annotators’ multiple observations of targets and the conducted discussion afterward. Table 2. and Table 3. detail our 27- classes of broiler behaviors. We categorized the behaviors by frequency to better understand the variability and distribution of behavioral patterns observed in the hens. This categorization allows for a more nuanced analysis by differentiating between common, less frequent, and rare behaviors, enabling us to identify both typical and atypical behavioral trends. Additionally, by distinguishing between Common Behaviors, Less Frequent Behaviors, Rare Behaviors, Very Rare Behaviors, Uncommon Events, and Highly Rare Events, we could prioritize behaviors that are of greater interest for further analysis and modeling. This approach helps highlight key behaviors that may not be immediately apparent from an overall analysis of the data, ensuring a comprehensive study of the hens’ behavioral spectrum.

3.1. Simulation

There are 9081 samples of sixteen broilers labeled by experts as resources to simulate active learning, as we expect to validate the real efficiency through an array of querying strategies and classifiers, we define the basis of the simulation process as below:

• Firstly, the whole dataset is separated into training and testing parts, as 70% (6356) vs 30% (2725), respectively.
• Secondly, choose 1% of training (63) to learn our classifiers as the initial label pool, where the remaining 99%(2725) parts are regarded as unlabeled pool.
• Thirdly, the batch size of 5, 10, 15, 20, 25, and 30 are the label frequencies to inform classifiers to update themselves and the informativeness rank of the unlabeled pool, which is viewed as an iteration of active learning.
• Lastly, iteratively runs the active learning process to query the sample from the unlabeled pool into the labeled pool until the latter is exhausted.

3.2. Metrics

We use the $Accuracy$ and $F{1}_{\mathrm{macro}}$ as our metrics, also, we recorded the time cost for running each iteration. In a multi-classification task, $Accuracy$ is viewed as an overall measure that is unable to inspect the model’s predictive capabilities on the minority classes, since the minority classes take a significantly small proportion. At the same time, the $F{1}_{\mathrm{macro}}$ score can expose the model interpretability of minority classes and the generalizability. Considering both are suitable for our imbalanced 27-classes laying hens behavior dataset. They are defined as below:

• True Positive ($TP$): The number of correctly predicted positive class sample.
• False Positive ($FP$): The number of incorrectly predicted positive class.
• True Negative ($TN$): The number of correctly predicted negative class sample.
• False Negative ($FN$): The number of incorrectly predicted negative class sample.

$$Accuracy={\displaystyle \frac{TP}{TP+FP+TN+FN}}$$

(11)

$$precision={\displaystyle \frac{TP}{TP+FP}}$$

(12)

$$recall={\displaystyle \frac{TP}{TP+FN}}$$

(13)

The $F_1$ score for a class i:

$$F_1{-\mathrm{score}}_i={\displaystyle \frac{2·{\mathrm{Precision}}_i·{\mathrm{Recall}}_i}{{\mathrm{Precision}}_i+{\mathrm{Recall}}_i}}$$

(14)

$F_1-\mathrm{macro}\mathrm{score}$ is to take the average of classes of number k:

$$F_1-\mathrm{macro}=\sum _i^k{\displaystyle \frac{F_1{-\mathrm{score}}_i}{K}}$$

(15)

4. Experimental Results

As our goal is to deal with real-world animal behavior annotation scenarios using our generalized annotation framework, therefore, the referred mechanisms of the model optimization process were analyzed qualitatively. The 11-Classifiers (linear-SVC, LR, MLP, BAG, KNN, DT, GNB, ET, RF, HGBC, and LGBM) and 7-Active learning strategies (ENTROPY, MARGIN, UNCERTAINTY, RANDOM, EMC, CSS, and DWU) were carried out to provide us with valuable implications and findings.

4.1. Simulation of Batch Size 5

Based on the simulation described in Section 3.2. We adopted the additional parameters as

• Active Learning: batch_size = 5
• Data Split: random_state = 2021
• Model : random_state = 42
• Scaler Wrapping: Model would be wrapped with a preprocessing function of standardscaler before fitting, which is $z={\displaystyle \frac{(x-u)}{s}}$, where the x represents feature vector, u is mean of the feature, and s is standard deviation of the feature.

As we split the dataset according to parameter setting, the seed data always maintain the same, this is crucial because the Random strategies and cluster-based strategies can be influenced significantly by initial seed data settings. Our DWU and CSS strategies, which query the structural information as analogous to cluster strategies, as prior study observed that applying active learning for document classification [66].

To demonstrate the overall performance of each pair qualitatively, we separate classifiers based on their learning curves into three levels: 1) $Optimum$, 2) $Suboptimum$, and 3) $Irregular$, based on their efficacy and resilience during the learning process.

In Figure 3, SVC, LR, and MLP drive ENTROPY, MARGIN, UNCERTAINTY, RANDOM, EMC, CSS, and DWU. We categorize these classifiers as "$Optimum$" since they generate robust learning curves without degradation.

Our analysis shows that ENTROPY, UNCERTAINTY, MARGIN, and DWU outperform RANDOM, EMC, and CSS in terms of learning curves for both $Accuracy$ and $F_1-\mathrm{macro}$. DWU incorporates density into UNCERTAINTY; however, it performs worse than UNCERTAINTY and ENTROPY on $F_1-\mathrm{macro}$. In contrast, for $Accuracy$, DWU surpasses ENTROPY in SVC, LR, and MLP. This suggests that DWU tends to query representative samples from high-density regions dominated by majority classes, thereby improving $Accuracy$ but inadvertently excluding minority-class instances, leading to lower $F_1-\mathrm{macro}$.

Furthermore, CSS outperforms RANDOM SAMPLING in SVC, LR, and MLP on $F_1-\mathrm{macro}$, indicating that CSS is more effective at capturing minority-class animal behaviors. Among all methods, EMC exhibits the weakest performance.

In Figure 4, ENTROPY, MARGIN, UNCERTAINTY, RANDOM, EMC, CSS, and DWU are driven by BAG, KNN, DT, and GNB. We classify these models as "$Suboptimum$" since their learning curves perform worse than those of the "$Optimum$" group on $F_1-\mathrm{macro}$, which we attribute to the weaker performance of the models themselves. Interestingly, in contrast to the "$Optimum$" classifiers, CSS behaves inconsistently on BAG and KNN. When comparing CSS with RANDOM SAMPLING, CSS is superior in $F_1-\mathrm{macro}$ but inferior in $Accuracy$. This suggests that CSS effectively explores minority classes, enhancing $F_1-\mathrm{macro}$, but at the expense of querying majority-class instances, thereby reducing overall $Accuracy$.

In Figure 5, ENTROPY, MARGIN, UNCERTAINTY, RANDOM, EMC, CSS, and DWU are driven by ET, RF, HGBC, and LGBM, which we categorize as "$Irregular$" due to their learning curves exhibiting severe or slight declines after reaching peak performance. A slight decline in $F_1-\mathrm{macro}$ was observed for both ET and RF after 2000 queries. However, after an initial drop, their curves stabilized and flattened. In contrast, HGBC and LGBM experienced drastic declines with extreme volatility, leading to unreliable predictions. This instability may stem from the inherent characteristics of ensemble models, where ET and RF outperform the boosting-based models HGBC and LGBM.

Beyond enhancing active learning efficiency, these strategies also help mitigate redundancy among data points. The best $Accuracy$, $F_1-\mathrm{macro}$, and their respective stopping points are presented in Table 5 and Table 4.

Table 5 presents the highest achieved $Accuracy$ for each model-strategy combination. The best-performing pairs were LGBM & UNCERTAINTY, achieving 0.953 at 1470 annotation points, and HGBC & DWU, achieving 0.953 at 1710 annotation points. However, as shown in Figure 5, their learning curves exhibited severe perturbations after reaching their peak performance. In contrast, ET & MARGIN and RF & MARGIN showed only slight perturbations in Figure 5, achieving 0.952 at 1685 and 0.946 at 1495 annotation points, respectively. Meanwhile, for DT, all strategies failed under large annotation volumes, nearly exhausting the unlabeled pool to reach a maximum $Accuracy$ of 0.86.

Table 4 compares the highest achieved F1-score across different model-strategy pairs. The best-performing combination was MLP & UNCERTAINTY, achieving 0.778 at 2440 annotation points, followed by HGBC & DWU with 0.770 at 2355. Other notable results include:

• LGBM & ENTROPY: 0.773 at 3160 annotation points
• ET & MARGIN: 0.756 at 1515 annotation points
• RF & MARGIN: 0.746 at 1310 annotation points

Once again, DT exhibited the worst performance across all strategies. However, CSS provided some improvement, helping DT achieve 0.614 at 5080 annotation points, indicating that CSS can aid in identifying minority-class behaviors in broiler data to some extent.

4.2. Query Batch Size Effect

Given the challenge of optimizing active learning, approaches can be categorized as aggressive or conservative, depending on whether they prioritize maximizing efficiency or optimizing performance through algorithmic complexity [67]. However, less attention has been given to the learning process itself when considering a more calibrated cost metric.

Active learning inherently involves two complementary processes: exploiting existing knowledge and exploring new patterns within the data distribution. To analyze these dynamics, we examine the effects of different batch sizes and introduce real-time annotation cost at the best performance point across different classifier-strategy pairs.

While the larger query batch size hinder exploitation but enhance exploration, smaller batch sizes hinder exploration but facilitate exploitation. To quantify annotation cost, we assume 3.28 seconds per annotation, comprising 1.28 seconds per sample plus 2 seconds for expert label verification. For instance, if a classifier stops at 1000 annotation points, the real-time cost is computed as:

"1000 * 3.28 + model fitting time" "1000 * 3.28 + model fitting time" The results presented in Figure 6, Figure 7, and Figure 8 illustrate the effects of progressively increasing batch sizes on accuracy, while Figure 9, Figure 10, and Figure 11 present their impact on F1-macro score.

Generally, larger batch sizes lead to a slight performance drop (approximately 0.01 to 0.02); however, this trade-off significantly reduces annotation time, as reflected in the gradual fading of the heatmap colors—particularly when comparing batch size 5 vs. batch size 30 from Figure 6 and Figure 8, and also, Figure 9 and Figure 11.

In most cases, ENTROPY, MARGIN, and UNCERTAINTY consistently outperform RANDOM, EMC, CSS, and DWU. Furthermore, ensemble classifiers—including ET, RF, LGBM, and HGBC—demonstrated remarkable efficiency and performance under the 3.28-second annotation assumption, underscoring their effectiveness in balancing exploitation and exploration. However, as seen in Figure 5, these classifiers were affected by perturbations to varying degrees.

5. Discussion

5.1. Discussion-Hybird of Diversity, Representativeness and Density

We made a comprehensive evaluation among combinations of Classifiers & Active learning to interpret and promote a light annotation framework, the key principle is to leveraging the feature engineering to transfer enriched knowledge of origin for active learning, so that the complex algorithmic design in query strategies would become unnecessary, where other animal species would be proposed as we depicted our framework in Fig. Figure 2, We propose the high-dimensional feature engineering forms containing linear, non-linear, radial, time-domain, and frequency-domain analyses, which also can be found in our previous studies focusing on laying hens [32,33].

We found that simple designed query strategies such as Entropy, Uncertainty, and Margin tend to take absolute superiority in both convergence efficiency and final predictive capability in most cases. The ensemble classifiers exhibited decreasing performance in learning curves, with LightGBM (LGBM) and HistGradientBoostingClassifier (HGBC) showing severe instability, while Random Forest (RF) and ExtraTrees (ET) classifiers returned to stability, nevertheless, they have exhibited impressive predictive performance and efficiency outperform the others, which indicates that a boosting-specific stopping criterion that can recognize the peak of performance is crucial.

When 11-Classifiers querying samples aggressively, by increasing the batch size from 5 to 30, the exploring efficiency can be boosted obviously with slight performance declinement around 0.01 for sacrifice. We adopted 11-Classifiers for simulation based on the same initial pool setting [66]. While some may argue that our results lack sufficient experimental evidence to support the general-purpose application of our generalized annotation framework for animal behavior annotation, we believe that active learning—where multiple models are trained at each update—can be viewed as a series of iterative experiments. At each step, the sample introduced into the labeled space represents an initial stage of learning, with each subsequent training session acting as a new "shuffle experiment." This process is influenced by the initial seed data, which impacts the final outcomes of the active learning process.

When we looking at recent studies of active learning, there are versatile techniques integrated into searching diversity, representativeness and density from data distribution space [68,69,70,71,72]. Most of them focusing on the algorithmic design and have achieved the state-of-the-art performances. The primary objective of machine learning is to drive the models to thoroughly learn the knowledge offered by data distribution and be able to extending the generalizability on the unseen dataset, where the primary objective of active learning is seeking for the points where the machine learning can be converged efficienly and precisely.

Study also poses that exploring and exploiting are two faces of active learning [67]. For exploiting, the boundary-closing samples are the typically similiarly confusing but informative if they could be exploited by the vehicle of classifiers, therefore, the key notions such as UNCERTAINTY or disagreement have been emerged as an essential aspect of active learning.

In previous studies comparing QBC and 3-uncertainty strategies, QBC generally take the advantage [73]. However, if we comprehense the mechanism that QBC leveraging multiple classifiers’ vote disagreements to perform classifications, there could be an analogous mechanism when integrating UNCERTAINTY into RF, since RF consisted of multiple weak classifiers, their posterior probabilities calculated by vote of each, and to the results, under the setting of batch size 5, in Figure 6 and Figure 9, RF with least confidence, MARGIN, and ENTROPY have achieved relative satisifying outcomes if considering the efficiency.

This study offers valuable insights for generalizing annotation tasks when selecting classifiers across diverse contexts. Specifically, in animal behavior recognition, the context can vary significantly because the behavioral classes are often mixed due to the spontaneous nature of animal actions. This leads to complex scenarios involving imbalanced and long-tailed data distributions. The Density-Weighted Uncertainty (DWU) method presents a versatile active learning paradigm. It is particularly effective for identifying representative samples, as it can improve accuracy more than UNCERTAINTY-based methods alone in certain cases. However, the Core-Set-Score (CSS) method, designed primarily for enhancing diversity, outperforms RANDOM SAMPLING strategies by better exploring minority classes. This approach is especially valuable for uncovering rare or scarce patterns in animal behaviors.

5.2. Supporting Artificial Neural Network

Deep learning, as an advanced version of Artificial Neural Networks (ANN), can learn complex non-linear relationships from data, and promoting higher generalizability on unseen datasets. However, their predictive capability relies on updating parameters based on learning from a large volume of annotated data. In the interdisciplinary field of animal behavior and deep learning, a major challenge is the shortage of well-annotated datasets and the operations involved in conducting animal experiments can differ [74,75].

5.3. Facilitate Muti-Modal Animal Behavior Recognition

Using computer vision to recognize animal behavior is another critical aspect of real-time monitoring compared with using wearable sensors. However, challenges such as image occlusions, shot height, and angle variations, along with environmental disturbances are inevitable. Fortunately, growing research is focusing on aggregating information from multiple sources or models, utilizing data forms that fuse both linear and non-linear characteristics to train the final model. This sort of approaches, known as multi-modal learning [76], has been shown to improve model performance. In most cases, they are based on the fusion of data coming from RFID, GPS, IMU as a wearable device, and an optical heart rate sensor. These methods are mainly focused on information fusions. More elaborate approaches align data structures with objective functions for richer contemplation. But working with multiple sources of data increases complexity, resulting in frictions in processes or risks of bias. Integrating these components may be useful in ascertaining certain facets of animal welfare when evident in non-stationary environments. Animal bio-mechanisms, topologies, and reinforcement learning can also be exposed through data mining and/or applied animal science or engineering methodologies. These methods usually depend on the collection of large, well-annotated datasets from all sources and devices. We propose a generalized annotation framework, which could serve as a potential solution for IoT-based applications for Industry 4.0.

6. Conclusion

In this study, we evaluated the mainstream machine learning and active learning strategies, the efficiency of "Classifier & Active Learning" are visualized by learning curves and heatmaps. The 27 classes of inherent high diversity and spontaneities of animal nature is an example for other animal behavior recognition. We proposed a framework that comprises high-dimensional feature extraction processes, light classifier selection, and a simple active learning design, the development of an annotation platform or interface with the purpose of high-quality data and scarce patterns would benefit from this framework. We used 3.28 seconds in the annotator’s action of observation and decision-making, whereas most active learning applications typically focus on the number of queries. The versatility but with a degree of complexity in algorithmic design has been emerged in recent active learning studies, however, in our comparison of DWU and standard UNCERTAINTY, we found that DWU was mostly inferior, which indicates that the high complexity introduced by feature extraction, can enrich the distribution information allowing alternative simpler active learning. Nevertheless, hybrid density and diversity are novel aspects of active learning and, have the risk of being redundant and burdensome in high-dimensional feature space. In future research, the stopping criterion for boosting-based classifiers and the identification of individual behavior of laying hens are expected.