Effective Label Subset Selection Method for Multi-Label Classification Based on Genetic Algorithm

1. Introduction

Classification is a core task in machine learning, finding widespread application across various domains such as medicine and finance. With technological advancements and the escalating demand for data analysis, classification techniques are no longer confined to simple prediction and labeling; rather, they have evolved into critical tools for addressing complex problems. However, against the backdrop of increasingly diverse application scenarios and the exponential growth of data scales, traditional binary and single-label classification methods have gradually revealed their limitations. They not only fail to fully satisfy dynamic practical requirements but also struggle to cope with the high complexity problem in real-world demands.

Consequently, to overcome these limitations, Multi-Label Classification (MLC) [1,2,3,4] has rapidly emerged as a prominent research direction within the machine learning field. Multi-label classification allows a single data instance to be assigned to multiple labels simultaneously. For instance, a news article might be tagged as both 'Politics' and 'Economics,' while a book could be categorized under both 'Technology' and 'Education.' By handling multiple related labels concurrently, multi-label classification not only captures the implicit relationships within the data more comprehensively but also reflects the complexity of real-world scenarios where categories are seldom mutually exclusive.

The challenges associated with multi-label classification have driven researchers to explore various solutions. Current research primarily concentrates on two main directions: Problem Transformation Methods and Algorithm Adaptation Methods. These two approaches address the multi-label classification problem from distinct perspectives. Problem Transformation Methods decompose the multi-label problem into multiple binary classification problems or a single-label classification task, thereby leveraging existing machine learning algorithms for processing. The most classic and representative methods are as follows: the Binary Relevance (BR) [5,6] method models each label independently as a separate binary classification problem. While simple and flexible to implement, it fails to capture the dependencies between labels. The Classifier Chains (CC) [7,8] method models labels sequentially, treating partial dependencies between labels as additional features. However, this method is highly sensitive to the selection of label order; improper order selection can lead to performance degradation. Finally, the Label Powerset (LP) [24] method treats distinct label combinations as unique classes (single labels). It is suitable for scenarios with strong inter-label dependencies. Although capable of capturing these dependencies, it suffers from data sparsity issues when the number of labels is large, which limits the model's generalization ability.

Algorithm Adaptation Methods focus on developing specialized models capable of directly addressing multi-label challenges and redesigning algorithms to tackle specific issues within multi-label classification. For example, Multi-Label k-Nearest Neighbors (MLkNN) [9,10] utilizes the distribution of neighboring labels for prediction; while suitable for low-dimensional scenarios, it struggles with high-dimensional data. The Multi-Label Decision Tree (MLDT) [11] method extends traditional decision trees to handle multi-label classification problems, flexibly modeling the joint probability of labels. However, it is highly sensitive to data noise, and its computational cost increases significantly with the number of labels.

In multi-label classification problems, each data instance may correspond to multiple labels simultaneously. These labels often exhibit specific dependencies or co-occurrence relationships; Label dependency refers to these latent correlations or conditional dependencies between distinct labels. For instance, in sentiment analysis, an article might be concurrently labeled as 'anger' and 'sadness,' suggesting a co-occurrence pattern between these emotional labels. Similarly, in medical diagnosis, labels such as 'hypertension' and 'heart disease' may demonstrate conditional dependencies. Consequently, effectively capturing and modeling these inter-label dependencies is crucial for enhancing the performance of multi-label classification. The Label Powerset (LP) [24] method holds a significant position in multi-label classification research due to its ability to implicitly consider label co-occurrence. By treating unique label combinations as single classes, it effectively captures inter-label dependencies. Nevertheless, its practical utility is often constrained by significant data sparsity and the high computational overhead required to manage an exponential number of label combinations.

To address these issues, the Random k-Labelsets (RAkEL) method [12,13] improves upon LP by dividing labels into multiple random subsets. This approach effectively mitigates the sparsity problem encountered by LP when dealing with a large number of labels and enhances classification performance through an ensemble of sub-classifiers. However, the random selection of label subsets can lead to suboptimal performance and performance fluctuations, making it difficult to fully capture the latent dependencies between labels. To overcome the limitations of RAkEL, the Active k-Labelsets Ensemble (ACkEL) method [14] optimizes label subset selection based on specific criteria, specifically considering separability in the feature space and the balance level of classes. Although ACkEL avoids the pitfalls of random selection and optimizes subset structure based on data characteristics, it still relies heavily on preset subset sizes and fails to explicitly model potential dependencies between labels. Furthermore, ACkEL suffers from high computational overhead and relies on predefined subset sizes and quantities, lacking a mechanism for flexible and dynamic adjustment.

Addressing these challenges and the deficiencies of existing methods, we propose an innovative approach that combines Genetic Algorithms (GA) [15] with a label subset selection strategy. Genetic Algorithms are optimization techniques that mimic natural selection and biological evolution. Possessing global search capabilities and dynamic adaptability, GA can effectively explore optimal label subset combinations and dynamically adjust subset sizes to accommodate different dataset characteristics, thereby alleviating the limitations of current methods regarding label selection. Through the design of the fitness function, the Genetic Algorithm implicitly accounts for label co-occurrence and dependencies.

2. Related Work

Problem transformation refers to the approach of converting a multi-label classification task into multiple binary classification problems or a single-label problem. The underlying principle of this methodology is to employ established binary or single-label classification algorithms to resolve more complex multi-label issues. Binary Relevance (BR) [5,6] represents the most fundamental problem transformation technique. Its central premise involves decomposing the multi-label task into a series of mutually independent binary classification sub-problems. By constructing an independent model for each label, the system achieves a decomposed processing of multi-label data. The BR method is particularly efficacious in scenarios where the label independence assumption holds. However, this assumption of independence entails a complete neglect of the latent correlations among labels. In practical scenarios, such an omission often constitutes a major performance bottleneck. When strong inter-label dependencies are present, the failure to leverage these relationships for predictive improvement can result in diminished classification accuracy. Nevertheless, owing to its operational simplicity, this approach has been widely adopted across a broad range of multi-label classification applications.

To address the limitations regarding label dependencies, the Classifier Chains (CC) method [7,8] was proposed. While the CC method still constructs an individual binary classifier for each label in principle, it differs by linking these classifiers into a "chain." This method arranges all labels in a specific sequence and constructs the binary classifiers sequentially. During the learning process, each binary classifier utilizes not only the original features but also the outputs of all preceding classifiers as augmented input features, thereby forming a continuous dependency modeling process. Consequently, the design of the CC method allows each classifier to account for the predictions of previous labels while predicting subsequent ones. This mechanism fully leverages the classification information from preceding labels to effectively capture latent correlations. This leads to enhanced classification performance, particularly in contexts characterized by strong label dependencies.

However, since the architecture of the Classifier Chains (CC) method incorporates the outputs of preceding classifiers as additional input features for subsequent ones, the label ordering exerts a critical influence on the final classification results. Different permutations may alter the structure and distribution of the input features. When a suboptimal label sequence is employed, the predictive accuracy of subsequent labels is susceptible to error propagation from preceding classifiers. This cumulative effect of cascading errors is particularly pronounced in scenarios involving a high number of labels or intense inter-dependencies, potentially leading to a significant degradation in classification performance or even limiting the model's applicability in certain domains. Consequently, determining the optimal label ordering has emerged as a crucial and challenging research task in practical applications.

The Label Powerset (LP) method [24] transforms a multi-label classification problem into a single-label classification task by considering the power set of the label set, essentially treating each unique combination of labels as a distinct class. While the LP approach effectively models inter-label dependencies, the number of possible label combinations grows exponentially as the number of labels increases. This exponential expansion leads to a significant computational burden, resulting in excessive training time and memory requirements. Furthermore, the performance of the LP method is hindered by data sparsity. When certain label combinations occur infrequently, the classifier may fail to extract sufficient information from the limited samples, leading to a marked degradation in learning efficacy. Such sparsity often causes the model to become biased toward more frequent label combinations while neglecting rare ones, thereby increasing the error rate for infrequent labels and ultimately undermining the overall performance and generalization capability of the model.

To address the curse of dimensionality and data sparsity inherent in the LP method, the Random k-Labelsets (RAkEL) [12,13] approach was introduced as an enhancement. The core principle of RAkEL involves randomly selecting subsets of k labels and modeling each subset using the LP method. An ensemble strategy is then employed to fuse the results from multiple sub-classifiers. Specifically, for a given label, the outputs from all relevant sub-classifiers are aggregated and divided by the total number of sub-classifiers to obtain an average score. If this average score exceeds 0.5, the final prediction is assigned a value of 1; otherwise, it is 0. By capturing label correlations while mitigating the exponential expansion of label combinations, RAkEL effectively alleviates the computational challenges of the standard LP method. However, the random selection of label subsets may result in certain labels being insufficiently modeled. In practical applications, this inherent instability can lead to significant performance fluctuations across different datasets, a limitation that is particularly evident in scenarios with a vast number of labels or highly imbalanced label distributions.

The RAkEL method can be categorized into two variants: RAkELD (Disjoint Mode) and RAkELO (Overlap Mode). In RAkELD, the label subsets are mutually exclusive, ensuring that each label is partitioned into a single, distinct subset. However, this disjoint partitioning may cause the model to ignore critical dependencies between labels, thereby degrading classification performance. For instance, if two labels exhibit a strong correlation but are assigned to different subsets, the model fails to capture their interrelationship. Conversely, RAkELO allows for overlapping labels between subsets to enhance the modeling of inter-label correlations. While this approach improves the capacity to capture dependencies, the overlap results in higher computational overhead, imposing greater demands on training time and hardware resources. Furthermore, excessive overlapping may lead to certain labels being over-modeled, which can compromise the overall stability and performance of the model.

The RAkEL method requires the configuration of the subset size k, which specifies the number of labels contained within each random label set. Specifically, for RAkELO, because it permits overlapping labels between subsets, an additional parameter m must be specified to determine the total number of randomly generated label sets. These parameters exert a non-negligible influence on the model, as the selection of k and m not only determines the coverage of the label space but also directly dictates the computational cost and classification accuracy. If the value of k is too small, the subsets may fail to sufficiently capture complex inter-label correlations. Conversely, an excessively large k triggers a curse of dimensionality similar to that of the standard LP method, drastically increasing computational complexity. Similarly, a low m value results in an insufficient number of label sets, which limits the diversity of the ensemble and compromises the model's generalization capability. On the other hand, an overly high m value leads to a redundant number of subsets, causing a waste of computational resources. Consequently, in practical applications, these parameters often necessitate iterative experimentation and tuning tailored to specific datasets.

To address these issues, Active k-Labelsets Ensemble (ACkEL) [14] introduces an improved methodology. This approach utilizes an active learning-based label selection framework and incorporates specific metrics to evaluate the quality of label subsets, thereby optimizing the selection process. The core innovation of ACkEL lies in the adoption of two metrics to evaluate the quality of label subsets. First, Fisher’s Linear Discriminant Ratio is utilized to quantify class separability within the feature space, incorporating kernel techniques to calculate inter-class and intra-class scatter indices for enhanced computational precision. Second, the balance of the data distribution is measured via the joint entropy of label combinations; this is further normalized to ensure the balance indicator remains within the range of [0,1]. The integration of these two metrics enables ACkEL to effectively assess the quality of each label subset and select the optimal subsets for model training.

The Quality Function is defined as shown in Equation (1), where $\beta$ serves as a balancing parameter to adjust the weight between the separability index of the label subset and its normalized joint entropy. The term $\tilde{S}\left({\mathbb{C}}_{{\mathcal{L}}_{\alpha }}\right)$ represents the calculated separability of the label subset, as detailed in Equation (2), which measures the distinctness of the label subset within the feature space. Conversely, $H_{norm}\left({\mathcal{L}}_{\alpha }\right)$denotes the balance of the label subset, as expressed in Equation (6), which is used to evaluate the uniformity of the label subset's frequency distribution across the dataset.

$$Qua\left({\mathcal{L}}_{\alpha }\right)=\beta \tilde{S}\left({\mathbb{C}}_{{\mathcal{L}}_{\alpha }}\right)+\left(1-\beta \right)H_{norm}\left({\mathcal{L}}_{\alpha }\right)$$

(1)

The separability measure is defined in Equation (2), in which ${\tilde{S}}_{I,J}^{\left(b\right)}$ represents the between-class scatter index (Equation 3), which quantifies the distance between the mean embeddings of classes C_I and C_J within the feature space. Conversely, ${\tilde{S}}_I^{\left(w\right)}$ and ${\tilde{S}}_J^{\left(w\right)}$ denote the within-class scatter indices for C_I and C_J, respectively. As formulated in Equation (4), these indices characterize the sample dispersion within each class relative to the kernel-induced feature space.

$$\tilde{S}\left({\mathbb{C}}_{{\mathcal{L}}_{\alpha }}\right)={\displaystyle \frac{{\tilde{S}}_{I,J}^{\left(b\right)}}{{\tilde{S}}_I^{\left(w\right)}+{\tilde{S}}_J^{\left(w\right)}}}$$

(2)

The between-class scatter quantifies the degree of discrepancy between distributions, as defined in Equation (3). In this formulation, $x_i$ and $x_j$ denote samples drawn from the class sets $S_I$and $S_J$, respectively. Here, $S_I$ and $S_J$ represent the sample sets corresponding to classes $C_I$and $C_J$, while $N_I$ and $N_J$ denote the numbers of samples in $S_I$ and $S_J$, respectively. $K(x_i,x_j)$ denotes a kernel function that measures the similarity between samples $x_i$ and $x_j$ in the feature space. In ACkEL, the Gaussian Radial Basis Function (RBF) is adopted, as defined in Equation (5).

$${\tilde{S}}_{I,J}^{\left(b\right)}={\displaystyle \frac{1}{N_I^2}}{\sum }_{x_i\in S_I}{\sum }_{x_j\in S_I}K\left(x_i,x_j\right)+{\displaystyle \frac{1}{N_J^2}}{\sum }_{x_i\in S_J}{\sum }_{x_j\in S_J}K\left(x_i,x_j\right)-{\displaystyle \frac{1}{N_I{N}_J}}{\sum }_{x_i\in S_I}{\sum }_{x_j\in S_J}K\left(x_i,x_j\right)$$

(3)

In Equation (3), the first and second terms represent the average similarity among all samples within each respective class. The third term denotes the cross-class similarity, which varies with the degree of similarity between different classes. Specifically, as the similarity between classes increases, this negative term becomes larger in magnitude, leading to a smaller overall value. Conversely, if the two classes are more dispersed in the feature space, the cross-class similarity decreases, resulting in a larger overall value. The between-class scatter reflects the distance between two classes. When the within-class similarity is high and the cross-class similarity is low, the overall measure increases, indicating greater dissimilarity between the two classes. In contrast, when the cross-class similarity becomes higher, the measure decreases, suggesting that the two classes are more difficult to distinguish in the feature space induced by the kernel function.

Equation (4) defines the within-class scatter index, which quantifies how dispersed samples from the same class are within the kernel-induced feature space. Essentially, this index reflects intra-class similarity: High similarity results in a smaller scatter value, indicating that samples are compactly clustered. Low similarity results in a larger scatter value, meaning samples are more widely dispersed. In short, a lower ${\tilde{S}}_I^{\left(w\right)}$ signifies a tighter, more cohesive grouping of data points within their respective classes.

$${\tilde{S}}_I^{\left(w\right)}=1-{\displaystyle \frac{1}{N_I^2}}{\sum }_{x_i\in S_I}{\sum }_{x_j\in S_I}K\left(x_i,x_j\right)$$

(4)

The formula of the Gaussian Radial Basis Function (RBF), as shown in Equation (5), is used to measure the similarity between samples $x_i$ and $x_j$. In this formulation, $\sigma$is a parameter that controls the smoothness (or spread) of the function. The term $\mathrm{e}\mathrm{x}\mathrm{p}(\cdot )$ denotes the natural exponential function, which transforms the distance into a similarity measure and ensures that the similarity value lies between 0 and 1. The term $\parallel x_i-x_j\parallel$ represents the Euclidean distance between $x_i$ and $x_j$. Furthermore, $\parallel x_i-x_j{\parallel }^2$ quantifies the degree of difference between the two samples in the feature space. When $\parallel x_i-x_j\parallel$ is small, the exponent approaches zero, and thus $K(x_i,x_j)$ approaches 1, indicating high similarity between the two samples. In contrast, when $\parallel x_i-x_j\parallel$ is large, the exponent becomes a large negative value, causing $K(x_i,x_j)$ to approach 0, which indicates that the two samples are nearly dissimilar.

$$K\left(x_i,x_j\right)=exp\left(-{\displaystyle \frac{{‖x_i-x_j‖}^2}{2{\sigma }^2}}\right)$$

(5)

In the Quality Function defined in Equation (1), the formulation of the Normalized Joint Entropy $H_{\mathrm{norm}}\left(L\right)$ is given in Equation (6). Here, $C_L$ denotes all possible class combinations under the label subset $L$, and $⟦\mid C\mid ≠0⟧$ represents an indicator function, which takes the value 1 when $\mid C\mid \ne 0$(i.e., when the class actually appears), and 0 otherwise. The numerator follows the conventional definition of entropy, measuring the uncertainty and uniformity of the class distribution. Specifically, for each class $C$, the product of its probability and its logarithm is computed, and then summed over all non-empty classes, where $P\left(C\right)$ denotes the probability of class $C$ in the entire dataset. The denominator is the logarithm of the total number of non-empty classes, which represents the maximum achievable entropy under the given label set. Consequently, the value of $H_{\mathrm{norm}}\left(L\right)$ is constrained within the range [0,], facilitating comparison across different label subsets. When all samples are concentrated in a single class, the value of $H_{\mathrm{norm}}\left(L\right)$ approaches 0, indicating a highly imbalanced distribution. Conversely, when the class probabilities are similar, the distribution is more uniform, resulting in a higher entropy value; after normalization, it becomes closer to 1, suggesting that the corresponding label subset exhibits a higher degree of balance.

$$H_{norm}\left(\mathcal{L}\right)={\displaystyle \frac{-{\sum }_{C\in {\mathbb{C}}_{\mathcal{L}},\left|C\right|\ne 0}P\left(C\right){\mathit{log}}_{2}P\left(C\right)}{{\mathit{log}}_2{\sum }_{C\in {\mathbb{C}}_{\mathcal{L}}}⟦\left|C\right|\ne 0⟧}}$$

(6)

The ACkEL method builds upon the principles of RAkEL but introduces an adaptive selection process to improve label subset selection. It operates in two distinct modes: In the Disjoint Mode (ACkELD), the goal is to create subsets that do not share labels, ensuring a clean partition of the label space. A single label is chosen at random to start a subset. Subsequent labels are added to the subset one by one, guided by a specific quality function until the desired subset size k is reached. The Overlap Mode (ACkELO) is more complex, designed to capture the intricate relationships between labels that frequently appear together by allowing subsets to share labels. Instead of generating only the final m subsets, the method first generates a massive pool of candidate subsets L_j, each containing k labels. Every candidate subset is evaluated using a quality score and ranked in descending order. To prevent certain common labels from dominating the ensemble while others are ignored, ACkELO monitors the selection frequency of each label: If a top-ranked subset consists only of labels that have already met a predefined coverage threshold, it is skipped. The algorithm skips ahead to the next highest-scoring subset that contains underrepresented labels. This continues until m high-quality subsets are selected that satisfy both the overlap constraints and the requirement for label coverage.

In practical applications, the ACkEL method is subject to several limitations and challenges. Primarily, the method entails high computational complexity, particularly during the label selection phase. This is because ACkEL requires the calculation of separability and balance for each candidate label subset based on the Fisher Linear Discriminant Ratio and joint entropy, respectively. Within this process, the separability calculation involves computing scatter indices for both inter-class and intra-class distributions within the feature space. Simultaneously, the balance calculation necessitates statistical frequency analysis of label combinations to compute entropy. These two components significantly increase the computational burden when dealing with a high dimensionality of labels or large-scale datasets. For instance, the separability calculation involves extensive kernel function operations, resulting in its computational complexity is proportional to the square of the number of samples. Consequently, when the sample size is extremely large, this computational overhead may become prohibitive.

Furthermore, the ACkEL method exhibits a strong dependence on parameter settings, requiring careful and fine-grained tuning. This tuning process is often computationally expensive and time-consuming. Specifically, the performance of the ACkEL method is highly sensitive to the choice of the parameter k, and the optimal value of k varies across different datasets, necessitating additional tuning. When employing the Gaussian Radial Basis Function (RBF) kernel to compute separability, the kernel parameter $\sigma$ plays a crucial role and is sensitive to the results. An inappropriate choice of $\sigma$ may lead to inaccurate estimations of separability. Moreover, the weighting coefficient $\beta$, which balances separability and label distribution uniformity, must also be adjusted according to the characteristics of the dataset in order to properly reflect their relative importance. However, such parameter tuning typically requires extensive experimentation and computational resources. For scenarios with limited computational capacity, this may pose significant practical challenges.

3. Our Approach

To address the limitations of existing methods, we propose a label subset selection approach based on a genetic algorithm, termed Genetic Algorithm for Label Selection (GALS). Unlike the RAkEL and ACkEL methods, which require a prespecified fixed label subset size, the proposed approach eliminates the need to specify the label subset size. Instead, all label subsets are generated based on the evaluation of a fitness function, which automatically identifies the most suitable label combinations. In addition, through the evolutionary mechanisms of genetic variation and selection, the proposed approach effectively mitigates the impact of random initialization on model performance.

The Genetic Algorithm [15] is a classical heuristic search algorithm whose working mechanism involves a multi-step iterative process, including population initialization, fitness evaluation, selection, crossover, mutation, and result generation. A population refers to the set of all candidate solutions, where each chromosome represents a potential solution composed of multiple genes. Each gene is encoded as a binary value (0 or 1), indicating whether a specific label is selected: a value of 1 denotes that the label is selected, whereas 0 indicates that it is not selected. Each gene corresponds to the selection decision of a particular label, and a chromosome therefore represents a specific label subset configuration. The population, consisting of multiple chromosomes, represents the entire search space of possible label subset combinations. Through the evolutionary optimization process over the population, the genetic algorithm continuously explores and evaluates different combinations of label subsets, ultimately identifying an optimal solution that enhances classification performance.

The main parameters of the algorithm include population size, number of generations, mutation rate, and crossover rate. Specifically, the population size determines the number of chromosomes in each generation; the number of generations controls the maximum number of evolutionary iterations; the mutation rate governs the probability of gene flipping within a chromosome; and the crossover rate controls the probability of gene exchange during the recombination process, thereby introducing randomness and diversity into the solution space. During the initialization phase, the initial population is generated by randomly creating multiple chromosomes. Each chromosome is represented as a one-dimensional binary vector, where each gene corresponds to a label. A value of 1 indicates that the label is selected, while 0 indicates that it is not selected. The number of genes in each chromosome is equal to the total number of labels in the multi-label classification dataset. In the fitness evaluation phase, each chromosome in the population is evaluated using a fitness function, which aims to measure the quality of the solution. The fitness function designed in our method GALS is defined as shown in Equation (7). It incorporates a weighting parameter $\alpha$ to balance the predictive accuracy of the model and the rationality of label subset selection. The final weighting scheme is determined through experimental validation across different parameter settings.

$$Fitness=\alpha \times \left(1-\mathrm{H}\mathrm{a}\mathrm{m}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{g} \mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}\right)+\left(1-\alpha \right)\times \mathrm{L}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l} \mathrm{S}\mathrm{u}\mathrm{b}\mathrm{s}\mathrm{e}\mathrm{t} \mathrm{R}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}$$

(7)

The calculation of the fitness function, as defined in Equation (7), begins by selecting a specific subset of labels based on the positions of genes with a value of 1 within the chromosome. Subsequently, the Label Powerset (LP) transformation is applied to the training set to wrap a Random Forest classifier [16], thereby converting the multi-label subset into a single-label problem for model training. Predictions are then executed on the validation set, and the following metrics are computed based on the results. The term 1 - Hamming Loss, also referred to as Label-based Accuracy, represents local accuracy and is utilized to evaluate the predictive performance for each individual label. The formula for Hamming Loss is presented in Equation (8). The value of the Hamming Loss ranges within [0,1]; a value closer to 0 indicates fewer predictive errors across all labels. In this context, N denotes the number of samples, L represents the number of labels, $y_{ij}$ is the actual value of the j-th label for the i-th sample, and ${\widehat{y}}_{ij}$ is the corresponding predicted value. The term $\mathbb{I}\left(x\right)$ serves as the indicator function, which outputs 1 if the condition $y_{ij}\ne {\widehat{y}}_{ij}$ is true, and 0 otherwise.

$$Hamming Loss={\displaystyle \frac{1}{N\times L}}{\displaystyle {\sum }_{i=1}^N}{\displaystyle {\sum }_{j=1}^L}\mathbb{I}\left(y_{ij}\ne {\widehat{y}}_{ij}\right)$$

(8)

Label Subset Ratio (LSR) is employed to evaluate the rationality of the number of selected labels within a subset. As defined in Equation (9), in which $L_{selected}$ represents the number of labels selected by a specific chromosome, while $L_{ideal}$ denotes the ideal number of labels, defined as half of the total label count. The LSR ranges between [0,1]. An LSR value closer to 1 indicates a more appropriate quantity of selected labels. Conversely, a surplus or deficiency in the label count will result in a lower LSR. This metric is designed to encourage a balanced label selection, thereby preventing scarcity of information or excessive computational burden on the model.

$$\mathrm{L}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}\mathrm{S}\mathrm{u}\mathrm{b}\mathrm{s}\mathrm{e}\mathrm{t}\mathrm{R}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}=1-{\displaystyle \frac{\left|L_{selected}-L_{ideal}\right|}{L_{ideal}}}$$

(9)

For the selection operation in the genetic algorithm, GALS adopts an elitism strategy. Chromosomes are ranked according to their fitness scores, and the top 40% with the highest fitness values are retained as “elites” to ensure the preservation of high-quality genetic information. For the crossover operation, GALS employs the uniform crossover method. Two elite chromosomes are randomly selected as parents, and each gene position is examined sequentially. A random number is generated for each gene: If the random number is less than the crossover rate, the gene at the corresponding position from the second parent is copied to the offspring. Otherwise, if the random number is greater than or equal to the crossover rate, the offspring retains the gene from the first parent at that position. The crossover process produces only a single new chromosome, i.e., one offspring. For the mutation operation, the bit-flip mutation method is adopted. Each gene in the chromosome is subjected to a probabilistic check based on the mutation rate. If mutation is triggered, the gene value is flipped (i.e., from 0 to 1 or from 1 to 0), thereby increasing genetic diversity within the population and preventing the search process from becoming trapped in local optima.

In the termination phase, we design two stopping criteria to control the execution process, aiming to balance the convergence speed and search capability of the algorithm while avoiding premature convergence or inefficient search. First, the algorithm terminates automatically when the maximum number of generations is reached. Second, if the growth rate of the fitness value remains below 1% for five consecutive generations, it indicates that the improvement in fitness has stagnated and the algorithm may have reached a stable state. In such a case, the process is terminated early to conserve computational resources. The fitness growth rate is defined in Equation (10), where ${\max \text{\_}\mathrm{Fitness}}_t$ denotes the maximum fitness value at generation $t$, and ${\max \text{\_}\mathrm{Fitness}}_{t-1}$ represents the maximum fitness value at generation $t-1$.

$$\mathrm{C}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e} \mathrm{R}\mathrm{a}\mathrm{t}\mathrm{e}={\displaystyle \frac{\left|{\mathrm{m}\mathrm{a}\mathrm{x}\_\mathrm{F}\mathrm{i}\mathrm{t}\mathrm{n}\mathrm{e}\mathrm{s}\mathrm{s}}_t-{\mathrm{m}\mathrm{a}\mathrm{x}\_\mathrm{F}\mathrm{i}\mathrm{t}\mathrm{n}\mathrm{e}\mathrm{s}\mathrm{s}}_{t-1}\right|}{\mathrm{m}\mathrm{a}\mathrm{x}\_{\mathrm{F}\mathrm{i}\mathrm{t}\mathrm{n}\mathrm{e}\mathrm{s}\mathrm{s}}_{t-1}}}$$

(10)

For each generation, our algorithm GALS evaluates the fitness of every chromosome. If a chromosome’s fitness is greater than or equal to a predefined threshold, it is stored in the “best chromosome” set. However, if no chromosome satisfies this threshold, the best chromosome set becomes empty. To prevent this situation from affecting subsequent execution, GALS bypasses the current termination condition and forcibly proceeds to generate the next generation, ensuring that the search process can continue exploring the solution space.

To ensure that the final set of “best chromosomes” covers all labels and to prevent prediction failures caused by unselected labels, we introduce a label compensation mechanism. For each label that is not covered by the selected best chromosomes, GALS performs forced compensation individually. Specifically, the gene corresponding to the missing label is set to 1 in the chromosome, and the entire genetic algorithm process is re-executed from the beginning, including population initialization, fitness evaluation, selection, crossover, and mutation. Although the overall framework of the algorithm GALS remains unchanged, during this compensation process, the fitness threshold is reduced by a little number (0.075 in our experiments) to incorporate potential solutions that may fall slightly below the original threshold. Furthermore, the label compensation mechanism addresses only one missing label at a time to avoid the risk of overcorrection caused by enforcing multiple labels simultaneously. This mechanism effectively ensures that every label has the opportunity to be selected by at least one chromosome, thereby maintaining the completeness of label coverage and predictive capability.

In the model construction phase, GALS adopts the Label Powerset (LP) method to transform the multi-label classification problem into a single-label problem, and employs Random Forest [16] as the base classifier. For each best chromosome, the corresponding labels are selected as label subset according to its gene encoding, and an independent classification model is constructed based on the training dataset for the label subset. These models correspond to different label subsets and collectively form multiple mutually independent classifiers.

To integrate the prediction from multiple models, GALS employs a voting-based decision mechanism. For each label, the number of models involved in predicting the label is determined and the number of votes from all models is counted. For the $i$-th instance and the $j$-th label, the predictions from all models participating in that label (i.e., models corresponding to chromosomes with gene value 1 at the $j$-th position) are first summed and then divided by the number of participating models to obtain the average prediction score. If this average value is greater than or equal to 0.5, the final prediction is set to 1; otherwise, it is set to 0. The formulation is given in Equation (11), where ${\widehat{y}}_{ij}$ denotes the final prediction for the $i$-th instance on the $j$-th label, ${{\widehat{p}}_{ij}}^{\left(m\right)}$represents the prediction made by the $m$-th model, $M_j$ is the set of models corresponding to chromosomes with gene value 1 for the $j$-th label, and $\mid M_j\mid$ denotes the size of the set $M_j$, that is the number of models participating in the prediction of the $j$-th label. Through the aforementioned voting mechanism, the decisions of multiple best chromosomes across different labels can be effectively integrated, thereby improving the stability and overall quality of the predictions.

$${\widehat{y}}_{ij}=\left\{\begin{array}{c}1,if{\displaystyle \frac{1}{\left|M_j\right|}}{\sum }_{m\in M_j}{{\widehat{p}}_{ij}}^{\left(m\right)}\ge 0.5,\\ 0,otherwise\end{array}\right.$$

(11)

4. Experiments

To comprehensively evaluate the performance of the proposed method in multi-label classification, we select several publicly available and representative multi-label datasets for experimentation. These datasets are widely utilized in relevant multi-label classification research and span various domains, including music (cal500 [17] and emotions [18]), image processing (flags [19], Scene [6], and corel5k [20]), biology (GnegativePseAAC [21], GpositivePseAAC [21], PlantPseAAC [21], and yeast [22]), and video (mediamill [23]). These datasets exhibit high diversity in terms of the number of instances, features, labels, and label density. Such variety not only provides a multi-faceted validation environment but also ensures the comparability and reference value of the experimental results. To enhance the clarity and readability of the experimental results, certain dataset names have been simplified: GnegativePseAAC, GpositivePseAAC, and PlantPseAAC are referred to as Gnegative, Gpositive, and Plant, respectively.

Table 1 describes the fundamental characteristics of each dataset, including the dataset name (Name), application domain (Domain), number of samples (Instance), number of attributes (Attribute), number of input features (Feature), number of labels (Label), number of label sets (Label Set), label cardinality (Cardinality), and label density (Density). Label Set refers to the collection of all label combinations present in the dataset, representing the diversity of label distributions. Theoretically, if a dataset contains L labels, the total number of possible label combinations is $2^L$. However, in practice, the number of observed label sets is usually significantly lower than $2^L$ due to the sparsity of label distributions, as many theoretical combinations may never occur in real-world data. Cardinality reflects the average number of labels associated with each sample, defined as the ratio of the total number of label occurrences to the total number of instances. This metric measures the degree of multi-label assignment; a higher value indicates that samples are, on average, annotated with more labels. To further quantify the distribution, Density is employed to describe the degree of label sparsity, calculated as the ratio of Cardinality to the total number of Labels. A lower density value signifies a sparser label distribution, meaning most samples contain only a few labels. By analyzing these metrics, we provide a comprehensive overview of dataset characteristics, ensuring the applicability and robustness of the proposed method across diverse scenarios.

4.1. Evaluation Metric

In our experiments, we use four evaluation metrics: Hamming Loss, Subset Accuracy, Weighted F1-score, and Micro F1-score. Hamming Loss is employed to measure the fraction of incorrectly predicted labels relative to the total number of labels in a multi-label classification task. It is defined as shown in Equation (8). Furthermore, 1 - Hamming Loss corresponds to the Label-based Accuracy. Subset Accuracy represents the proportion of samples for which the predicted label set exactly matches the ground truth set. This is a stringent metric, as a prediction is considered correct only if all labels for a given sample are predicted accurately. It is defined as shown in Equation (12), where $Y_i$ represents the set of actual labels for the i-th sample, $\widehat{Y}$ represents the set of predicted labels for the i-th sample. $\mathbb{I}\left(x\right)$ is the indicator function, which equals 1 if the condition $Y_i={\widehat{Y}}_i$ is true and 0 otherwise.

$$Subset Accuracy={\displaystyle \frac{1}{N}}{\displaystyle {\sum }_{i=1}^N}\mathbb{I}\left(Y_i={\widehat{Y}}_i\right)$$

(12)

The F1-score is utilized to evaluate the predictive performance for a specific label, as defined in Equation (13). Precision represents the ratio of the number TP of instances correctly predicted as that label to the total number T of instances predicted as that label. Specifically, it quantifies the proportion of instances correctly identified by the model among all predictions made for that label. Recall, on the other hand, is the ratio of the number TP of instances correctly predicted as that label to the total number P of actual instances belonging to that label, measuring the model's ability to successfully capture all relevant instances from the dataset.

$$\mathrm{F}1-\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$$

(13)

The Weighted F1 is computed by first calculating the F1-score for each label and then taking the weighted average of these scores, where the weight of each label is determined by its proportion in the overall dataset, as shown in Equation (14), where $F{1}_k$ denotes the F1-score of the $k$-th label, and $w_k$ represents the ratio of the number of samples in the $k$-th label to the total number of samples. If certain labels account for a larger proportion of the dataset, their corresponding F1-scores will have a greater influence on the final result. The Weighted F1-score mitigates the impact of data imbalance by preventing the evaluation metric from being disproportionately affected by labels with a small number of samples. Therefore, it is particularly suitable for scenarios where the overall performance should primarily reflect the performance on the majority labels, while still partially addressing the issue of class imbalance.

$$\mathrm{W}\mathrm{e}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}\mathrm{e}\mathrm{d} \mathrm{F}1={\displaystyle {\sum }_{k=1}^k}{w}_k\times {\mathrm{F}1}_k$$

(14)

Micro F1 is calculated by first aggregating the total number TP of instances correctly predicted across all labels, the total number T of predicted instances for all labels, and the total number P of actual instances belonging to all labels, and then computing the overall Precision and Recall based on these aggregated values. Finally, the Micro F1 score is obtained by using the F1-score formulation as shown in Equation (13). Similar to the Weighted F1-score, Micro F1 is more influenced by high-frequency labels and thus better reflects the predictive performance on the majority of samples. It is particularly suitable for evaluating overall aggregated performance, especially in scenarios with highly imbalanced data, where most labels have a large number of positive instances while a few labels contain only a small number of samples, and the primary focus is on overall performance.

4.2. The parameter Setting

We design the fitness function by integrating Hamming Loss and the Label Subset Ratio (LSR) as evaluation criteria for selecting label subsets. The aim is to balance model accuracy and the distributional consistency of label predictions through appropriate weight allocation between these two components, as shown in Equation (7), where α and 1−α denote the weights assigned to (1 − Hamming Loss) and the Label Subset Ratio, respectively. To investigate the impact of different weight configurations on model performance, three sets of weighting parameters were designed. In the first configuration, α = 0.8, where 80% of the weight is assigned to Hamming Loss, emphasizing the accuracy of label set prediction, while the remaining 20% is allocated to LSR to ensure a balanced distribution of predicted labels. In the second configuration, α = 0.9, 90% of the weight is assigned to Hamming Loss, placing greater emphasis on classification accuracy compared to the first configuration, while 10% is reserved for LSR to maintain diversity in label prediction. In the third configuration, α = 1.0, the entire weight is assigned to Hamming Loss, considering only the accuracy of multi-label prediction while completely disregarding the rationality of label subset selection. Based on extensive experimental comparisons, the parameter α which is set to 0.9 achieved the best overall performance across most datasets. Therefore, α = 0.9 was selected as the final weight configuration for the fitness function.

To evaluate performance in multi-label classification tasks for various problem transformation methods, we compare classical multi-label classification methods Binary Relevance (BR) [5,6], Classifier Chains (CC) [7,8], Label Powerset (LP) [2], and label selection methods RAkELD [12,13], RAkELO [12,13], ACkELD [14], and ACkELO [14] for improving LP methods, with our proposed method GALS (Genetic Algorithm for Label Selection). To ensure experimental consistency for RAkELD and RAkELO, we adopted uniform settings for all datasets, with the corresponding k and m values presented in Table 2. Specifically, the label subset size k was set to 3. Such a configuration is widely utilized in the literature to maintain a balance between complexity and representativeness, allowing sub-classifiers to adequately learn inter-label dependencies. The number of sub-classifiers, denoted by m, was set to be equal to the number of best chromosome in the GALS method. This configuration was adopted to ensure a fair comparison between the RAkEL-based methods and GALS under an equivalent number of models. By controlling for model quantity, performance differences between the approaches can be more directly attributed to their label subset selection strategies.

To conduct a comprehensive comparison and validation, ten datasets with diverse characteristics from different domains were selected for experimentation. Due to the mediamill dataset having as many as 101 labels and up to 6,555 unique label combinations, the Label Powerset (LP) method was unable to complete the experiments because of memory limitations. Therefore, the results for the mediamill dataset are presented in a separate table, allowing for a clearer comparison among the methods that successfully completed the experiments. In the experimental tables, bold values indicate the best performance among different methods under the same dataset for a given multi-label classification metric. Each bold value is followed by a number in parentheses, which represents the ranking of the method under that metric, where (1) denotes first place, i.e., the best performance.

4.3. Experimental Results

Table 3 presents the Hamming Loss for various multi-label classification methods across nine dataset, which measures the prediction error rate, with values ranging from 0 to 1; a lower value indicates that the predicted labels are closer to the true labels and correspond to a lower error rate. As observed in Table 3, the GALS method achieves the lowest Hamming Loss on most of the datasets compared to other methods and demonstrating strong adaptability. While GALS yields an average Hamming Loss of 0.132, the difference from the best-performing method is minimal. Its stability and overall accuracy highlight its strong competitiveness as an effective approach for multi-label classification.

Table 4 presents the Subset Accuracy of various multi-label classification methods across nine datasets. Subset Accuracy is a fundamental evaluation metric in multi-label classification that measures a model’s ability to correctly predict the entire set of labels for each instance. Its values range from 0 to 1, with higher scores indicating superior predictive performance. Nevertheless, Subset Accuracy is considered an extremely strict metric because an instance is regarded as correctly classified only when all associated labels are predicted accurately; the misclassification of even a single label causes the entire prediction to be counted as incorrect. Consequently, in datasets with a large label space, weak dependencies among labels, or severe label imbalance, Subset Accuracy often yields relatively low values. Therefore, this metric is particularly effective for evaluating a model’s capability to precisely predict complete label combinations.

As shown in Table 4, the LP method achieves the best performance on the majority of the datasets. This is primarily attributed to its strategy of transforming the multi-label classification task into a single-label problem by treating each label combination as a distinct class. Consequently, LP can more precisely capture label co-occurrence relationships. However, while this direct transformation yields superior results in terms of Subset Accuracy, it incurs significant computational complexity and memory requirements. These overheads limit the applicability of the LP method to datasets with a large number of labels or an excessive variety of label combinations. Our GALS method achieves performance consistently comparable to that of the Label Powerset (LP) method across all datasets and even outperforms it on the flags dataset. These results indicate that GALS effectively employs a genetic algorithm to identify the most representative label subsets, allowing the Subset Accuracy metric to attain performance close to that of the LP method, despite LP requiring the computation of all possible label combinations.

Table 5 presents the performance of different multi-label classification methods on nine datasets in terms of the Weighted F1-score. The Weighted F1-score is computed by taking a weighted average of the F1-scores for each label, where the weights are determined by the number of samples associated with each label. Because this metric balances the importance of each label according to its sample size, it provides a more accurate evaluation of overall model performance in scenarios with imbalanced label distributions.

As shown in Table 5, the GALS method demonstrates stable and competitive performance across most datasets, achieving the best results on majority of the datasets. This indicates that GALS can effectively capture label correlations and produce accurate predictions, while maintaining a certain degree of adaptability across different types of datasets. Moreover, the average Weighted F1-score of GALS is higher than that of the other methods. It consistently achieves strong and stable performance on most datasets and surpasses the LP method. These results demonstrate that GALS can effectively select the optimal label combinations, and its label compensation mechanism not only ensures that every label is selected but also effectively preserves the co-occurrence relationships among labels.

Table 6 presents the Micro F1-score performance of various multi-label classification methods across nine datasets. The Micro F1-score is a critical metric for evaluating the global predictive performance of multi-label classification models. It is calculated by aggregating the prediction results across all samples to determine an overall F1-score. Ranging from 0 to 1, a higher value signifies superior predictive accuracy. This metric provides an accurate reflection of the model's overall efficacy across different labels. As observed from Table 6, our GALS method demonstrates stable and superior performance relative to all other methods, highlighting its robust predictive capability across diverse multi-label classification domains. Its consistent top-tier ranking across individual datasets reflects its robust competitive advantage and reliability in multi-label classification tasks.

Finally, Table 7 summarizes the evaluation results of various multi-label classification methods on the Mediamill dataset. Due to the large number of labels (101) and the substantial number of label combinations (6,555) in the Mediamill dataset, the Label Powerset (LP) method failed to complete the experiments within the designated experimental environment due to insufficient memory resources. Consequently, the experimental results for the mediamill dataset are discussed separately. The primary reason for the LP method's failure is its requirement to treat every label combination as a distinct new class for model training. This leads to an exponential increase in memory demand that exceeded the system's capacity.

As shown in Table 7, GALS achieved the top ranking in Hamming Loss, Subset Accuracy, Weighted F1-score, and Micro F1-score, demonstrating its superior adaptability and competitiveness in high-label-count scenarios. Our GALS method outperformed all other methods across all the indicators, validating that the Genetic Algorithm based label selection strategy possesses robust overall predictive performance and stability in large-scale label spaces.

5. Conclusions

In traditional multi-label classification, the Label Powerset (LP) method [2] captures label dependencies by treating each label combination as a single class, achieving superior predictive accuracy. However, it suffers from severe data sparsity due to the exponential growth of label combinations and imposes significant computational overhead, which restricts its practical application. To mitigate these issues, the Random k-Labelsets (RAkEL) method [12,13] partitions the label space into several small subsets. Although RAkEL effectively alleviates the complexity associated with label combinations and improves performance through ensemble learning, its random selection of label subsets often fails to adequately capture latent label dependencies, resulting in suboptimal performance and high variance. In addition, the Active k-Labelsets (ACkEL) method [14] introduces specific criteria for label subset selection; however, it relies on predefined parameters for both the size and number of subsets, which restricts its ability to fully optimize label subset construction.

Therefore, we propose a Genetic Algorithm based dynamic label subset selection method GALS to overcome the inherent limitations of RAkEL and AckEL which the size and number of label subsets must be predefined as fixed values. Unlike the previous approaches, GALS dynamically adjusts subset sizes through a genetic algorithm-driven fitness function. This allows label subset dimensions to adaptively align with specific label characteristics, thereby eliminating the constraints and lack of flexibility associated with static parameterization. Furthermore, GALS can automatically generate optimal label combinations via the fitness function, which mitigates the risk of performance fluctuations.To ensure robust performance, GALS leverages genetic variation and selection mechanisms to minimize the impact of initial randomization. Additionally, a specialized label compensation mechanism is integrated to guarantee that all labels participate in the modeling process. Experimental results demonstrate that GALS maintains stable performance across comprehensive metrics and outperforms all other methods on most datasets. These findings indicate that GALS can effectively identify representative label subsets and improve predictive accuracy.