Exploring Metaheuristic Optimized Machine Learning for Software Defect Detection on Natural Language and Classical Datasets

1. Introduction

The role of software has already reached a critical point where a widespread issue could have serious consequences for society [1]. This emphasizes the need for a robust system for error handling. The consequences of software defects can range from trivial to life-threatening, as the applications of software range from entertainment to medical purposes. The Internet of Things (IoT) and artificial intelligence (AI) massively contribute to this software dependency [2], as more devices are running software and more devices are running more complex software. The responsibility of developers increases as some use cases like autonomous vehicles and medicine require much more extensive testing [3]. Even though a software defect can be a mere inconvenience in some cases, even those cases would benefit from software defect prediction (SDP) [4]. The key contribution of SDP in the software development is in the testing phase. The goal is to prioritize modules that are prone to errors. Such insights into the state of the project can allow developers to discover errors sooner or even prepare for them in certain environments.

The process of producing software is called the software development life-cycle (SDLC) during which SDP should be applied to minimize the number of errors. Software is almost never perfect and it has become common practice for developers to release unfinished projects and work on them iteratively through updates. With the use of code writing conventions and other principles, errors can be minimized, but never fully rooted out. Therefore, a robust system that would assist developers in finding errors is required. With a system like such, the errors could be found earlier which could prevent substantial financial losses. To measure the quality of the code defect density is calculated. The most common form of such measurement is the number of defects per thousand lines of code (KLOC).

The advancements in AI technologies, specifically machine learning (ML) show great potential for various NLP applications [5]. Considering that the code is a language as well, this potential should be explored for predictions regarding programming languages as well. When natural and programming languages are compared various similarities can be observed, but the programming languages are more strict in terms of writing rules which aids pattern recognition. The quality control process can be improved through AI use which can simplify the error detection process and ensure better test coverage. To detect the errors in text it is necessary to understand what techniques are applied through NLP to achieve this. Tokenization is one of the key concepts and its role is to segment text into smaller units for processing. Stemming reduces words to their basic forms, while lemmatization identifies the root of the word through dictionaries and morphological analysis. Different parts of sentences such as nouns, verbs, and adjectives are identified through the parsing process. The potential of NLP for the SDP problem is unexplored and a literature gap is observed.

However, the use of these techniques does not come without a cost. Most sophisticated ML algorithms have extensive parameters that directly affect their performance. The process of finding the best subset of these parameters is called hyperparameter optimization. The perfect solution in most cases cannot be achieved in a realistic amount of time, which is why the goal of the optimization is to find a sub-optimal solution that is very close to the best solution. However, a model with hyperparameters optimized for one use case does not yield the same performance for other use cases. This is the problem that is described by the no free lunch (NFL) theorem [6] that states that no solution provides equally high performance across all use cases. This work builds upon preceding research [7] to provide a more in depth comparison between optimizers, as well as explore the potential of NLP to boost error detection in software source code.

• A proposal for a two layer framework that combines CNN and ML classifiers for software defect detection.
• An introduction of a NLP based approach in combination with ML classifiers for software defect detection.
• A introduction of a modified optimization algorithm that builds upon the admirable performance of the original PSO.
• The application of explainable AI techniques to the best performing model in order to determine the features importance on model decisions

The described research is presented in the following manner: Section 2 gives fundamentals of the applied techniques, Section 3 describes the main method used in this research, Section 4 provides the settings of the performed experiments along necessary information for experiment reproduction, Section 5 follows with the results of experiments, and Section 6 provides the final thoughts on performed experiments along possibilities for future research.

2. Related works

Software defects, otherwise known as bugs, are software errors that result in incorrect and unexpected behavior. Various scenarios produce errors but most come from design flaws, unexpected component interaction, and coding. Such errors affect performance, but in some cases, the security of the system can be compromised as well. Through the use of statistical methods, historical data analysis, and ML such cases can be predicted and reduced.

The use of ensemble learning for SDP was explored by Ali et al. [8]. The authors presented a framework that trains random forest, support vector machine (SVM), and naive Bayes individually, which are later combined into an ensemble technique through the soft voting method. The proposed method obtained one of the highest results while maintaining solid stability. On the other hand, Khleel et al. [9] explore the potential of a bidirectional long short-term model for the same problem. The technique was tested with random and synthetic minority oversampling techniques. While high performance was exhibited in both experiments, the authors conclude that the random oversampling was better due to the class imbalances for the SDP problem. Lastly, Zhang et al. [10] propose a framework based on a deep Q-learning network (DQN) with the goal of removing irrelevant features. The authors performed thorough testing of several techniques with and without DQN, and the research utilized a total of 22 SDP datasets.

The application of NLP techniques in ML is broad and Jim et al. [5] provide a detailed overview of ML techniques that are suitable for such use cases. The use cases reported include fields from healthcare to entertainment. The paper also provides depth into the nature of computational techniques used in NLP. Different approaches include image-text, audio-visual, audio-image-text, labeling, document level, sentence level, phrase level, and word level approaches to sentiment analysis. Furthermore, the authors surveyed a list of datasets for this use case. Briciu et al. [11] propose a model based on bidirectional encoder representations from transformers (BERT) NLP technique for SDP. The authors compared RoBERTa and CodeBERT-MLM language models for the capture of semantics and context while a neural network was used as a classifier. Finally, Dash et al. [12] performed an NLP-based review for sustainable marketing. The authors performed text-mining through keywords and string selection, while the semantic analysis was performed through the use of term frequency-inverse document frequency.

The use of metaheuristics as optimizers has proven to yield substantial performance increases when combined with various AI techniques [13]. Jain et al. [14] proposed a hybrid ensemble learning technique optimized by an algorithm from the swarm intelligence subgroup of ML algorithms. Some notable examples of metaheuristic optimizers include the well established variable neighborhood search (VNS) [15], artificial bee colony (ABC) [16] and bat algorithm (BA) [17]. Some recent additions also include COLSHADE [18] and the recently introduced sinh cosh optimizer (SCHO) [19]. Optimizers have shown promising outcomes applied in several field including timeseries forecasting [20], healthcare [21] and anomaly detection in medical timeseries data [22]. Applying hybrid optimizers to parameter tuning has demonstrated decent outcomes in preceding works as well [23,24,25].

The SDP problem is considered to have nondeterministic polynomial time hardness (NP-hard), as the problem cannot be solved by manual search in a realistic amount of time. Hence the optimizers are to be applied such as swarm intelligence algorithms. However, the process is not as simple since for every use case a custom set of AI techniques has to be applied due to the NFL theorem [6]. Furthermore, the problem of hyperparameter optimization which is required to yield the most out of the performance of AI techniques is considered NP-hard as well. The application of NLP techniques for SDP is limited in literature, indicating a research gap that this work aims to bridge.

2.1. Text Mining Techniques

Introduced in 2018 [26], the BERT technique is based on the mechanism of attention which is used to determine the meaning of a text or sentence. The technique was developed by a Google research team and since it has been applied for diverse NLP purposes [27]. The BERT method can be modified for special use cases like text summarization and meaning inference. The attention mechanism allows for the transformers, which are the basis of BERT, to shift focus between the segments of the input. With the use of this technique, the understanding of the sentence is improved, due to the meaning of words it provides. This process can be applied in parallel over multiple parts of sentences increasing the speed of data processing and overall efficiency. Furthermore, the natural language is processed bidirectionally as stated in the name of the technique. The benefit of such a mechanism is the context of the word, as it analyzes the words that come before the analyzed word, as well as those that come after it.

The masked language model (MLM) is used for the training of BERT. The goal of such training is to determine a hidden word only from its context. This results in the model learning the structure of the language and its patterns. Training with large datasets can prepare BERT models for specialized tasks due to their transfer learning functionality. The application of BERT in software detection is not explored. The BERT has proven a reliable and efficient technique across many different NLP use cases where there are many more factors to account for than with programming languages. The latter are more strict in their rules of writing and are more prone to repeating patterns which reduces the complexity of the text analysis. A promising text mining technique with roots in statistical text analysis is term frequency inverse document frequency (TF-IDF) . As the name implies it is comprised of two components. Term frequency is determined as the ration between a term occurring in a document versus the total number of terms in said document. Mathematically it can be determined as Eq 1:

$$\mathrm{TF}(t,d)={\displaystyle \frac{\mathrm{Number}\mathrm{of}\mathrm{times}\mathrm{term}t\mathrm{appears}\mathrm{in}\mathrm{document}d}{\mathrm{Total}\mathrm{number}\mathrm{of}\mathrm{terms}\mathrm{in}\mathrm{document}d}}$$

(1)

The second term in TF-IDF defines the inverse document frequency. The role of this factor is to emphasize important works, while decreasing the importance of filler and stop words that comprise the grammatical structure of a language. It can be determined as per Eq 2:

$$\mathrm{IDF}(t,D)=log\left({\displaystyle \frac{N}{\left|\right\{d\in D:t\in d\left\}\right|}}\right)$$

(2)

Combining TF and IDF (TF-IDF) calculates the importance of terms within a document in relation to the entire corpus as per Eq. 3.

$$\mathrm{TF}-\mathrm{IDF}(t,d,D)=\mathrm{TF}(t,d)\times \mathrm{IDF}(t,D)$$

(3)

2.2. CNN

The CNNs are recognized in the deep learning field for their high performance and versatility [28,29]. The multi-layered visual cortex of the animal brain served as the main inspiration for this technique. The information is moved between the layers where the input of the next layer is the output of the previous layer. The information gets filtered and processed in this manner. The complexity of the data is decreased through each layer while the ability of the model to detect finer details increases. The architecture of the CNN model consists of the convolutional, pooling, and fully connected layers. Filters that are most commonly used are 3×3, 5×5, and 7×7.

To achieve the highest performance CNNs require hyperparameter optimization. The most commonly tuned parameters based on their impact on performance are the number of kernels and kernel size, learning rate, batch size, the amount of each type of layer used, weight regularization, activation function, and dropout rate. s, the activation function, the dropout rate, and so on are some examples of hyperparameters. This process is considered NP-hard, however, the metaheuristic methods have proven to yield results when applied as optimizers for CNN hyperparameter tuning [14].

The convolutional function provides the input vector described in Equation (4).

$$z_{i,j,k}^{\left[l\right]}=w_k^{\left[l\right]}{x}_{i,j}^{\left[l\right]}+b_k^{\left[l\right]},$$

(4)

where the output of the k-th feature at position $i,j$ and layer l is given as $z_{i,j,k}^{\left[l\right]}$, the input at $i,j$ is x, the display filters are given as w, and the bias is shown as b.

The activation function follows the convolution function shown in Equation (5).

$$g_{i,j,k}^{\left[l\right]}=g\left(z_{i,j,k}^{\left[l\right]}\right)$$

(5)

where non-linear function exploiting the output is given as $g(·)$.

After the activation function, the pooling layers process the input toward resolution reduction. Different pooling functions can be used, and some of the most popular ones are average and max pooling. This behavior is described in Equation (6).

$$y_{i,j,k}^{\left[l\right]}=pooling\left(g_{i,j,k}^{\left[l\right]}\right).$$

(6)

where the y represents the result of pooling.

Finally, the classification is performed by the fully connected layers. The most common applications of these layers include the softmax layers for multi-class datasets, and the sigmoid function is applied for binary classification along gradient descent methods.

The weights and biases are adjusted in each iteration which is in the case of the CNN called epochs during which the goal is to minimize the loss function provided by Equation (7).

$$H(p,q)=-\sum _{x}p\left(x\right)ln\left(q\left(x\right)\right)$$

(7)

where the discrete variable x has two distributions defined over it, p and q.

2.3. AdaBoost

During the previous decade, ML has constantly grown as a field. As a result, a large number of algorithms have been produced due to their disproportionate contributions based on the area of application. Adaptive boosting (AdaBoost) aims to overcome this through the application of weaker algorithms as a group. The algorithm was developed by Freund and Schapire in 1995 [30]. Algorithms that are considered weak perform classification slightly better than random guessing. The AdaBoost technique applies more weak classifiers through each iteration and balances the classifier’s weight which is derived from accuracy. For errors in classification, the weights are decreased, while increases in weights are performed for good classifications.

The error of a weak classifier is calculated according to Equation 8.

$${ϵ}_t={\displaystyle \frac{{\sum }_{i=1}^N{w}_{i,t}·\mathrm{I}(h_t\left(x_i\right)\ne y_i)}{{\sum }_{i=1}^N{w}_{i,t}}}$$

(8)

where the error weight in the t-th iteration is given as ${ϵ}_t$, the number of training samples is given as N, and the weight of i-th training sample during the t-th iteration $w_{i,t}$. The $h_t\left(x_i\right)$ represents the predicted label, and $y_i$ shows the true label. The function $\mathrm{I}(·)$ provides 0 for false cases, and 1 for true cases.

After the weights have been established, the modification process for weights begins for new classifiers. To achieve accurate classification large groups of classifiers should be used. The combination of sub-models and their results represents a linear model. The weight are calculated in the ensemble as per Equation (9).

$${\alpha }_t={\displaystyle \frac{1}{2}}ln\left({\displaystyle \frac{1-{ϵ}_t}{{ϵ}_t}}\right)$$

(9)

where the ${\alpha }_t$ changes for each weak learner and represents its weight in the final model. The weights are updated according to the Equation (10).

$$w_{i,t+1}=w_{i,t}·exp\left(-{\alpha }_t·y_i·h_t\left(x_i\right)\right)$$

(10)

where $y_i$ marks the true mark of the i-th instance, $h_t\left(x_i\right)$ represents the prediction result of the weak student i-th instance in the t-th round, and $w_{i,t}$ denotes the weight, i-th instance in the t-th round.

The advantages of AdaBoost are that it reduces bias through learning from previous iterations, while the ensemble technique reduces variance and prevents overfitting. Therefore, the AdaBoost can provide robust prediction models. However, AdaBoost is sensitive to noisy data and exceptions.

2.4. XGBoost

The XGBoost method is recognized as a high-performing algorithm [31]. However, the highest performance is achieved only through hyperparameter tuning. The foundation of the algorithm is ensemble learning which exploits many weaker models. Optimization with regularization along gradient boosting significantly boosts performance. The motivation behind the technique is to manage complex input-target relationships through previously observed patterns.

The objective function of the XGBoost that combines the loss function and the regularization term is provided in Equation (11.

$$\mathrm{obj}(\Theta )=L\left(\theta \right)+\Omega (\Theta ),$$

(11)

where the $\Theta$ shows the hyperparameter set, $L(\Theta )$ provides loss function, while the $\Omega (\Theta )$ shows the regularization term used for model complexity management.

Mean square error (MSE) is used for utilization of the loss function given in Equation (12).

$$\mathrm{L}(\Theta )={\sum }_i{(y_i-\widehat{y_i})}^2,$$

(12)

where the $\widehat{y_i}$ provides the value predicted for the target for each iteration i, and the $y_i$ provides the predicted value.

The process of differentiating actual and predicted values is given in Equation (13).

$$L(\Theta )={\sum }_i[y_{i}ln(1+e^{-{\widehat{y}}_i})+(1-y_i)ln(1+e^{{\widehat{y}}_i})].$$

(13)

3. Methods

3.1. Basic PSO

The PSO algorithm was introduced in 1995 by Kennedy and Eberhart [32]. The main motivation behind the technique includes flocking with fish and birds. The individuals in the population are particles. Both discrete and continuous optimization problems have been solved successfully with this algorithm [33].

Firstly, initial velocities are given to each particle that is a member of the population. The velocities are described with three weights which define their movement toward the optimal solution. The weights are the terminal velocity, the best obtained so far, and the best-obtained direction by the neighbor.

$$\left\{\begin{array}{c}\overrightarrow{v_i}\leftarrow \overrightarrow{v_i}+\overrightarrow{U}(0,{\varphi }_1)⨂(\overrightarrow{p_i}-\overrightarrow{x_i})+\overrightarrow{U}(0,{\varphi }_2)⨂(\overrightarrow{p_g}-\overrightarrow{x_i})\hfill \\ \overrightarrow{x_i}\leftarrow \overrightarrow{x_i}+\overrightarrow{v_i},\hfill \end{array}\right.$$

(14)

where the component-wise multiplication is represented with ⨂, all the components in that range $[-V_{max},+V_{max}]$ are given as $v_i$, and vector that shows each particle through all generations uniformly distributed between 0 and ${\varphi }_i$ is given as $\overrightarrow{U}(0,{\varphi }_1)$.

Every particle is a possible solution in D-dimensional space. The Equation (15) describes the position of the solution.

$$X_i=(x_{i1},x_{i2},...,x_{iD})$$

(15)

The best-obtained position is given with Equation (16).

$$P_i=(p_{i1},p_{i2},...,p_{iD})$$

(16)

The velocities are described by Equation (17).

$$V_i=(v_{i1},v_{i2},...,v_{iD})$$

(17)

the solution that is found to be the best yet is marked as $pbest$, and the best in the group as $gbest$. Both values are used to calculate the next particle’s position. When applying the inertia weight approach the equation can be modified as provided in Equation (18).

$$v_{id}=W\ast v_{id}+c_1\ast r_1\ast (P_{id}-X_{id})+c_2\ast r_2\ast (P_{gd}-X_{id})$$

(18)

where the $v_{id}$ indicates particle velocity, $x_{id}$ shows the current position, w gives the inertia factor, relative cognitive influence is provided by $c_1$, while the social component influence is given as $c_2$, and the $r_1$ and $r_2$ are random numbers. The $pbest$ and $gbest$ are provided respectively through $p_{id}$ and $p_{gd}$.

The inertia factor is modeled by Equation (19).

$$w=w_{max}-{\displaystyle \frac{w_{max}-w_{min}}{T}}·t$$

(19)

where the initial weight is provided with $w_{max}$, the final weight as $w_{min}$, the maximum number of iterations as T, and the current iteration is t.

3.2. Modified PSO

The original PSO demonstrates admirable preference, especially for an algorithm proposed in 1995. However, given the recent leaps in optimization research, advanced techniques could be integrated in to the PSO to help improve overall performance. The drawbacks such as the lack of exploration innate to the PSO can be therefore addressed and overall performance boosted. This work explores a multi-population methods combined with principles borrowed form the genetic algorithm (GA) [34] to help boost exploration. The introduced algorithm is therefore dubbed the multi-population PSO (MPPSO).

The first stage in the MPPSO algorithm is generation a population of agents. This is where the first modification is introduced. The first 50% of agents is generated as par Eq. 20.

$$X_{i,j}=l{b}_j+\psi ·(u{b}_j-l{b}_j),$$

(20)

where $X_{i,j}$ denotes the j-th factor assigned to individual i, $l{b}_j$ and $u{b}_j$ signify parameter boundaries for j, $\psi$ is a factor used to introduced randomness selected form a uniform $[0,1]$ disturbing. The remaining 50% of the population is initialized using the quasi-reflection-based learning (QRL) [35] mechanism as per Eq. 21.

$$X_j^{qr}=\mathrm{rnd}\left({\displaystyle \frac{l{b}_j+u{b}_j}{2}},x_j\right),$$

(21)

in this equation $rnd$ is used to determine an arbitrary value form $\left[{\displaystyle \frac{l{b}_j+u{b}_j}{2}},x_j\right]$ limits. Bu incorporating QRL in to the initialization process, diversification is achieved during the initialization increasing chances of locating more promising solutions.

Once these two populations are generated, they are kept separate during the optimization taking a multi-population inspired approach. Two mechanisms inspired by the GA are used to communicated between the populations, genetic crossover and genetic mutation. Crossover is applied to combine agent parameters between two agents, creating offspring agents. Mutation is used to introduced further diversification within existing agents by randomly tweaking parameter values within constraints. These two processes are described in Figure 1

When tackling optimization problems it is essential to balance exploration and exploitation of the search space. In the modified algorithm this is done by modifying agents that participate in the reproductive simulations using crossover. Namely in the initial ${\displaystyle \frac{1}{2}}$ of iterations, two random agents for each sub-population are selected and recombined. Mutation is then applied with a mutation probability ($mp$) of $0.1$. Similarly crossover between agents is governed by the crossover probability parameter ($cp$) set to an empirically determined value of $0.1$ for this work. In the later half of iterations, the best performing agents for each sub-population. Thus exploration is encouraged in the early optimization stages, and intensification supported in the latter stages.

The pseudocode for the presented modified algorithm is provided in Algorithm 1.

Algorithm 1 MPPSO pseudocode

Initialize ${\displaystyle \frac{1}{2}}$ particle population P array using random positions and velocities on D dimensions in the search space   Apply QRL to generate remaining agents in population  Separate agents in to two equal sub populations  repeat     Evaluate the desired optimization fitness function of each particle in D variables     Compare the outcome with $pbes{t}_i$     if Current value is better than the $pbes{t}_i$ then         Set $pbes{t}_i$ to the value of current best         Set $\overrightarrow{p_i}$ to the current location $\overrightarrow{x_i}$ in D-dimensional space     end if     Assign the index of the best solution so far to the variable g     Adjust the particle’s position according to the Eq. 14    for all Agents in P        Generate random value R in range $[0,1]$         if $R>cp$ then             if t < ${\displaystyle \frac{T}{2}}$ then                 Apply crossover to random agents in sub-populations         else                 Apply crossover to best agents agents in each sub-populations          end if     end if      end for until The criterion is met  return Best performing agent as solution

3.3. Proposed Experimental Optimization Framework

The framework proposed in this work utilized a two layer approach. The first layer focuses on CNN optimization. Metaheuristics optimizes are applied to parameter selection of CNN. Data is processed and prepared accordingly. Text data is suitably encoded and CNN optimization is carried out in the first layer of the framework.

Once a suitable CNN model is generated, it is applied to the whole dataset and the final layer tapped to recover intermediate data. This intermediate data is then used as inputs for XGBoost and AdaBoost optimizes. Separate optimizations are carried for each classifier. Further details of the experimental setup are provided in the following section. A flowchart of the porpoised framework is provided in Figure 2.

4. Experimental Setup

In this work two separate simulations are carried out. The first set of simulations is conducted using the a group of 5 datasets which are KC1, JM1, CM1, KC2, and PC1. These datasets are part of NASA’s promise repository aimed at SDP. The instances in datasets represent various software models and the features depict the quality of that code. McCabe [36] and Halstead [37] metrics were applied through 22 features. The McCabe approach was applied to its methodology that emphasizes less complex code through the reduction of pathways, while Halstead uses counting techniques as the logic behind it was that the larger the code the more it is error-prone. A tow layer approach is applied to these simulations. The class balance in the utilized dataset is provided in Figure 3. The first utilized dataset is dis balanced with 22.5% of samples representing software with errors and 77.5% without.

A correlational heat map of the features is provided in Figure 4.

The second set of simulation utilized a dataset comprised of around 1000 crowd-sourced Python programming problems. The dataset is publicly available 1, and last accessed on the 01.08.2024. The problems are on the beginner level and consist of a description of the task, the solution, and 3 automated test cases. A generative model based on GPT-2 is used to generate solutions to these problems based off descriptive problems. Generated solutions that pass as well as fail at least one test case are collected and combined in to a unified dataset. The resulting dataset combines ground truth solutions with generated code that has some errors. The final combined dataset is comprised of 10000 samples and is balanced.

Both datasets are separated in to training and testing portions for simulations. An initial 70% of data is used for training and a later 30% for evaluation. Evaluations are carried out using a standard set of classification evaluation metrics including accuracy, recall, f1-score and precision. The Matthews correlation coefficient (MCC) metric is used as the objective function that can be determined as per 22. An indicator function is also tracked. The indicator function is the Cohen’s kappa which is determined as per Eq. 23. Metaheuristics are tasked with selecting optimal parameters, maximizing the MCC score.

$$\mathrm{MCC}={\displaystyle \frac{(TP\times TN)-(FP\times FN)}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}}$$

(22)

here true positives (TP) denotes samples correctly classified as positive, true negatives (TN) denotes instances correctly classified as negative. Similarly false positives (FP) and false negatives (FN) denote samples incorrectly classified as positive and negative respectively.

$$\mathrm{Cohen}’\mathrm{s}\mathrm{kappa}={\displaystyle \frac{\mathrm{Classification}-\mathrm{Classification}\mathrm{Expected}}{1-\mathrm{Classification}\mathrm{Expected}}}$$

(23)

In the first layer CNN parameters are optimized. The respective parameters and their constraints are provided in Table 1.

In the second optimization layer, intermediate outputs of the CNN are used. The final dense layer is recorded during classifications of all the samples available in the dataset. This is once again separated in to 70% for training and 30% for testing. These are then utilized to training and optimize AdaBoost and XGBoost models. Parameter ranges for AdaBoost and XGBoost are provided in Table 2 and Table 3 respectively.

Several optimizers are included in a comparative analysis against the proposed modified algorithm. These include the original PSO [32], as well as several other established optimizers such as the GA [34] and VNS [15]. Additional optimizers included in the comparison are the ABC [16], BA [17] and COLSHADE [18] optimizers. A recently proposed SCHO [19] optimizer is also explored. Each optimizer is implemented using original parameter settings suggested in the works that introduced said algorithm. Optimizes are allocated a population size of 10 agents and allowed a total of 8 iterations to locate promising outcomes within the given search range. Simulations are carried out through 30 independent executions to facilitate further analysis.

For NLP simulations, only a the second layer of the framework is utilized. Input text is encoded using TF-IDF encoding to a maximum of 1000 tokens. These are used as inputs for model training and evaluation. Optimization is carried out using the second layer of the introduced framework and a comparative analyses is conducted under identical conditions as previous simulations to demonstrate the flexibility of the proposed approach.

5. Simulation Outcomes

The following subsections present the simulation outcomes of the conducted simulation. First outcomes of the simulations with traditional dataset are conducted in a two layer approach. Secondly two simulations with NLP are presented using only the second layer of the framework.

5.1. Layer One CNN Simulations

First layer CNN optimization outcomes in terms of objective function are provided in Table 4. The introduced MPPSO demonstrates most favorable outcomes, attaining an objective metric score of 0.273280 in the best case execution, and 0.262061 in the worst. In turn score in the mean and median are also the best with 0.268824 and 0.269873 respectively. The highest stability rate is demonstrated by the SCHO optimizer, however, this optimizer does not manage to match the favorable outcomes demonstrated by other algorithms, suggesting a lack of exploration of exploitation potential.

First layer CNN optimization outcomes in terms of indicator function are provided in Table 5. The introduced MPPSO demonstrates similarly favorable results attaining the best outcomes with a score of 0.231201. Mean and median scores of 0.233446 and 0.232697 are also the best among evaluated optimizes. A high rate of stability in terms of indicator metric is demonstrated by the original PSO, however the performance of the modified version descend that of the original in all cases.

Visual comparisons in terms of optimizer stability is provided in Figure 5. While the stability of the introduced optimizer is not the highest compared to all other optimizes, algorithms with higher rates of stability attain less favorable outcomes. This suggests that VNS, BA, and COLSHADE for instance overly focus on local optima rather then looking for more favorable outcomes. This lack of exploration leads to higher stability but less favorable outcomes in this test case. The introduced optimizer also outperformed the base PSO while demonstrating higher stability, suggesting that the introduced modification has a positive influence on performance.

Convergence rate of algorithms provide valuable feedback on optimizer behaviour. Convergence rates for both the objective and indicator functions are tracked and provided in Figure 6 and Figure 7 for each optimizer. The introduced optimizer demonstrates favorable performance locating the best solution by iteration eight. Other optimizes stagnate prior to reaching favorable outcomes, suggesting that the boost in exploration incorporated in to the MPPSO helps overcome the shortcomings inherent to the original PSO. Similar conclusions can be made in terms of indicator function convergence demonstrated in Figure 7. As the indicator function was not the target of the optimization, there is a small decrease in the first iteration of the optimization. However, the best best outcome of the the comparison is once again located by the introduced optimizer by iteration 3.

Comparisons between the best preforming models optimized by each algorithm included in the comparative analysis is provided in Table 6. The introduced optimize demonstrates a clear dominance in terms of accuracy as well as precision. However, other optimizers demonstrate certain favorable characterises as well, which is to be somewhat expected and further supports the NFL theorem of optimization.

Further details for the best performing model optimized by the MPPSO algorithm are provided in the form of the ROC curve and confusion matrix provided in Figure 8 and Figure 9. A plot of the best performing model is also provided in Figure 10. Finally, to support simulation repeatability, the parameter selections made by each optimizer for their respective best performing optimized models is provided in Table 7.

5.2. Layer Two Simulations

Once a suitable CNN model is trained, the final layer is utilized as inputs for AdaBoost and XGBoost classifiers. These classifiers are themselves subjected to hyperparameter optimization using a set of metaheursitic optimizers. This process in done in the second layer of the optimization framework. The following subsections present the outcomes of these simulations.

5.2.1. AdaBoost Layer 2 Classifications

The outcomes of AdaBoost model optimization in terms of objective function are provided in Table 8. The introduced MPPSO demonstrates the best outcomes in the worst, mean and median scenarios, attaining scores of 0.276185, 0.279771 and 0.280600 respectively. In terms of the best performing model, the MPPSO and original PSO managed to attain models that match performance. The highest stability rate is demonstrated by the SCHO, however, this optimizer does not manage to match the favorable outcomes demonstrated by other algorithms, suggesting a lack of exploration of exploitation potential.

Second layer AdaBoost optimization outcomes in terms of indicator function are provided in Table 9. Interestingly, in terms of indicator function outcomes the BA demonstrates Superior performance to all tested algorithms. However, since this was not the goal and metric of the optimisation process these findings are quite interesting and further support the NFL theorem. High stability rates are demonstrated by the SCHO algorithm.

Visual comparisons in terms of optimizer stability are provided in Figure 11. Two interesting spikes in objective function distributions can be observed in the introduced optimizer outcomes. Interestingly, other optimizer overly focus on the first (lower) region. However, the introduced optimizer overcomes these limitations and locates a more promising region. The BA locates a promising region within the aerospace. However, the BA stability greatly sufferers providing by far the lowest stability compared to other evaluated optimizers leading to overall mixed results.

Convergence rates for both the objective and indicator functions for L2 adaboost optimizations are tracked and provided in Figure 12 and Figure 13 for each optimizer. The porpoised MPPSO shows a favorable rate of convergence, locating a good solution by iterations 7, and further improving on this solution in iteration 19. While other optimizers manage to find decent outcomes, a fast rate of convergence limits the quality of the located solution. Similar conclusions can be made in terms of indicator function convergence demonstrated in Figure 13 where the optimizer once again locates the pest solution near the end of the optimization in iteration 19.

Comparisons between the best preforming AdaBoost L2 models optimized by each algorithm included in the comparative analysis is provided in Table 10. The best performing models match performance across several metrics suggesting that several optimizers are equally well suited for AdaBoost optimization demonstrating a favorable accuracy of 0.772166 outperforming scored attained by using just CNN in L1.

Further details for the best performing adaboost L2 model optimized by the MPPSO algorithm are provided in the form of the ROC curve and confusion matrix provided in Figure 14 and Figure 15. Finally, to support simulation repeatability, the parameter selections made by each optimizer for their respective best performing optimized models is provided in Table 11.

5.2.2. XGBoost Layer 2 Classifications

The outcomes of XGBoost L2 model optimization in terms of objective function are provided in Table 13. The introduced MPPSO demonstrates the best outcomes in the best case simulation with an objective function score of 0.283956. Interestingly in terms of worst, mean and median fictions the VNS algorithm demonstrates the most favorable outcomes. In turn this algorithm also demonstrates a high rate of stability in the conducted simulations on this challenge.

Second layer XGBoost optimization outcomes in terms of indicator function are provided in Table 13. The introduced optimizer does not attain the best outcomes in terms of indicator function results, as the ABC algorithm attains the most favorable results. These interesting findings further support the NFL theorem indicating that no single approach is equally suited to all challenges and across all metrics. In terms of indicator function stability, the PSO attained the best outcomes.

Visual comparisons in terms of optimizer stability is provided in Figure 16. While the introduced optimizer attains more favorable outcomes in terms of objective function, an evident disadvantage can be observed in comparison to the original PSO, with the PSO algorithm favoring a better location within the search space in more cases. An overall low stability can be observed for the BA algorithm, while the ABC algorithm focuses of a sub optimal search space in many solution instances.

Convergence rates for both the objective and indicator functions for L2 XGBoost optimizations are tracked and provided in Figure 17 and Figure 18 for each optimizer. The introduced optimize manages to find a promising region in the 19 iteration, surpassing solutions located by other algorithms. However, this improvement comes at a cost of indicator function outcomes, where the performance is slightly reduced. Often times the trade off in terms of one metric can mean improvements in another. As the indicator function was not the primary goal of the optimisation the MPPSO algorithm attained favorable optimization performance overall.

Comparisons between the best preforming XGBoost L2 models optimized by each algorithm included in the comparative analysis is provided in Table 14. The favorable performance of the VNS algorithm is undeniable for this challenge further supporting the NFL theorem of optimization. Nevertheless, constant experimentation is needed to determined suitable optimizers for any given problem. Furthermore, it is important to consider all the metrics when determining which algorithm is the best suited to the demands of a given optimization challenge.

Further details for the best performing XGBoost L2 model optimized by the MPPSO algorithm are provided in the form of the ROC curve and confusion matrix provided in Figure 19 and Figure 20. Finally, to support simulation repeatability, the parameter selections made by each optimizer for their respective best performing optimized models is provided in Table 23.

5.3. Simulations with NLP

The second layer of the framework is also utilized in conjunction with text mining techniques to handle NLP. Encoding is handled using TF-IDF vectorization to a limit of 1000 features. Vectorized values are then used as inputs to the second layer of the framework where AdaBoost and XGBoost are used to handled classifications. Optimization is carried out on these classification models to improve overall performance though parameter selection.

5.3.1. AdaBoost NLP Classifications

The outcomes of AdaBoost NLP classifier model optimization in terms of objective function are provided in Table 16. The introduced MPPSO demonstrates the best outcomes in the best case simulation with an objective function score of 0.959947 alongside the BA optimizer. In terms of stability as well as the works, and mean outcomes the original PSO demonstrates favorable performance, while the BA attain the best results for the median performance. The PSO also demonstrates a high rate of stability in terms of objective scores.

AdaBoost NLP classifier optimization outcomes in terms of indicator function are provided in Table 17. High stability rates are showcased by the PSO and favorable outcomes by the VNS algorithm, attaining the best overall indicator score, as well as the ABC that demonstrates the best outcomes in the worst case execution.

Visual comparisons in terms of AdaBoost NLP optimizer stability is provided in Figure 21. High stability rates are demonstrated by the PSO, however, the best scores are still located by the modified optimizer suggesting that modification hold further potential. A trade off is always present when tackling multiple optimizations problems it is essential to explore multiple potential optimizes in order to determine a suitable approach, as stated by the NFL theorem.

Convergence rates for both the objective and indicator functions for NLP AdaBoost optimizations are tracked and provided in Figure 22 and Figure 23 for each optimizer. While all optimizes show a favorable convergence rate several optimizers dwell in sub optimal regions within the search space. The introduced optimizer overcomes the local minimum issue and locates a promising solution in the 18 iteration of the optimisation. Similar operations can be made in terms of indicator functions with the best solution located by the optimizer in the latter iterations.

Comparisons between the best preforming AdaBoost NLP classifier models optimized by each algorithm included in the comparative analysis is provided in Table 18. Favorable outcomes can be observed for all evaluated models suggesting that metaheuristic optimizers, as well as the AdaBoost classifier are well suited to problem of error detection through applied NLP.

Further details for the best performing AdaBoost NLP classifier model optimized by the MPPSO algorithm are provided in the form of the ROC curve and confusion matrix provided in Figure 24 and Figure 25. Finally, to support simulation repeatability, the parameter selections made by each optimizer for their respective best performing optimized models is provided in Table 23.

5.3.2. XGBoost NLP Classifications

The outcomes of XGBoost NLP classifier model optimization in terms of objective function are provided in Table 20. The introduced optimizer demonstrates the most favorable outcomes for the best case scenario. Nevertheless, the original PSO demonstrates overall favorable scores across all other test scenarios. Accordingly, the overall stability of the PSO is also equally favorable, with the highest rates of stability in compassion to other optimizes.

XGBoost NLP classifier optimization outcomes in terms of indicator function are provided in Table 21. High stability rates are demonstrated by the PSO, while the ABC algorithm demonstrates the best scores across all other metrics for the indicator function.

Visual comparisons in terms ofXGBoost NLP optimizer stability is provided in Figure 26. While the best scores in the best case scenario are showcased by the introduced optimizer, PSO outcomes are also favorable, with many solutions overcoming local optima and locating more promising outcomes overall.

Convergence rates for both the objective and indicator functions for NLP XGBoost optimizations are tracked and provided in Figure 27 and Figure 28 for each optimizer. The introduced optimizer overcomes local minimum traps, locating a favorable outcomes in iteration 18, suggesting that the boost in exploration especially in later stages helps overall performance improve, while the baseline PSO sticks to a less favorable regions. These outcomes are mirrored in terms of indicator function with the best solution determined by the introduced optimiser in the 18 iteration.

Comparisons between the best preforming XGBoost NLP classifier models optimized by each algorithm included in the comparative analysis is provided in Table 22. The best performance is demonstrated by the introduced optimizer across many metrics attaining an accuracy of 0.983893. The GA also demonstrated good performance in terms of precision for non error and recall for errors in code further affirming the NFL theorem.

Further details for the best performing XGBoost NLP classifier model optimized by the MPPSO algorithm are provided in the form of the ROC curve and confusion matrix provided in Figure 29 and Figure 30. Finally, to support simulation repeatability, the parameter selections made by each optimizer for their respective best performing optimized models is provided in Table 15.

5.4. Statistical Validation

When comparison algorithms that rely on randomness, such as metaheuristics, it is essential to verify the statistical significance of findings as a single simulation may not be representative of overall performance. With this in mind, simulations in this work are carried out in 30 individually independent runs and data collected for further analysis.

Two types of statistical test can be applied to validate statistical significance of outcomes, parametric and non-parametric test. The first step, therefore, involves validating that the safe used of parametric tests is warranted or if non-parametric analysis should be applied. The used of parametric tests is justified if conditions of independence, normality, and homoscedasticity [38] are met. As all simulations are conducted using independent random seeds the independence criteria is met. Homoscedasticity is confirmed via Levene’s test [39] which resulted in a $p-values$ of 0.68 for each case, indicating that this condition is also met.

The last conduction, normality, is assessed using the Shapiro-Wilk’s test [40], with $p-values$ computed for each method included in the comparative analysis. As all the values are bellow a threshold of $0.05$ the null hypothesis (H0) may be rejected, suggesting outcomes do not originate form normal distributions. These findings are further enforced by the objective function KDE diagrams presented in Figure 31. The outcomes of the Shapiro-Wilk test are provided in Table 24.

As one of the criteria needed to justify the safe use of parametric tests is not met further evaluations are conducted via non-parametric tests. The Wilcoxon signed-rank test is therefore applied to compare the performance of the MPPSO to other algorithms included in the comparative analysis. Test scores are presented in Table 25. A threshold of $\alpha =0.05$ is not exceeded in any of the test cases indicating that the outcomes attained by the MPPSO algorithm hold statistically significant improvements over other evaluated methods.

5.5. Best Model Interpretations

Model interpretation plays an increasingly important role in modern IA research. Often times understanding the factors and their degree of influence on decision can help better understand the problem at hand, as well as detect any unintended biases in data. There are several emerging techniques for interpreting models, and for simpler models these interpretations are fairly straight-forward. However, more complicated models, and models that utilize a multi-tier framework can be more difficult to subject to interpretation.

This work utilizes SHAP analysis in order to determined feature importance and feature impact on optimized model decisions. SHAP analysis utilizes an approach rooted in game theory treating features as contestants in a cooperative competition. The contributions of each features as well as their collaborative contributions are considered when determining importance. Importance can also be analyzed on a global and local level making SHAP a versatile and promising approach for model analysis.

Feature importances for the best performing models constructed in L1 and L2 simulations are provided in Figure 32. Each constructed classifier places a slightly different importance of each feature. However, I, Uniq_OP, IOBland and d are universally highly ranked. The best CNN adn AdaModels show the highest impact for the I feature while this feature is second best for the XGBoost classifier, with the highest importance placed on uniq_OP feature.

Feature importances for the best performing models in NLP simulations are likewise provided in Figure 33 for the AdaBoost model and in Figure 34 for the XGBoost model. TF-IDF vectorized feature importances are once again interpreted using SHAP analysis and the top 20 contributing features shown for both models. Both models place a high importance on the def keyword, as well as the print and write keywords where errors can often occur. The AdaBoost model places a significant importance of the def keyboard, with print holding the second spot with a significantly lower importance. The XGBoost model however has a more even distribution of importance. While def holds a high importance the second highest impact is on the write keyword followed by print with a slightly decreased importance.

Understanding feature importance on model decisions can further aid in understanding the challenges associated with software defect detection. Furthermore, determining what features play and important role in these classifications can further aid in reducing computational costs of models in deployment as well as aid in improving data collection in the future. Detecting hidden model biases is also essential for enforcing trust in model decisions improving the generalizability and objectivity of decisions.

6. Conclusion

Software has become increasingly integral to societal infrastructure, governing critical systems. Ensuring the reliability of software is therefore paramount across various industries. As the demand for accelerated development intensifies, the manual review of code presents growing challenges, with testing frequently consuming more time than development. A promising method for detecting defects at the source code level involves the integration of AI with NLP. Given that software is composed of human-readable code, which directs machine operations, the validation of the highly diverse machine code on a case-by-case basis is inherently complex. Consequently, source code analysis offers a potentially effective approach to improving defect detection and preventing errors.

This work explores the advantages and challenges of utilizing AI for error detection in software code. Both classical and NLP methods are explored on two publicly available dataset with 5 experiments total conducted. A two layer optimization framework is introduced in order to manage the complex demands of error detection. A CNN architecture is utilized in the first layer to help process the large amounts of data in a more computational efficient manner, with the second layer handling intermediate results using AdaBoost and XGBoost classifiers. An additional set of simulations using only the second layer of the framework in combination with TF-IDF encoding is also carried out in order to provide a comparison between emerging NLP and classical techniques. As optimizer performance is highly dependant on adequate parameter selection, a modified version of the well established PSO is introduced, designed specifically for the needs of this research and with an aim to overcome some of the known drawback of the original algorithm. A comparative analysis is carried out with several state of the art optimizes with the introduced approach demonstrative promising outcomes in several simulations.

Twin layer simulations improve upon the baseline outcomes demonstrated by CNN boost accuracy form 0.768799 to 0.772166 for the AdaBoost models and 0.771044 for the XGBoost best performing classifier. This suggests that a two layer approach can yield favorable outcomes while maintaining favorable computational demands in comparison to more complex network solutions. Optimization carried out using NLP demonstrate an impressive accuracy of 0.979781 for the best performing AdaBoost model and a 0.983893 for the best performing XGBoost model. Simulations are further validated using statistical evaluation to confirm the significance of the observations. The best preforming models are also subjected to SHAP analysis to determine feature importance and help locate any potential hidden biases within the best performing models.

It is worth noting that the extensive computational demands of the optimizations carried out in this work limit the extend of optimizers that can be tested. Further limitations are associated with population sized and allocated numbers of iterations for each optimization due to hardware memory limitations. Future works hope to address these concerns as additional resources become available. Additional implementations of the proposed MPPSO hope to be explored for other implementations. Emerging transformer based architectures based on custom BERT encoding also hope to be explored for software defect detection in future works.