Eigenvalue Distributions in Random Confusion Matrices: Applications to Machine Learning Evaluation

1. Introduction

The distribution of eigenvalues of a confusion matrix is an interesting and important concept in machine learning (ML), particularly in the evaluation of classification models [1]. Confusion matrices are widely used to assess the performance of a classification algorithm by providing a detailed breakdown of the predicted and actual class labels [2]. The eigenvalues of a confusion matrix offer insights into the underlying structure and characteristics of the classification results [3,4,5]. Eigenvalues are a mathematical concept used to analyze linear transformations, and in the context of confusion matrices, they can reveal information about the matrix’s behaviour [6,7]. The distribution of eigenvalues provides a quantitative measure of the spread and concentration of information in the matrix. Understanding the distribution of eigenvalues of a confusion matrix can be valuable for various purposes, including model assessment, variable selection, high-dimensional analysis, dimension reduction, model comparison, anomaly detection, and generalization or overfitting issues [1,7,8,9,10,11].

For example, in Ref. [1], the significance of eigenvalue analysis for selecting important features in big data was explored. The authors emphasize the importance of understanding the patterns of eigenvalues in covariance matrices for various analytical purposes, such as model comparison and anomaly detection. They highlight how eigenvalues provide insights into the underlying structure of classification results, contributing to an overall understanding of model performance. In a similar study by [7], the authors utilized eigenvalue analysis in conjunction with principal component analysis (PCA) methods to reduce the dimensionality of big data before exploring the performances of several classification methods. The results of their analysis revealed that the outcomes from eigenvalue and PCA are much superior to those from the linear discriminant analysis (LDA) procedure.

In another study by [11], various eigenvalue-based dimension reduction techniques were compared for high-dimensional analysis. Specifically, the authors investigated the performances of PCA, LDA, and singular value decomposition (SVD). The findings from the study validate the utility of eigenvalue-based dimension reduction techniques in handling high-dimensional data. By comparing the effectiveness of PCA, LDA, and SVD, the research underscores the importance of eigenvalue analysis in addressing the challenges posed by high-dimensional datasets. Moreover, [12] utilized eigenvalue analysis to tackle the generalization error problem in two-layered neural networks for high-dimensional analysis. By leveraging eigenvalue properties, the study aimed to enhance the understanding of how neural networks generalize from training data to unseen data. Eigenvalue analysis in this context provides valuable insights into the behavior and performance of neural networks, particularly in high-dimensional spaces. Ref. [12] approach highlights the significance of incorporating eigenvalue-based techniques in optimizing and refining machine learning models for complex data analysis tasks.

In a different context within high-dimensional analysis, Ref. [10] employed eigenvalue and eigenvector analyses to improve the performance of high-dimensional LDA classifier in spiked covariance model. The author introduced a modified regularized R-LDA that is based on eigenvalue and eigenvector analyses. Numerical simulations, using both real and simulated data, revealed that the proposed classifier yields better classification performance than the classical R-LDA while requiring lower computational complexity. In a similar context, Ref. [9] increased the performance of support vector machine (SVM) by employing eigenvalue analysis of the features covariance matrices and subsequently performing PCA to reduce the dimension of the features. This approach helps to increase the prediction accuracy of hepatitis disease.

In a Bayesian analysis of confusion matrices, Ref. [13] delve into Bayesian methods for analyzing confusion matrices in machine learning. While Bayesian approaches are widely used in various aspects of ML, their application to confusion matrices provides a unique perspective on model evaluation. Ref. [13] provided Bayesian interpretations of various evaluation metrics derived from confusion matrices of machine learning models. The authors presented posterior distributions for these metrics from the confusion matrix and used them to compare the performances of several ML models.

The findings of various studies reviewed indicate a significant body of work on eigenvalue analysis within the context of dimension reduction, particularly in Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA). However, there is a notable gap in research concerning eigenvalue analysis of confusion matrices arising from machine learning (ML) models. In high-dimensional analysis and variable selection, dimension reduction serves as a filtering mechanism wherein techniques like eigenvalue analysis are employed to select important variables before training a classification model. Many authors, including [2,14,15,16,17,18,19,20,21], among others, have criticized this approach. They argue that it eliminates the possibility of interaction effects present in variables. Therefore, embedded and wrapper variable selection methods, which combine selection techniques and ML models, are preferred. In this regard, comparing ML models based on confusion matrices from trained model will be beneficial than covariance matrix of a pre-trained model.

Moreover, by leveraging eigenvalue analysis, researchers can objectively compare different machine learning models, discerning their relative strengths and weaknesses based on the underlying structure of their confusion matrices [22,23,24,25]. Hence, this paper presents the distribution of eigenvalues for a $2\times 2$ random confusion matrix arising from a machine learning evaluation scenario. Furthermore, we provide distributions for both the matrix’s trace and the difference between the traces of two random confusion matrices. We also demonstrate how these distributions can be utilized to compute the superiority probability of ML models.

2. Distribution of Eigenvalues of Random Confusion Matrix

Suppose we have a learning problem given by data $\mathcal{D}=\{X_i,Y_i\}$, where $i\in 1,2,\cdots ,n$, $X_i$ is the matrix of the features, and $Y_i$ is the response vector which we assume to be categorical with k classes. For simplicity, we consider the binary case with $k=2$ as the derivation in this paper can be easily generalized to the multiclass k classes. In any binary classification problem, the goal is to predict the $Y_i$ based on new information $x\in \mathcal{X}$ using a classifier $\widehat{y}:f\left(x\right)$. Consider a testing dataset denoted as $T={\left\{(X_i,Y_i)\right\}}_{i=1}^{n_T}$, comprising $n_T$ independent samples drawn from an unknown distribution $F(X,Y)$. To assess the accuracy of predictions made by $\widehat{y}$ on the samples in T, we introduce a loss function $\mathcal{L}:y\times \widehat{y}\to \{a,b,c,d\}$. Let y belong to $\{{\theta }_0,{\theta }_1\}$ as the true class, and $\widehat{y}$ belong to $\{{\theta }_0,{\theta }_1\}$ as the predicted class. Following convention in [13], we define the mapping of the $\mathcal{L}$ function as follows:

$$\mathcal{L}=\left\{\begin{array}{cc}a,\hfill & \mathrm{if}y={\theta }_1\mathrm{and}\widehat{y}={\theta }_1\hfill \\ b,\hfill & \mathrm{if}y={\theta }_1\mathrm{and}\widehat{y}={\theta }_0\hfill \\ c,\hfill & \mathrm{if}y={\theta }_0\mathrm{and}\widehat{y}={\theta }_1\hfill \\ d,\hfill & \mathrm{if}y={\theta }_0\mathrm{and}\widehat{y}={\theta }_0\hfill \end{array}\right.$$

(1)

where a denotes true positive, b denotes false negative, c denotes false positive and d denotes true negative. The elements of vector $\mathcal{L}$ can be presented in a $2\times 2$ matrix often referred to as a confusion matrix. Let $\mathit{A}$ represents the $2\times 2$ confusion matrix obtained from a classification learning problem defined above; $\mathit{A}$ can be defined as:

$$\mathit{A}=\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]$$

(2)

The obvious properties of $\mathit{A}$ are that (a.) it is not symmetric (b.) it is square and it is also random. Now, if we assume $\mathit{A}$ is diagonalizable such that there exist a scalar $\lambda$ and vector $\mathcal{V}$ that we can use to decompose $\mathit{A}$ using

$$\mathit{A}\mathcal{V}=\lambda \mathcal{V}$$

(3)

then $\lambda =\{{\lambda }_1,{\lambda }_2\}$ and $\mathcal{V}=\left[\begin{array}{cc}{v}_{11}& v_{12}\\ v_{13}& v_{13}\end{array}\right]$ are the eigenvalues and eigenvectors of $\mathit{A}$ respectively. One interesting property of eigenvalues of this type of diagonalizable square matrix is that the sum of the eigenvalues equals the trace of the matrix. That is;

$$tr\left(\mathit{A}\right)=\sum _{j=1}^2{\lambda }_j.$$

(4)

The $tr\left(\mathit{A}\right)$ is very useful in evaluating the accuracy of a classifier in a machine learning problem most especially if the categories of the response variable is balance that is $p_k=1/k$. In a balance binary classification problem with $n_T$ test cases, the accuracy $\left(\widehat{\varphi }\right)$ of a classifier can be computed using:

$$\begin{array}{cc}\hfill \widehat{\varphi }& =n_T^{-1}tr\left(\mathit{A}\right)\hfill \\ \hfill & =n_T^{-1}\sum _{j=1}^2{\lambda }_j.\hfill \end{array}$$

(5)

Note that, since elements of $\mathit{A}$ are resultants of random outcomes of randomly generated test instances used to validate classifier $\widehat{y}$, $\mathit{A}$ can be regarded as a random matrix. Also, since only $n_T$ is the only known parameter, the elements of $\mathit{A}$ can be assumed to be multinomially distributed with parameters $n_T,{\pi }_a,{\pi }_b,{\pi }_c$ and ${\pi }_d$. Thus, the joint density function of the elements in the random matrix $\mathit{A}$ can be given as:

$$P(a,b,c,d|n_T,{\pi }_a,{\pi }_b,{\pi }_c,{\pi }_d)=\frac{n_T!}{a!·b!·c!·d!·}{a}^{{\pi }_a}·b^{{\pi }_b}·c^{{\pi }_c}·d^{{\pi }_d}·I(a+b+c+d=n_T).$$

(6)

The last part of the RHS of (6) implies that it is required that the total sum of the four cells be equal to the number of test instances for it to be a proper pdf.

Theorem 1.

The joint probability density function (pdf) of the eigenvalues $({\lambda }_1,{\lambda }_2)$ of a $2\times 2$ confusion matrix is given by:

$$f({\lambda }_1,{\lambda }_2)=\frac{1}{4s^2\sqrt{\pi }}{e}^{-\frac{1}{2s^2}({\lambda }_1^2+{\lambda }_2^2-2\overline{\mathit{A}}({\lambda }_1+{\lambda }_2)+2{\overline{\mathit{A}}}^2)}|{\lambda }_1-{\lambda }_2|-\infty \le {\lambda }_1,{\lambda }_2\le \infty .$$

(7)

Proof. 

We begin this proof by standardizing the element of the confusion matrix $\mathit{A}$ as follows:

$$\mathbf{z}={\mathbf{s}}^{-1}(\mathit{A}-\overline{\mathit{A}}),$$

(8)

where $\overline{\mathit{A}}$ is the mean of all elements in $\mathit{A}$ and $\mathbf{s}$ is the standard deviation of each element from their mean. If the confusion matrix is balanced such that $p_k=1/4$ for all four elements, the mean $\overline{\mathit{A}}$ and standard deviation $\mathbf{s}$ are $\frac{n}{4}$ and $\sqrt{\frac{3n}{16}}$ respectively. If otherwise, the mean $\overline{\mathit{A}}$ and standard deviation $\mathbf{s}$ are computed as follows

$$\begin{array}{cc}\hfill \overline{\mathit{A}}& =\frac{a+b+c+d}{4},\hfill \\ \hfill \mathbf{s}& =\sqrt{\frac{{(a-\overline{\mathit{A}})}^2+{(b-\overline{\mathit{A}})}^2+{(c-\overline{\mathit{A}})}^2+{(d-\overline{\mathit{A}})}^2}{3}}.\hfill \end{array}$$

(9)

The next step involves the symmetrization of $\mathbf{z}$ to achieve symmetry as expected for a Gaussian Orthogonal Ensemble $\left(GOE\right)$ [26,27].

$${\mathbf{z}}_s=(\mathbf{z}+{\mathbf{z}}^T)/2.$$

(10)

where ${\mathbf{z}}_s$ is the standardized symmetrized confusion matrix, $\mathbf{z}$ is the standardized confusion matrix and ${\mathbf{z}}^T$ is it transpose. The elements of ${\mathbf{z}}_s$ is explicitly defined as:

$${\mathbf{z}}_s=\left[\begin{array}{cc}{a}^{\prime }& b^{\prime }\\ b^{\prime }& d^{\prime }\end{array}\right].$$

(11)

Now, that we have established that ${\mathbf{z}}_s$ is a $GOE$ with joint $pdf$ of $(a^{\prime },b^{\prime },d^{\prime })$ given by

$$f(a^{\prime },b^{\prime },d^{\prime })=\frac{1}{2\pi }{e}^{-\frac{tr\left({\mathbf{z}}_s^2\right)}{2}};-\infty \le a^{\prime },b^{\prime },d^{\prime }\le \infty ,$$

(12)

we can proceed to derive the distribution of eigenvalues of ${\mathbf{z}}_s$ and subsequently the distribution of eigenvalues of $\mathit{A}$. Note that by using the change of variable rule, the distribution of eigenvalues $({\eta }_1,{\eta }_2)$ of ${\mathbf{z}}_s$ is given by

$$f({\eta }_1,{\eta }_2)=f(a^{\prime },b^{\prime },d^{\prime })\left|det\left(J\right)\right|,$$

(13)

where J is the change of variable Jacobian matrix. Thus, since ${\mathbf{z}}_s$ matrix is invariant under orthogonal transformation such that

$${\mathbf{z}}_s={\mathbf{P}}^T{\mathbf{z}}_s^{\eta }\mathbf{P}$$

(14)

where $\mathbf{P}=\left[\begin{array}{cc}cos\left(\theta \right)& sin\left(\theta \right)\\ -sin\left(\theta \right)& cos\left(\theta \right)\end{array}\right]$ is an orthogonal matrix and ${\mathbf{z}}_s^{\eta }=\left[\begin{array}{cc}{\eta }_1& 0\\ 0& {\eta }_2\end{array}\right]$ is a diagonal matrix of the eigenvalues of matrix ${\mathbf{z}}_s$, we have

$$\begin{array}{cc}\hfill \left[\begin{array}{cc}{a}^{\prime }& b^{\prime }\\ b^{\prime }& d^{\prime }\end{array}\right]& =\left[\begin{array}{cc}cos\left(\theta \right)& -sin\left(\theta \right)\\ sin\left(\theta \right)& cos\left(\theta \right)\end{array}\right]\left[\begin{array}{cc}{\eta }_1& 0\\ 0& {\eta }_2\end{array}\right]\left[\begin{array}{cc}cos\left(\theta \right)& sin\left(\theta \right)\\ -sin\left(\theta \right)& cos\left(\theta \right)\end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{cc}{\eta }_1{cos}^2\left(\theta \right)+{\eta }_2{sin}^2\left(\theta \right)& ({\eta }_1-{\eta }_2)sin\left(\theta \right)\\ ({\eta }_1-{\eta }_2)sin\left(\theta \right)& {\eta }_1{sin}^2\left(\theta \right)+{\eta }_2{cos}^2\left(\theta \right)\end{array}\right].\hfill \end{array}$$

(15)

As we’re moving from ${\mathbf{z}}_s$ to ${\mathbf{z}}_s^{\eta }$, it is required to normalized the resultant $pdf$ of $({\eta }_1,{\eta }_2)$ using the Jacobian determinant $det\left(J\right)$. The change of variable Jacobian J is given as

$$\begin{array}{cc}\hfill J& =\left[\begin{array}{ccc}\frac{\partial a^{\prime }}{\partial {\eta }_1}& \frac{\partial a^{\prime }}{\partial {\eta }_2}& \frac{\partial a^{\prime }}{\partial \theta }\\ \frac{\partial d^{\prime }}{\partial {\eta }_1}& \frac{\partial d^{\prime }}{\partial {\eta }_2}& \frac{\partial d^{\prime }}{\partial \theta }\\ \frac{\partial b^{\prime }}{\partial {\eta }_1}& \frac{\partial b^{\prime }}{\partial {\eta }_2}& \frac{\partial b^{\prime }}{\partial \theta }\end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{ccc}{cos}^2\left(\theta \right)& {sin}^2\left(\theta \right)& ({\eta }_2-{\eta }_1)sin\left(2\theta \right)\\ {sin}^2\left(\theta \right)& {cos}^2\left(\theta \right)& ({\eta }_1-{\eta }_2)sin\left(2\theta \right)\\ \frac{1}{2}sin\left(2\theta \right)& -\frac{1}{2}sin\left(2\theta \right)& ({\eta }_1-{\eta }_2)cos\left(2\theta \right)\end{array}\right].\hfill \end{array}$$

(16)

Subsequently, the determinant of the Jacobian is given by

$$\begin{array}{cc}\hfill det\left(J\right)& =det\left[\begin{array}{ccc}{cos}^2\left(\theta \right)& {sin}^2\left(\theta \right)& ({\eta }_2-{\eta }_1)sin\left(2\theta \right)\\ {sin}^2\left(\theta \right)& {cos}^2\left(\theta \right)& ({\eta }_1-{\eta }_2)sin\left(2\theta \right)\\ \frac{1}{2}sin\left(2\theta \right)& -\frac{1}{2}sin\left(2\theta \right)& ({\eta }_1-{\eta }_2)cos\left(2\theta \right)\end{array}\right]\hfill \\ \hfill & ={cos}^2\left(\theta \right)\left[\begin{array}{cc}{cos}^2\left(\theta \right)& ({\eta }_1-{\eta }_2)sin\left(2\theta \right)\\ -\frac{1}{2}sin\left(2\theta \right)& ({\eta }_1-{\eta }_2)cos\left(2\theta \right)\end{array}\right]\hfill \\ & -{sin}^2\left(\theta \right)\left[\begin{array}{cc}{sin}^2\left(\theta \right)& ({\eta }_1-{\eta }_2)sin\left(2\theta \right)\\ \frac{1}{2}sin\left(2\theta \right)& ({\eta }_1-{\eta }_2)cos\left(2\theta \right)\end{array}\right]\hfill \\ & +({\eta }_2-{\eta }_1)sin\left(2\theta \right)\left[\begin{array}{cc}{sin}^2\left(\theta \right)& {cos}^2\left(\theta \right)\\ \frac{1}{2}sin\left(2\theta \right)& -\frac{1}{2}sin\left(2\theta \right)\end{array}\right]\hfill \\ \hfill & =({\eta }_1-{\eta }_2)({cos}^2\left(2\theta \right)+{sin}^2\left(2\theta \right))\hfill \\ \hfill & ={\eta }_1-{\eta }_2.\hfill \end{array}$$

(17)

Therefore, the corresponding joint $pdf$ of $({\eta }_1,{\eta }_2)$ for matrix ${\mathbf{z}}_s$ is given by

$$f({\eta }_1,{\eta }_2)=\frac{1}{4\sqrt{\pi }}{e}^{-\frac{1}{2}({\eta }_1^2+{\eta }_2^2)}|{\eta }_1-{\eta }_2|;-\infty \le {\eta }_1,{\eta }_2\le \infty .$$

(18)

Now that we have the distribution of the eigenvalues for the transformed matrix ${\mathbf{z}}_s$, we can obtain the distribution of eigenvalues for the required confusion matrix $\mathit{A}$ as follows

$$\mathit{A}=\mathbf{s}{\mathbf{z}}_s+\overline{\mathit{A}}.$$

(19)

From (19), it can be seen that there is a one-one correspondence between matrices $\mathit{A}$ and ${\mathbf{z}}_s$, thus, we can define the eigenvalues of $\mathit{A}$ as a function of eigenvalues of ${\mathbf{z}}_s$. This implies

$$\mathbf{\lambda }=\mathbf{s}\mathbf{\eta }+\overline{\mathit{A}}.$$

(20)

where $\mathbf{\lambda }=({\lambda }_1,{\lambda }_2)$ and $\mathbf{\eta }=({\eta }_1,{\eta }_2)$. Therefore, the joint $pdf$ of eigenvalues of $\mathit{A}$ is given by

$$\begin{array}{cc}\hfill f({\lambda }_1,{\lambda }_2)& =f_{{\eta }_1,{\eta }_2}({\lambda }_1,{\lambda }_2)\left|\frac{d\mathbf{\eta }}{d\mathbf{\lambda }}\right|\hfill \\ \hfill & =\frac{1}{4\sqrt{\pi }}{e}^{-\frac{1}{2s^2}({\lambda }_1^2+{\lambda }_2^2-2\overline{\mathit{A}}({\lambda }_1+{\lambda }_2)+2{\overline{\mathit{A}}}^2)}\left|\frac{{\lambda }_1-{\lambda }_2}{s}\right|\left|\frac{1}{s}\right|\hfill \\ \hfill f({\lambda }_1,{\lambda }_2)& =\frac{1}{4s^2\sqrt{\pi }}{e}^{-\frac{1}{2s^2}({\lambda }_1^2+{\lambda }_2^2-2\overline{\mathit{A}}({\lambda }_1+{\lambda }_2)+2{\overline{\mathit{A}}}^2)}|{\lambda }_1-{\lambda }_2|;-\infty \le {\lambda }_1,{\lambda }_2\le \infty ,\hfill \end{array}$$

(21)

where ${\eta }_1^2+{\eta }_2^2={\left(\frac{{\lambda }_1-\overline{\mathit{A}}}{s}\right)}^2+{\left(\frac{{\lambda }_2-\overline{\mathit{A}}}{s}\right)}^2$, ${\eta }_1-{\eta }_2=\left(\frac{{\lambda }_1-\overline{\mathit{A}}}{s}\right)-\left(\frac{{\lambda }_2-\overline{\mathit{A}}}{s}\right)$ and $\frac{d\mathbf{\eta }}{d\mathbf{\lambda }}=\frac{1}{s}$.

□

Remark 1.

Equation (21) implies $f({\lambda }_1,{\lambda }_2)$ is a shifted $GOE$ with mean and variance $\overline{\mathit{A}}$ and $s^2$ respectively.

2.1. Distribution of trace of a random confusion matrix

Theorem 2.

The probability density function (pdf) of the trace $t=tr\left(\mathit{A}\right)$ of a $2\times 2$ random confusion matrix $\mathit{A}$ is given by:

$$f\left(t\right)=\frac{1}{\sqrt{4\pi s^2}}{e}^{-\frac{1}{4s^2}{(t-2\overline{\mathit{A}})}^2};-\infty \le t\le \infty ,$$

(22)

Proof. 

Again, considering the standardized symmetrized confusion matrix ${\mathbf{z}}_s$ defined in (11). The eigenvalues $({\eta }_1,{\eta }_2)$ of ${\mathbf{z}}_s$ can be estimated from the characteristics equation

$${\eta }^2-(a^{\prime }+d^{\prime })\eta +(a^{\prime }{d}^{\prime }-b^{\prime 2})=0.$$

(23)

Solving (23), gives

$$\begin{array}{cc}\hfill {\eta }_1& =\frac{(a^{\prime }+d^{\prime })+\sqrt{{(a^{\prime }+d^{\prime })}^2-4b^{\prime 2}}}{2},\hfill \\ \hfill {\eta }_2& =\frac{(a^{\prime }+d^{\prime })-\sqrt{{(a^{\prime }+d^{\prime })}^2-4b^{\prime 2}}}{2}.\hfill \end{array}$$

(24)

Recall that, the trace $\left(w\right)$ for matrix ${\mathbf{z}}_s$ is given by

$$\begin{array}{cc}\hfill tr\left({\mathbf{z}}_s\right)& ={\eta }_1+{\eta }_2\hfill \\ \hfill w& ={\eta }_1+{\eta }_2\hfill \\ \hfill & =\frac{(a^{\prime }+d^{\prime })+\sqrt{{(a^{\prime }+d^{\prime })}^2-4b^{\prime 2}}}{2}+\frac{(a^{\prime }+d^{\prime })-\sqrt{{(a^{\prime }+d^{\prime })}^2-4b^{\prime 2}}}{2}\hfill \\ \hfill w& =a^{\prime }+d^{\prime }\hfill \end{array}$$

(25)

Lemma 1.

Suppose matrix ${\mathbf{z}}_s$ is a $GOE$, thus its elements $(a^{\prime },d^{\prime })$ are independent and identically distributed as normal, $N(0,1)$, and $b^{\prime }$ is distributed normally as $N(0,1/2)$.

Remark 2.

Lemma (1) implies that the distribution of the trace of the standardized symmetrized matrix ${\mathbf{z}}_s$ is the sum of two normal distributions denoted by $N(0,2)$. Thus,

$$f\left(w\right)=\frac{1}{2\sqrt{\pi }}{e}^{-w^2/4};-\infty \le w\le \infty .$$

(26)

Again, by change of variable, we can derive the distribution of the trace of matrix $\mathit{A}$ as follows

$$\begin{array}{cc}\hfill f\left(t\right)& =f_w\left(t\right)\left|\frac{dw}{dt}\right|\hfill \\ \hfill & =\frac{1}{2\sqrt{\pi }}{e}^{-(t-2\overline{{\mathit{A})}^2/4s^2}}\left|\frac{dw}{dt}\right|\hfill \\ \hfill & =\frac{1}{2\sqrt{\pi }}{e}^{-{(t-2\overline{\mathit{A}})}^2/4s^2}\left|\frac{1}{s}\right|\hfill \\ \hfill f\left(t\right)& =\frac{1}{\sqrt{4\pi s^2}}{e}^{-{(t-2\overline{\mathit{A}})}^2/4s^2};-\infty \le t\le \infty \hfill \end{array}$$

(27)

□

Remark 3.

Equation (27) implies $f\left(t\right)$ is normally distributed with mean and variance $2\overline{\mathit{A}}$ and $2s^2$ respectively and it is denoted by $N(2\overline{\mathit{A}},2s^2)$.

Lemma 2.

The cumulative distribution function $F\left(t\right)$ for the trace of matrix $\mathit{A}$ is given by

$$\begin{array}{cc}\hfill F\left(t\right)& ={\int }_{-\infty }^{t}f\left(t\right)dt\hfill \\ \hfill & ={\int }_{-\infty }^t\frac{1}{\sqrt{4\pi s^2}}{e}^{-{(t-2\overline{\mathit{A}})}^2/4s^2}dt\hfill \\ \hfill F\left(t\right)& =\mathsf{\Phi }\left(\frac{t-2\overline{\mathit{A}}}{\sqrt{2s^2}}\right),\hfill \end{array}$$

(28)

where Φ is the $cdf$ of standardized normal distribution with mean 0 and variance 1.

Figure 1. Graphs of the $pdf$ of the trace of a random $2\times 2$ confusion matrix for different diagonal probabilities ${\pi }_1$ and ${\pi }_4$.

Figure 1. Graphs of the $pdf$ of the trace of a random $2\times 2$ confusion matrix for different diagonal probabilities ${\pi }_1$ and ${\pi }_4$.

Figure 2. Graphs of the $cdf$ of the trace of a random $2\times 2$ confusion matrix for different diagonal probabilities ${\pi }_1$ and ${\pi }_4$.

Figure 2. Graphs of the $cdf$ of the trace of a random $2\times 2$ confusion matrix for different diagonal probabilities ${\pi }_1$ and ${\pi }_4$.

2.2. Distribution of Difference of Two Traces of Random Confusion Matrices

In machine learning, it is often valuable to compare the confusion matrices of two algorithms, such as decision trees and random forests [2,14,15]. Understanding the distribution of differences is crucial because it quantifies the degree of superiority one algorithm holds over the other. Therefore, in this section, we have developed the distribution of differences between two sets of $2\times 2$ random confusion matrices.

Theorem 3.

The probability density function $\left(pdf\right)$ of the difference of two traces of $2\times 2$ random confusion matrices $\mathit{A}$ and $\mathit{B}$ denoted by $m=tr\left(\mathit{A}\right)-tr\left(\mathit{B}\right)$ is given by:

$$f\left(m\right)=\frac{1}{\sqrt{4\pi S_{\mathit{A}+\mathit{B}}^2}}{e}^{-\frac{1}{4S_{\mathit{A}+\mathit{B}}^2}{(m-2\overline{\mathit{A}}+2\overline{\mathit{B}})}^2};-\infty \le m\le \infty ,$$

(29)

where $S_{\mathit{A}+\mathit{B}}^2=S_{\mathit{A}}^2+S_{\mathit{B}}^2$.

Proof. 

This proof follows from the earlier distribution of t which follows $N(2\overline{\mathit{A}},2s^2)$.

Lemma 3.

Suppose the traces of matrices $\mathit{A}$ and $\mathit{B}$ are independently distributed normal $N(2\overline{\mathit{A}},2s^2)$, then the distribution of their difference is also normal with mean $2\overline{\mathit{A}}-2\overline{\mathit{B}}$ and variance $S_{\mathit{A}+\mathit{B}}^2=S_{\mathit{A}}^2+S_{\mathit{B}}^2$.

Remark 4.

Lemma (3) implies that the $pdf$ of the difference of two traces of $2\times 2$ random confusion matrices $\mathit{A}$ and $\mathit{B}$ is $N(2\overline{\mathit{A}}-2\overline{\mathit{B}},S_{\mathit{A}+\mathit{B}}^2)$.

Thus, the $pdf$ of the difference of two traces of $2\times 2$ random confusion matrices $\mathit{A}$ and $\mathit{B}$ denoted by $m=tr\left(\mathit{A}\right)-tr\left(\mathit{B}\right)$ is given by:

$$f\left(m\right)=\frac{1}{\sqrt{4\pi S_{\mathit{A}+\mathit{B}}^2}}{e}^{-\frac{1}{4S_{\mathit{A}+\mathit{B}}^2}{(m-2\overline{\mathit{A}}+2\overline{\mathit{B}})}^2};-\infty \le m\le \infty ,$$

(30)

□

Lemma 4.

The cumulative distribution function $F\left(m\right)$ for the difference of two traces of $2\times 2$ random confusion matrices $\mathit{A}$ and $\mathit{B}$ denoted by $m=tr\left(\mathit{A}\right)-tr\left(\mathit{B}\right)$ is given by

$$\begin{array}{cc}\hfill F\left(m\right)& ={\int }_{-\infty }^{m}f\left(m\right)dm\hfill \\ \hfill & ={\int }_{-\infty }^m\frac{1}{\sqrt{4\pi S_{\mathit{A}+\mathit{B}}^2}}{e}^{-\frac{1}{4S_{\mathit{A}+\mathit{B}}^2}{(m-2\overline{\mathit{A}}+2\overline{\mathit{B}})}^2}dm\hfill \\ \hfill F\left(m\right)& =\mathsf{\Phi }\left(\frac{m-2\overline{\mathit{A}}+2\overline{\mathit{B}}}{\sqrt{S_{\mathit{A}+\mathit{B}}^2}}\right),\hfill \end{array}$$

(31)

where Φ is the $cdf$ of standardized normal distribution with mean 0 and variance 1.

Figure 3. Graphs of the $pdf$ of for the difference of two traces of $2\times 2$ random confusion matrices for different effect size $\delta =tr\left(\mathit{A}\right)-tr\left(\mathit{B}\right)$.

Figure 3. Graphs of the $pdf$ of for the difference of two traces of $2\times 2$ random confusion matrices for different effect size $\delta =tr\left(\mathit{A}\right)-tr\left(\mathit{B}\right)$.

Figure 4. Graphs of the $cdf$ of for the difference of two traces of $2\times 2$ random confusion matrices for different effect size $\delta =tr\left(\mathit{A}\right)-tr\left(\mathit{B}\right)$

Figure 4. Graphs of the $cdf$ of for the difference of two traces of $2\times 2$ random confusion matrices for different effect size $\delta =tr\left(\mathit{A}\right)-tr\left(\mathit{B}\right)$

3. Example

To demonstrate our approach, let’s examine an example featuring two classifiers, A and B, generating the following confusion matrices on the identical testing dataset T, where the size of T is $n_T=200$:

$$\begin{array}{cc}\hfill \mathit{A}& =\left[\begin{array}{cc}62& 36\\ 51& 51\end{array}\right]\hfill \\ \hfill \mathit{B}& =\left[\begin{array}{cc}50& 53\\ 50& 47\end{array}\right].\hfill \end{array}$$

(32)

The eigenvalues of matrices $\mathit{A}$ and $\mathit{B}$ are denoted by $({\lambda }_1^A,{\lambda }_2^A)$ and $({\lambda }_1^B,{\lambda }_2^B)$ respectively. Correspondingly, the traces of matrices $\mathit{A}$ and $\mathit{B}$ can be computed as follows:

$$\begin{array}{cc}\hfill tr\left(\mathit{A}\right)& ={\lambda }_1^A+{\lambda }_2^A\hfill \\ \hfill tr\left(\mathit{B}\right)& ={\lambda }_1^B+{\lambda }_2^B.\hfill \end{array}$$

(33)

The estimates for the eigenvalues and traces of matrices $\mathit{A}$ and $\mathit{B}$ are as follows: $({\lambda }_1^A=99.7,{\lambda }_2^A=13.3,\mathrm{tr}\left(\mathit{A}\right)=113)$ and $({\lambda }_1^B=100,{\lambda }_2^B=-3,\mathrm{tr}\left(\mathit{B}\right)=97)$ respectively. With these trace values, we can compute the accuracies of the two classifiers: $({\varphi }^A=0.57,{\varphi }^B=0.49)$. According to this criterion, it seems that classifier A outperforms B. However, without sufficient information, we cannot conclusively determine whether this superiority is genuine or merely a result of chance. By analyzing the distribution of the difference between the two traces, as shown in (30) and (31), we can quantify the extent to which classifier A is superior to classifier B. Therefore, the probability that classifier A genuinely outperforms B is given by:

$$\begin{array}{cc}\hfill P[({\varphi }^A=0.57-{\varphi }^B=0.49)>0]& =1-P[({\varphi }^A=0.57-{\varphi }^B=0.49)<0]\hfill \\ \hfill & =1-P(m\le 0)\hfill \\ \hfill & =1-F\left(m\right)\hfill \\ \hfill & =1-\mathsf{\Phi }\left(\frac{m-2\overline{\mathit{A}}+2\overline{\mathit{B}}}{\sqrt{S_{\mathit{A}+\mathit{B}}^2}}\right)\hfill \\ \hfill & =1-\mathsf{\Phi }\left(\frac{m-0}{\sqrt{S_{\mathit{A}+\mathit{B}}^2}}\right)\hfill \\ \hfill & =1-\mathsf{\Phi }\left(\frac{0.08}{\sqrt{0.0775}}\right)\hfill \\ \hfill & =0.8492.\hfill \end{array}$$

(34)

This estimated probability value suggests a strong likelihood that model A significantly surpasses model B in terms of accuracy performance.

4. Applications

We utilize the following datasets to demonstrate the practical application of analyzing the distribution of differences between two traces of random confusion matrices in machine learning, particularly within the field of medicine and health:

• Heart disease [28]: This dataset comprises information from 303 patients with heart disease at Cleveland Hospital, including 14 features. The objective is to determine the presence or absence of heart disease.
• Breast cancer [29]: Originating from the University Medical Centre, Institute of Oncology, Ljubljana, Yugoslavia, this dataset contains data from 286 patients with breast cancer, encompassing 9 features. The goal is to predict the presence or absence of breast cancer recurrence.
• Liver disease [30]: This dataset consists of 584 patient records from the NorthEast region of Andhra Pradesh, India, across 10 features. The objective is to predict whether a patient has liver disease using various biochemical markers.

The aim of this section is to implement and compare four baseline machine learning algorithms applied to these datasets: logistic regression (LR), decision trees (DT), random forest classification (RF), and XGboost classification (XG) [31]. The evaluation criterion utilized to compare the ML algorithms is accuracy. In addition, the supremacy of each of the algorithms is computed by computing the probability distribution in (31). Note that this probability can empirically be computed by bootstrapping the dataset L times and then obtaining the empirical distribution of the difference between two accuracies or traces. Thus, the approximate bootstrap estimate of (31) is

$$\widehat{F}\left(m\right)=L^{-1}\sum _{l=1}^L(m_l<m),$$

(35)

where $L=5000$ is set as the bootstrap sample size. The significance of the approach presented in this study lies in its provision of a closed-form solution for this distribution. This solution offers a faster and more accurate method for calculating the distribution of differences between two accuracies.

Table 1 presents bootstrap accuracy estimates, denoted as ${\widehat{\varphi }}_L$, along with their standard errors, $SE\left({\widehat{\varphi }}_L\right)$, and accuracy estimates for eigenvalue distribution, denoted as ${\widehat{\varphi }}_{\lambda }$, along with their standard errors, $SE\left({\widehat{\varphi }}_{\lambda }\right)$, for the three datasets using the four baseline ML methods. The results indicate that the accuracy estimates and associated standard errors using both the bootstrap and eigenvalue distribution approaches are similar across the machine learning (ML) methods and datasets. This finding empirically validates the eigenvalue distribution approach for estimating the accuracy of an ML method based on the eigenvalue of a confusion matrix.

Table 2 presents pairwise comparison results of the accuracies of the four ML methods using both bootstrap and eigenvalue distribution approaches. Again, the estimates of the pairwise differences are similar in most cases in terms of direction (positive or negative). However, significant differences exist in the estimates of the superiority probability between the bootstrap and eigenvalue distribution approaches. On average, the results are approximately similar for positive differences but exhibit distinct differences for negative differences. The bootstrap tends to be conservative on average when the difference between the accuracies of two ML methods is negative but restrictive when the difference is positive. It is worth noting that bootstrap estimates are approximations to the distribution of the difference of ML accuracy, while the eigenvalue distribution provides the actual distribution of the difference based on Theorem (3). Thus, the results of the superiority probability obtained using the eigenvalue distribution are more reliable than bootstrap estimates, which have been reported in previous studies to have potentially biased estimates [32,33].

In terms of ML performance based on superiority probability, XG is on average better than LR and DT, while RF is on average better than XG. Thus, RF emerges as the best among the four ML methods across the three datasets in terms of prediction accuracy and superiority of accuracy on several replications of the experiment.

5. Conclusion

In conclusion, our study presents bootstrap accuracy estimates and accuracy estimates for eigenvalue distribution across various machine learning (ML) methods and datasets. The findings suggest that both approaches yield similar accuracy estimates and standard errors, validating the effectiveness of the eigenvalue distribution method for ML accuracy estimation based on confusion matrix eigenvalues. Pairwise comparison results reveal consistent estimates of differences between ML methods, yet significant variations exist in superiority probability estimates between bootstrap and eigenvalue distribution approaches. Notably, the bootstrap method tends to be conservative for negative differences and restrictive for positive ones. This underscores the importance of considering the actual distribution provided by the eigenvalue approach for more reliable superiority probability assessments.