Exploring Multivariate Statistics: Unveiling the Power of Eigenvalues in Wishart Distribution Analysis

1. Introduction

The Wishart distribution is a standard tool in multivariate statistics due to its importance as a multivariate probability distribution. John Wishart first presented this distribution in 1928[1]; it generalizes the matrix-variate gamma distribution straightforwardly, if the data follow a multivariate normal distribution, then the sample covariance matrices will have this distribution.

The Wishart distribution has been the subject of thorough investigation within the literature of multivariate analysis concerning its properties and theoretical foundations[2,3,4].

Let $\mathbf{X}$ denote a random positive definite matrix with dimensions $p\times p$. If $\mathbf{X}$ satisfies the probability density function (PDF), it is considered to adhere to a Wishart distribution with parameters $\mathbf{V}$ and n. The PDF of $\mathbf{X}$ in the Wishart distribution is given by:

$$f(\mathbf{X};\mathbf{V},n)=\frac{{\left|X\right|}^{\frac{n-p-1}{2}}exp\left(-\frac{1}{2}\mathrm{tr}\left({\mathbf{V}}^{-1}\mathbf{X}\right)\right)}{2^{\frac{np}{2}}{\left|\mathbf{V}\right|}^{\frac{n}{2}}{\Gamma }_p\left(\frac{n}{2}\right)},$$

(1)

where $\mathbf{V}$ is an $n\times n$ positive definite matrix, $v>n-1$ is the degrees of freedom, ${\Gamma }_n(·)$ is the multivariate gamma function, and $\mathrm{tr}(·)$ denotes the matrix trace. The sample covariance matrix’s probability distribution follows a Wishart distribution $\mathbf{S}=\mathbf{X}{\mathbf{X}}^{\prime }$, where the columns of $\mathbf{X}$ are i.i.d multivariate normal ${\mathcal{N}}_n(\mathbf{0},\mathbf{V})$. In this context, $\mathbf{V}$ represents the population covariance matrix.

The expected value and variance of $\mathbf{S}$ are given by:

$$\begin{array}{cc}\hfill \mathbb{E}\left[\mathbf{S}\right]& =v\mathbf{V}\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill \mathrm{Var}\left(\mathbf{S}\right)& =v(\mathbf{V}\otimes \mathbf{V})\hfill \end{array}$$

(3)

where ⊗ denotes the Kronecker product.

Multivariate statistics use the Wishart distribution for covariance matrix estimation, uncertainty analysis, and inference[5,6,7].The Wishart distributed data is a prior that pairs well with the inverse covariance matrix of a multivariate-normal random vector in Bayesian statistics. [6]. It is commonly used in random matrix spectral theory [8]. Furthermore, The Wishart distribution helps statisticians and machine learners identify covariance matrices and uncertainty.

Various machine learning methods employ Wishart distributions to represent covariance patterns in multivariate data and evaluate Rayleigh fading MIMO wireless channel performance in wireless communications[9][10]In finance, Wishart-based covariance forecasts improve portfolio risk assessment[11]. Wishart-based stochastic volatility models are used to analyze changes in financial time series covariance matrices[12].

Székely developed energy statistics to assess distribution differences using "potential energy"[13]. The energy distance E between Wishart-distributed matrix eigenvalue distributions is:

$$E=\frac{1}{2n}\sum _{i=1}^n{(F_1\left({\lambda }_i\right)-F_2\left({\lambda }_i\right))}^2$$

(4)

The empirical cumulative distribution functions (ECDFs) of eigenvalues from two Wishart-distributed matrices are $F_1\left(\lambda \right)$ and $F_2\left(\lambda \right)$, with ${\lambda }_i$ representing the i-th eigenvalue This energy distance measure quantifies structural differences in eigenvalue distributions, making Wishart-distributed data comparisons robust.

In this sense, the points of data that are statistically closer together have less "potential energy," while those that are farther away have more[14]. The metric of energy distance is applied to distinguish distributions and underlying statistical analysis and inference [15]. Energy statistics can evaluate differences between samples or proposed distributions based on observation distances [14]. These techniques are used for feature selection, nonparametric distribution equality tests, and independence testing [16].

Energy distance based on Wishart distribution eigenvalue distribution may be beneficial for distribution comparisons. Instead of comparing individual components in the covariance matrices, this method measures the disparity between the underlying structures of the distributions, as expressed by their eigenvalue distributions. High-dimensional Wishart distributions may make conventional methods computationally costly or unintelligible because of to the "curse of dimensionality." The energy distance technique leverages the unique properties of eigenvalue distributions to provide a precise and useful measure of dissimilarity across Wishart distributions, facilitating for comparisons and analysis in a variety of application fields.

There are some problems with using Szekely’s energy statistics directly with pairwise distances for Wishart distributions, even though they provide a strong foundation. First, Euclidean distance is unsuitable because covariance matrices, which express linear interactions between features, fall into non-Euclidean space. Additionally, by ignoring the underlying structure and connections of the covariance matrices and instead computing element-wise differences many valuable information would be lost. These distances also struggle to capture substantial differences across Wishart distributions due to scaling sensitivity and transformation invariance. These difficulties need alternate ways like the suggested energy distance based on eigenvalue distribution, which overcomes these constraints and uses distinctive properties of eigenvalue distributions for discrepancy measurement.

The eigenvalue distribution-based energy distance eliminates pairwise distance drawbacks. To address issues with non-Euclidean space, such as information loss and scale sensitivity, we focus on Wishart distribution eigenvalue distributions. By comparing empirical cumulative distribution functions (ECDFs) of eigenvalues, the energy distance measures Wishart sample structural dissimilarity. Dissimilarities may be difficult to discover with pairwise distance.

2. Eigenvalue Decomposition for Wishart-Distributed Matrices

Understanding eigenvalue decomposition is crucial for understanding linear transformations in Wishart-distributed matrices $\mathbf{W}$ with dimensions $(n\times n)$, where n indicates data dimensionality. The eigenvalues of $\mathbf{W}$, ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_n$, are linked to their corresponding eigenvectors ${\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_n$ in non-increasing order, forming the eigenvector matrix $\mathbf{V}$ with a dimension of $(n\times n)$. The decomposition procedure starts with the determination of the characteristic equation, $|\mathbf{W}-\lambda \mathbf{I}|=0$, which results in the n eigenvalues, where $\lambda$ represents an eigenvalue and $\mathbf{I}$ is the identity matrix matching the dimensions of $\mathbf{W}$.

Subsequently, for each derived eigenvalue ${\lambda }_i$, its corresponding eigenvector ${\mathbf{v}}_i$ is determined by solving $(\mathbf{W}-{\lambda }_i\mathbf{I}){\mathbf{v}}_i=\mathbf{0}$, a system ensuring non-trivial solutions due to the determinant condition of $(\mathbf{W}-{\lambda }_i\mathbf{I})$ being zero. Given the real and symmetric nature of $\mathbf{W}$ in Wishart distributions, all ${\lambda }_i$ are real and non-negative, and eigenvectors associated with distinct eigenvalues are orthogonal, i.e., ${\mathbf{v}}_i^T{\mathbf{v}}_j=0$ for $i\ne j$. Ultimately, the eigenvalue decomposition of $\mathbf{W}$ is formulated as $\mathbf{W}=\mathbf{V}\Lambda {\mathbf{V}}^T$, where $\Lambda =\mathrm{diag}({\lambda }_1,{\lambda }_2,\dots ,{\lambda }_n)$ and $\mathbf{V}$ consists of columns of eigenvectors. Decomposing the data represented by $\mathbf{W}$ reveals its structure, including the variance (eigenvalues) and orientation (eigenvectors) of its major components.

2.1. Eigenvalues and Eigenvectors in Covariance Matrix

Eigenvalue decomposition is essential for understanding matrix behaviors, especially in covariance matrices, where eigenvalues and eigenvectors reveal data structure and variability. A covariance matrix, $\Sigma$, outlines linear relationships between dataset features, forming a $p\times p$ square matrix for p features, where each element ${\Sigma }_{ij}$ denotes the covariance between the i-th and j-th features. Eigenvalues (${\lambda }_i$) of $\Sigma$, derived from solving the characteristic equation $|\Sigma -\lambda I|=0$ where I is the identity matrix of corresponding dimensions, represent the variance amounts captured by linear data transformations, highlighting the spread or dispersion. Each eigenvalue is associated with a non-zero eigenvector (${\mathbf{v}}_i$), defining the direction of maximum variance, obtained by solving $(\Sigma -{\lambda }_{i}I){\mathbf{v}}_i=\mathbf{0}$, indicative of a homogeneous system. Eigenvectors corresponding to distinct eigenvalues are orthogonal, satisfying ${\mathbf{v}}_i^T{\mathbf{v}}_j=0$ for $i\ne j$, thereby ensuring independent variance directions in data space. The eigenvalue decomposition of $\Sigma$ is $\Sigma =V\Lambda V^T$, where V is the matrix of eigenvectors and $\Lambda$ is the diagonal matrix containing eigenvalues, which reveals the structure of the covariance matrix. Further, Principal Component Analysis (PCA) uses eigenvalues and eigenvectors to emphasize the most significant variance directions of the data, reducing dimensionality while preserving important information.

2.2. Energy Distance for Discrepancy between Eigenvalue Distributions

Energy distance, conceptualized by Szekely, provides a quantitative measure for assessing the discrepancy between probability distributions, adapted in this context to evaluate the differences between eigenvalue distributions from Wishart-distributed matrices. Denote ${\lambda }_i^1$ and ${\lambda }_i^2$ as the i-th eigenvalues from two distinct Wishart-distributed matrices ${\mathbf{W}}_1$ and ${\mathbf{W}}_2$, respectively, ordered non-increasingly. Let $F_1\left(\lambda \right)$ and $F_2\left(\lambda \right)$ represent the empirical cumulative distribution functions (ECDFs) for the eigenvalues of ${\mathbf{W}}_1$ and ${\mathbf{W}}_2$, respectively. The ECDF at a specific value $\lambda$ signifies the probability of an eigenvalue being less than or equal to $\lambda$, with n indicating the dimensionality of the matrices. The energy distance E between the two eigenvalue distributions is mathematically defined as in equation 4

where the discrepancy at each eigenvalue level is accentuated by the differences in ECDFs at each ${\lambda }_i$, with larger discrepancies indicating greater differences. The squared differences emphasize significant gaps, while summing across all n eigenvalues furnishes a comprehensive assessment.

The normalization factor $\frac{1}{2}$ ensures that the energy distance E remains non-negative and proportional to the count of eigenvalues. A higher value of E suggests significant differences between the eigenvalue distributions of the two matrices, indicative of a substantial discrepancy in their Wishart distributions. Conversely, a lower E reflects closer similarity between distributions. This metric, distinct from the Frobenius norm, specifically addresses structures inherent to eigenvalues, circumventing limitations associated with non-Euclidean space or loss of information, and thus provides a robust measure for matrix dissimilarity by leveraging the unique properties of eigenvalue distributions.

3. Simulation Study

This simulation study examines Wishart distribution eigenvalue distribution energy distance under many circumstances. Different degrees of freedom (v) will be studied to understand how Wishart distribution and eigenvalue dispersion affect energy distance. An analysis will be conducted to assess the influence of various configurations of the theoretical covariance matrix (${\mathbf{W}}_0$), including the identity matrix (${\mathbf{W}}_0=\mathbf{I}$) for uncorrelated features with identical eigenvalues, a diagonal matrix (${\mathbf{W}}_0=\mathrm{diag}({\sigma }_1,{\sigma }_2,\dots ,{\sigma }_n)$ for distinct variances per feature, and structured matrices with specific correlation patterns. We will create $n=100$ Wishart-distributed matrices for each configuration and parameter combination for statistical simulation.

3.1. Exploration of Scenarios Impacting Energy Distance

Here, we focus on the impact of degrees of freedom (v) and theoretical covariance matrix structures (${\mathbf{W}}_0$) on energy distance.

3.1.1. Degrees of Freedom ($\nu$)

We investigate how three degrees of freedom impact Wishart distribution and eigenvalue dispersion as follow:

• $\nu =5$: Less degree of freedom leads to a more skewed distribution with a higher eigenvalue dispersion, which means a less stable eigenvalue spectrum.
• $\nu =10$: At this moderate degree of freedom, a predicted equilibrium between the eigenvalue spread and distribution stability balances distribution features.
• $\nu =20$: Higher degree of freedom denotes a distribution that approximates normality with a smaller eigenvalue variance, indicating a more stable and less variable distribution.

3.1.2. Theoretical Covariance Matrix (${\Sigma }_0$)

We investigate three covariance matrix configurations to determine their effect on energy distance in different data interrelationships:

• Identity Matrix (${\Sigma }_0=I$): Symbolizes uncorrelated features with equal variance (diagonal elements equal to 1, and off-diagonal elements being 0), leading to identical eigenvalues due to absent feature correlations.
• Diagonal Matrix with Varied Elements (${\Sigma }_0=\mathrm{diag}({\sigma }_1,{\sigma }_2,\dots ,{\sigma }_p)$): This configuration, varying diagonal elements (${\sigma }_i$), introduces distinct variances per feature, from ${\sigma }_1=2$ down to ${\sigma }_p=0.1$, aiming to scrutinize the effect of a gradient in variance on the eigenvalue distribution dynamics.

3.1.3. Structured Matrix with Correlation Patterns (${\mathbf{W}}_0$)

Instills specific feature dependencies via a predefined structured covariance matrix, exemplified by:

$${\mathbf{W}}_0=\left[\begin{array}{ccc}{\omega }_{11}& {\omega }_{12}& {\omega }_{13}\\ {\omega }_{21}& {\omega }_{22}& {\omega }_{23}\\ {\omega }_{31}& {\omega }_{32}& {\omega }_{33}\end{array}\right],$$

introducing discernible positive correlations between select features, thereby influencing the resultant eigenvalue distributions compared to the initial scenarios.

Through these scenarios, we aim to unravel the nuanced responses of energy distance to alterations in the Wishart distributions’ structure and parameterization.

Figure 1 shows the integrated visual analysis of energy distance distributions across Identity, Diagonal, and Structured scenarios and degrees of freedom (DoF: 5, 10, 20). The degree of freedom (DoF) has a substantial impact on the dispersion and central tendency of distribution since bigger DoFs lead to more reduced distributions. The diversity of energy distance is significantly impacted by the structures of covariance matrices, as the Diagonal and Structured scenarios reveal larger distributions at lower DoFs. Visualizing Wishart-distributed matrices helps statistical modeling and analysis of high-dimensional data.

The covariance matrix structure and degrees of freedom in dealing with energy distances in Wishart-distributed matrices are shown to be strongly related. Specifically, matrices with fewer degrees of freedom have more energy distance variability, which predicts more eigenvalue distribution deviations. Energy distance distributions decrease with increasing degrees of freedom, thereby maintaining eigenvalue dispersion.In degrees of freedom, identity matrices vary little, whereas diagonal and structured matrices are more sensitive. This work shows a detailed relationship between matrix structure and statistical properties, shedding light on Wishart distributions in multivariate statistical research.

4. Discussion

Energy distance evolution in Wishart-distributed matrices with variable degrees of freedom and covariance structures is naturally conveyed through simulations. Parametric enhancements transform energy distances, conveniently appreciated through spatial renditions. Energy distance distributions are larger for reduced degrees of freedom matrices, signifying corresponding larger eigenvalue dispersions. As degrees of freedom increase, this behavior becomes progressively ameliorated, leveling off for higher-degree matrices. The Structure of the matrix identity diagonal is disclosed as mediating these distributions. Higher energy distances for both diagonal- and structured matrices identity matrices implies data correlations and variances affect eigenvalue distributions. These phenomena deliver a comprehensive understanding of differential eigenvalue distribution spread and similarity given influential factors, germane to Wishart distributions and data covariance estimates. Cognizant patterns and narratives conform to theoretical presuppositions and portend distinct multivariate statistical methodologies and data models.

5. Conclusion

In this study, we consider the effects of degrees of freedom and covariance matrix structures on energy distances in wishart distributed matrices in a large simulation setting. The results we obtained show that degrees of freedom and covariance matrix structure have strong influence on eigenvalue dispersion and energy distance distribution. It was observed that energy distance distributions tend to become narrower with larger degrees of freedom, thus suggesting that higher parameter values stabilize them. More crucially, the Identity, Diagonal, and Structured matrix settings indicate that eigenvalue dispersion is also conditioned on the underlying relationships of the data.

These results could offer practical utility to fitting statistical models and making data analysis in diverse fields such as finance, genetics, and environmental research where the structure of data covariance plays a critical role. Further, the results could serve to provide a body of empirical evidence accompanying theoretical predictions to further understand Wishart distributions and applications, adding to the existing pool of knowledge.

To confirm the practical application of these results, future research should expand this study to higher-dimensional settings, investigate distributional assumptions, and apply these insights to real-world datasets. Researchers may strengthen multivariate statistical analysis and deepen our knowledge of Wishart distributions by expanding on the work of this research. This study improves theoretical understanding and strengthens the relationship between simulation studies and modern statistics research.