Geometric Residual Projection in Linear Regression: Rank-Aware Operators and a Geometric Multicollinearity Index

1. Introduction

Linear regression remains one of the most widely used modeling tools in statistics, data science, and machine learning [2]. In its familiar form

$$Y=X\beta +\epsilon ,\mathbb{E}\left[\epsilon \right]=0,X\in {\mathbb{R}}^{n\times p},Y\in {\mathbb{R}}^n,\beta \in {\mathbb{R}}^p,$$

(1)

The ordinary least squares (OLS) estimator chooses coefficients that minimize the squared discrepancy between observations and predictions. When the design matrix X has the full column-rank, the fitted values and the associated orthogonal projector onto the column space $C\left(X\right)$ take the classical form

$$\widehat{\beta }={\left(X^{\top }X\right)}^{-1}{X}^{\top }Y,\widehat{Y}=X\widehat{\beta },P=X{\left(X^{\top }X\right)}^{-1}{X}^{\top },$$

(2)

with P symmetric and idempotent [5]. More generally, the orthogonal projector onto $C\left(X\right)$ can be written as $P=X{X}^{+}$ in terms of the Moore–Penrose pseudoinverse $X^{+}$ [5].

The fitted component and residual are

$$\widehat{Y}=PY,r=(I-P)Y,$$

(3)

so that $r\in C{\left(X\right)}^{\perp }$ and

$$Y=PY+r,$$

is the orthogonal decomposition of the response into a part explained by $C\left(X\right)$ and a part orthogonal to it. This algebraic formulation is standard, but it can obscure the underlying geometry that governs the behavior of the residuals, and inversion-based formulas can be fragile in high-dimensional or nearly singular regimes [5].

A classical identity from numerical linear algebra makes the geometry explicit:

$$I-P=N{N}^{\top },$$

(4)

where the columns of N form an orthonormal basis of $C{\left(X\right)}^{\perp }$ [5]. When $\rho =n-1$, the orthogonal complement is one-dimensional and the projector reduces to the rank-one form

$$I-P=n{n}^{\top },$$

(5)

where n is the unit vector (ambiguous sign) perpendicular to $C\left(X\right)$. In this codimension-one setting, the residual is the projection of Y into a single direction, and the operator (5) completely avoids matrix inversion. For higher-codimension, the residual lies in a subspace of dimension $k=n-\rho$, and the factorization (4) makes this structure explicit.

The construction of normal directions motivates generalizations of the classical cross-product to ${\mathbb{R}}^n$. Belhaouari et al. formalize a determinant-based vector product that yields a vector orthogonal to $n-1$ linearly independent vectors in ${\mathbb{R}}^n$ [7]. In regression problems with $dimC{\left(X\right)}^{\perp }=1$, a single normal vector uniquely defines the rank-one projector (5). For higher-codimensions, these constructions naturally combine with null-space and QR-based methods to form an orthonormal residual basis N.

Multicollinearity is a persistent challenge in regression modeling, as highly dependent predictors inflate variances, destabilize coefficient estimates, and impede interpretation [11]. Classical diagnostics such as the variance inflation factor (VIF) quantify this effect but offer limited geometric insight. We therefore introduce the Geometric Multicollinearity Index (GMI), a scale-invariant, volume-based measure derived from the polar sine and the Gram determinant. For a full column-rank matrix $X=[x_1,\dots ,x_p]$ with $p\le n$, the polar sine normalizes $det\left(X^{\top }X\right)$ by the product of the column norms, taking values in $[0,1]$: it equals 1 for orthogonal predictors and approaches 0 as multicollinearity increases.

In rank-deficient or over-parameterized regimes, the Gram determinant vanishes and the GMI saturates at 1, indicating extreme geometric degeneracy. As a normalized volume measure of the parallelotope spanned by the columns of X, the GMI provides a compact, scale-invariant geometric diagnostic across dimensions. In this regime, $\mathrm{GMI}=1$ should be interpreted as a diagnostic flag for complete geometric collapse, rather than as a measure of severity beyond rank deficiency.

Although both regression and principal component analysis (PCA) rely on orthogonal projections, their objectives differ fundamentally [4]. PCA projects X onto directions of maximum variance, yielding residuals $(I-P_{\mathrm{PCA}})X$ that remain in $C\left(X\right)$. By contrast, regression projects the response, decomposing Y into a fitted component $PY$ and a residual $r=(I-P)Y\in C{\left(X\right)}^{\perp }$ [5]. This distinction between feature-space and response-space residuals is essential for interpreting linear-model diagnostics.

Scope of the present paper. The identities $I-P=N{N}^{\top }$ and $I-P=n{n}^{\top }$ are standard results from orthogonal projection theory. The contribution of this paper lies in a rank-aware geometric interpretation of these identities via multivector and cross-product constructions, together with the introduction of the volume-based Geometric Multicollinearity Index (GMI) as a complementary geometric diagnostic.

This work is theoretical in nature. We develop a rank-aware residual projector, characterize the associated residual subspaces, and formalize GMI as an intrinsic measure of multicollinearity. Algorithmic, large-scale, regularized, and nonlinear extensions are beyond the scope of this paper and are left for future work.

Contributions. The main contributions of this theoretical paper are as follows:

• Rank-aware geometric formulation. We develop a unified, rank-aware description of the residual projector based on the identity $I-P=N{N}^{\top }$, which reduces to the rank-one form $n{n}^{\top }$ when $dimC{\left(X\right)}^{\perp }=1$ and generalizes to arbitrary rank.
• Multivector perspective on residual spaces. We reformulate $C{\left(X\right)}^{\perp }$ and the associated projector using multivectors, wedge products, and cross-products, yielding a basis-independent geometric characterization valid in both codimension-one and higher-codimension settings.
• Geometric Multicollinearity Index (GMI). We introduce a scale-invariant, volume-based diagnostic derived from the Gram determinant and polar sine, which quantifies geometric degeneracy in both full-rank and rank-deficient regimes and complements classical VIF-type measures.
• Regression versus PCA residuals. We clarify the conceptual distinction between regression residuals and PCA reconstruction residuals, emphasizing the impact of projecting the response versus the predictors on residual geometry and interpretation.
• Illustrative examples. We provide simple numerical examples illustrating rank-one and multivector residual projections and the behavior of GMI under controlled perturbations.

Organization of the paper.Section 2 introduces the notation and basic geometric concepts. Section 3 develops the main theoretical results on residual projectors and the geometric structure of $C{\left(X\right)}^{\perp }$. Section 4 presents numerical illustrations, including the behavior of GMI under controlled multicollinearity. Section 5 and Section 6 discuss implications and conclude.

2. Notation and Preliminaries

We briefly fix notation and recall the algebraic and geometric concepts that underpin the proposed framework. We follow standard linear algebra conventions as in Strang [1], Lay et al. [3], and Golub and Van Loan [5].

2.1. Basic Notation and Subspaces

The transpose of a matrix or vector is denoted by the superscript ⊤. Lowercase letters (e.g., x, n) denote column vectors, and uppercase letters (e.g., X, N) denote matrices. For a matrix $X\in {\mathbb{R}}^{n\times p}$, we write $\mathcal{C}\left(X\right)$ for its column space and denote its rank by $\rho$. Throughout, r denotes the residual vector, while $\rho$ denotes the rank of X, to avoid notational ambiguity. All notation is defined once and used consistently throughout the manuscript; in particular, N denotes an orthonormal basis of the orthogonal complement $\mathcal{C}{\left(X\right)}^{\perp }$.

We reserve r exclusively for the residual vector in (8).

The orthogonal complement of a subspace $V\subseteq {\mathbb{R}}^n$ is denoted by $V^{\perp }$, and the null-space of a matrix A is

$$\mathcal{N}\left(A\right)=\{z\in {\mathbb{R}}^m:Az=0\}.$$

A summary of all symbols and subspace conventions is provided in Abbreviations.

Given $n\in {\mathbb{R}}^n$, the matrix $n{n}^{\top }\in {\mathbb{R}}^{n\times n}$ denotes its outer product. The identity matrix in ${\mathbb{R}}^{n\times n}$ is denoted by I.

2.2. Projection and Residual Operators

Let $X\in {\mathbb{R}}^{n\times p}$ have rank $\rho \le min\{n,p\}$; when X has full column-rank ($\rho =p$), the orthogonal projector admits the classical closed form $P=X{\left(X^{\top }X\right)}^{-1}{X}^{\top }$.

The orthogonal projector onto the column space $\mathcal{C}\left(X\right)$ is the unique symmetric idempotent matrix P that satisfies

$$\mathcal{C}\left(P\right)=\mathcal{C}\left(X\right),P^{\top }=P,P^2=P.$$

When X has full column-rank $\rho$, this projector admits the well-known closed form

$$P=X{\left(X^{\top }X\right)}^{-1}{X}^{\top },$$

(6)

which is symmetric and idempotent [5]. In the general case, including rank-deficient designs, P can be written as

$$P=X{X}^{+},$$

(7)

where $X^{+}$ denotes the Moore–Penrose pseudoinverse of X [5].Throughout, $X^{+}$ denotes the Moore–Penrose pseudoinverse of X, computed via an SVD-based routine.

Both (6) and (7) implement the same geometric operation: orthogonal projection onto $\mathcal{C}\left(X\right)$.

When $X^{\top }X\succ 0$, the same orthogonal projector can equivalently be constructed by Cholesky factorization of $X^{\top }X$, producing identical residuals while requiring positive definiteness of the Gram matrix.

For any response vector $Y\in {\mathbb{R}}^n$, the fitted component is

$$\widehat{Y}=PY,$$

and the residual is

$$r=(I-P)Y,$$

(8)

which obeys $X^{\top }r=0$ and hence lies in the orthogonal complement $\mathcal{C}{\left(X\right)}^{\perp }$ [5].

Let $\rho =rank\left(X\right)$ and set $k=n-\rho =dim\mathcal{C}{\left(X\right)}^{\perp }$. Choose any matrix $N\in {\mathbb{R}}^{n\times k}$ whose columns form an orthonormal basis of $\mathcal{C}{\left(X\right)}^{\perp }$:

$$N^{\top }N=I_k,X^{\top }N=0.$$

(9)

Then the residual projector admits the geometric factorization

$$I-P=N{N}^{\top },$$

(10)

and the null-space of $X^{\top }$ is

$$\mathcal{N}\left(X^{\top }\right)=\{z\in {\mathbb{R}}^n:X^{\top }z=0\}=\mathcal{C}{\left(X\right)}^{\perp }.$$

(11)

This characterization is standard in numerical linear algebra and regression theory [5] and forms the algebraic backbone of our geometric residual formulation. Equation (10) makes explicit that the residual r lies in a k-dimensional orthogonal complement spanned by the columns of N.

2.3. Cross-Product and Cross–Wedge–QR Method

To construct explicit orthogonal directions, especially in the case of codimension-one where $dim\mathcal{C}{\left(X\right)}^{\perp }=1$, we employ the cross-product in ${\mathbb{R}}^n$ [7]. Given $n-1$ linearly independent vectors

$$x_1,\dots ,x_{n-1}\in {\mathbb{R}}^n,$$

the determinant-based construction of Belhaouari et al. produces

$$cross(x_1,\dots ,x_{n-1})\in {\mathbb{R}}^n,$$

which is orthogonal to each $x_i$:

$$x_i^{\top }cross(x_1,\dots ,x_{n-1})=0,i=1,\dots ,n-1.$$

This extends earlier n-dimensional vector products and related constructions in eigenanalysis and geometry [8,9].

In the regression setting, when $\rho =n-1$ the predictor space $\mathcal{C}\left(X\right)$ has codimension one and $\mathcal{C}{\left(X\right)}^{\perp }$ is spanned by a single unit vector n. Choosing any basis of $\mathcal{C}\left(X\right)$ and applying the cross-product yields a nonzero vector orthogonal to $\mathcal{C}\left(X\right)$; normalizing it gives a unit normal n spanning $\mathcal{C}{\left(X\right)}^{\perp }$. In this case, the residual projector reduces to the rank-one form

$$I-P=n{n}^{\top },$$

(12)

and the residual is simply $r=\left(n^{\top }Y\right)n$.

In this codimension-one setting, the rank-one projector $I-P=n{n}^{\top }$ coincides with the classical construction of the cross-product, and we refer to this case as the residual projector of the cross-product (rank-one).

A key geometric property of the cross-product is that its norm equals the $(n-1)$-dimensional volume of the parallelotope spanned by its arguments [7]:

$$∥cross(v_1,v_2,\dots ,v_{n-1})∥={vol}_{n-1}(v_1,\dots ,v_{n-1}).$$

(13)

Thus, the cross-product simultaneously provides an orthogonal direction and encodes a volume. When $dim\mathcal{C}{\left(X\right)}^{\perp }>1$, a single cross-product yields at most one null direction; in that case additional null directions must be obtained by other means (e.g., null-space methods and QR), as discussed later.

2.4. Wedge Products, Gram Matrices, and Polar Sine

To describe volumes and linear independence in a coordinate-free manner, we rely on standard constructions from linear algebra and multilinear geometry [1,3,5]. Given vectors $u_1,\dots ,u_k\in {\mathbb{R}}^n$, their wedge product $u_1\wedge \dots \wedge u_k$ encodes the oriented k-dimensional parallelotope spanned by $\left\{u_i\right\}$. Geometrically, $u_1\wedge \dots \wedge u_k=0$ if and only if the vectors are linearly dependent, and its norm $\parallel u_1\wedge \dots \wedge u_k\parallel$ equals the volume k of the associated parallelotope. This interpretation underlies many geometric treatments of subspaces, volumes, and orthogonality in ${\mathbb{R}}^n$ [1,5].

For computational purposes, such volumes can be expressed using Gram matrices. Given vectors $v_1,\dots ,v_m\in {\mathbb{R}}^n$, the associated Gram matrix is

$$G=\left[\begin{array}{ccc}{v}_1^{\top }{v}_1& \dots & v_1^{\top }{v}_m\\ ⋮& \ddots & ⋮\\ v_m^{\top }{v}_1& \dots & v_m^{\top }{v}_m\end{array}\right]={[v_1,\dots ,v_m]}^{\top }[v_1,\dots ,v_m].$$

(14)

The m-dimensional volume of the parallelotope spanned by $\left\{v_i\right\}$ is given by

$${vol}_m(v_1,\dots ,v_m)=\sqrt{det\left(G\right)},$$

(15)

a classical identity that plays a central role in covariance analysis and principal component analysis [4,5].

We quantify the angular separation of p predictor vectors using the polar sine. Let $X=[x_1,\dots ,x_p]\in {\mathbb{R}}^{n\times p}$ with Gram matrix $G=X^{\top }X$. Following geometric constructions based on determinants and cross-products [7,8,9], the polar sine is defined as

$$psin(x_1,\dots ,x_p)={\displaystyle \frac{\sqrt{det\left(X^{\top }X\right)}}{\parallel x_1\parallel \dots \parallel x_p\parallel }}.$$

(16)

When $p\le n$ and the columns of X are linearly independent, $det\left(X^{\top }X\right)>0$ and $psin(x_1,\dots ,x_p)\in (0,1]$; it equals 1 if and only if the vectors are mutually orthogonal. If the columns of X are linearly dependent or $p>n$, then $X^{\top }X$ is singular and $det\left(X^{\top }X\right)=0$, and we adopt the convention $psin(x_1,\dots ,x_p)=0$. Thus, the polar sine takes values in $[0,1]$, is scale-invariant, and decreases toward zero as the set of columns becomes geometrically degenerate.

Within the present framework, the polar sine provides a normalized geometric measure of predictor-space collapse under multicollinearity and forms the basis of the Geometric Multicollinearity Index (GMI) introduced in the following section.

2.5. Projection Strategies: QR and Cross-Product Formulations

Two projection strategies play a central role in the remainder of the paper.

First, QR decomposition is used when an entire orthonormal basis of $\mathcal{C}\left(X\right)$ or $\mathcal{C}{\left(X\right)}^{\perp }$ is required. If $X=QR$ is a (thin) QR factorization with orthonormal columns in Q, then

$$P=Q{Q}^{\top },$$

is the orthogonal projector onto $\mathcal{C}\left(X\right)$, and orthonormal complements can be obtained by extending Q to an orthogonal basis of ${\mathbb{R}}^n$ [5]. QR-based projectors are numerically stable and well suited for high-dimensional or nearly singular problems.

Second, when the residual space is one-dimensional (for example, when $\rho =n-1$ ), the cross-product provides a compact alternative. It produces a normal vector n directly from which the residual projector is derived:

$$I-P=n{n}^{\top }.$$

This follows without forming or inverting $X^{\top }X$ [7]. In higher-codimension settings ($dim\mathcal{C}{\left(X\right)}^{\perp }>1$), cross-products can still be used to generate individual null directions, but a full residual basis is obtained more naturally from $\mathcal{N}\left(X^{\top }\right)$ and orthonormalised via QR.

When null directions are generated sequentially via cross-products and accumulated into a multivector basis that is subsequently orthonormalised using QR, we refer to the resulting construction as the Recursive Cross–Wedge–QR method.

These geometric primitives—orthonormal bases, cross-products, wedge products, Gram matrices, and polar sine—constitute the toolkit on which the residual projection framework developed in the following sections is built.

Here s denotes the sketch size (number of random probe vectors) used in the Gaussian range-finding procedure.

2.5.0.1. Sketch–QR Baseline (Reproducibility)

The Sketch–QR baseline follows a standard randomized range-finding procedure. Given $X\in {\mathbb{R}}^{n\times p}$ and a sketch size s, we draw a Gaussian sketching matrix $\Omega \in {\mathbb{R}}^{p\times s}$ with i.i.d. $\mathcal{N}(0,1)$ entries and form $Y=X\Omega \in {\mathbb{R}}^{n\times s}$. We then compute a thin QR factorization $Y=\tilde{Q}R$, where $\tilde{Q}\in {\mathbb{R}}^{n\times s}$ has orthonormal columns. The approximate projector is $\tilde{P}=\tilde{Q}{\tilde{Q}}^{\top }$, and the approximate residual is $\tilde{r}=(I-\tilde{P})y$. Unless stated otherwise, we use a single sketch without power iterations or oversampling [6,16].

3. Core Theory

3.1. Rank-One Residual Projection

Assume that $X\in {\mathbb{R}}^{n\times p}$ has rank $n-1$. Then $dim\mathcal{C}\left(X\right)=n-1$ and $dim\mathcal{C}{\left(X\right)}^{\perp }=1$. There exists a (unique up to sign) unit vector $n\in {\mathbb{R}}^n$ such that

$$X^{\top }n=0,\parallel n\parallel =1.$$

Proposition 1 (Rank-One Residual Projection).

Let $X\in {\mathbb{R}}^{n\times p}$ have $\rho =n-1$, and let P denote the orthogonal projector onto $\mathcal{C}\left(X\right)$ (for example, $P=X{X}^{+}$ in terms of the Moore–Penrose pseudoinverse $X^{+}$). Then the orthogonal residual projector is

$$I-P=n{n}^{\top }.$$

Proof. 

Because $\mathcal{C}{\left(X\right)}^{\perp }$ is one-dimensional, any residual $r\in \mathcal{C}{\left(X\right)}^{\perp }$ can be written as $r=\alpha n$ for some scalar $\alpha$. Since $n\in \mathcal{C}{\left(X\right)}^{\perp }$ and P is the orthogonal projector onto $\mathcal{C}\left(X\right)$, we have $Pn=0$ and therefore $(I-P)n=n$. For any $Y\in {\mathbb{R}}^n$, we may write $Y=u+v$ with $u\in \mathcal{C}\left(X\right)$ and $v\in \mathcal{C}{\left(X\right)}^{\perp }=span\left\{n\right\}$. Then

$$(I-P)Y=(I-P)(u+v)=v,$$

while

$$n{n}^{\top }Y=n{n}^{\top }(u+v)=n{n}^{\top }u+n{n}^{\top }v=0+v=v,$$

because $n^{\top }u=0$ and v is a scalar multiple of n. Thus $(I-P)Y=n{n}^{\top }Y$ for all Y, and therefore $I-P=n{n}^{\top }$, which is symmetric and idempotent.    □

3.2. Generalization to n Dimensions

The rank-one projector $I-P=n{n}^{\top }$ extends naturally to the general setting in which the predictor matrix $X\in {\mathbb{R}}^{n\times p}$ has $\rho <n$. In this case the orthogonal complement $\mathcal{C}{\left(X\right)}^{\perp }$ has dimension $k=n-\rho$, and the residual is no longer confined to a single direction but lies in a k-dimensional subspace.

Theorem 2 (Multivector Residual Projection).

Let X have rank $\rho <n$ and set $k:=n-\rho$. Let $N\in {\mathbb{R}}^{n\times k}$ have orthonormal columns spanning $\mathcal{C}{\left(X\right)}^{\perp }$ (so $N^{\top }N=I_k$). Then the residual projector is $I-P=N{N}^{\top }$ and $r=(I-P)y=N{N}^{\top }y$.

Proof. 

Since $N^{\top }N=I_{n-\rho }$, the matrix $N{N}^{\top }$ is symmetric and satisfies

$${\left(N{N}^{\top }\right)}^2=N\left(N^{\top }N\right)N^{\top }=N{I}_{n-\rho }{N}^{\top }=N{N}^{\top },$$

so $N{N}^{\top }$ is an orthogonal projector. Its range equals $\mathcal{C}\left(N\right)=\mathcal{C}{\left(X\right)}^{\perp }$.

On the other hand, P is the orthogonal projector onto $\mathcal{C}\left(X\right)$, so $I-P$ is the orthogonal projector onto $\mathcal{C}{\left(X\right)}^{\perp }$. For any $Y\in {\mathbb{R}}^n$, write $Y=u+v$ with $u\in \mathcal{C}\left(X\right)$ and $v\in \mathcal{C}{\left(X\right)}^{\perp }$. Then

$$(I-P)Y=v.$$

Since $v\in \mathcal{C}\left(N\right)$ and the columns of N are orthonormal, we also have

$$N{N}^{\top }Y=N{N}^{\top }(u+v)=N{N}^{\top }u+N{N}^{\top }v=0+v=v,$$

because $N^{\top }u=0$ and $N{N}^{\top }$ acts as the identity on $\mathcal{C}\left(N\right)$. Thus $(I-P)Y=N{N}^{\top }Y$ for all Y, and hence $I-P=N{N}^{\top }$.    □

Figure 1 illustrates the situation in a simple three-dimensional example. The predictor space occupies a lower-dimensional subspace (blue), while the residual lies in its orthogonal complement (orange), spanned by multiple orthonormal directions when $dim\mathcal{C}{\left(X\right)}^{\perp }>1$.

3.2.1. Geometric Multicollinearity Index (GMI)

The multivector formulation of the residual projector reflects the angular configuration of the columns of X. When the predictors are nearly linearly dependent, the volume of the parallelotope spanned by these columns collapses, signaling instability. To capture this behavior geometrically, we introduce the Geometric Multicollinearity Index (GMI), a scale-invariant measure based on the polar sine.

Definition 1 (Polar Sine and GMI).

Let $X=[x_1,\dots ,x_p]\in {\mathbb{R}}^{n\times p}$ have nonzero columns $x_j\ne 0$, and let $psin(x_1,\dots ,x_p)$ be the polar sine defined in Equation (16)

. We adopt the shorthand

$$psin\left(X\right):=psin(x_1,\dots ,x_p),$$

(17)

and define the Geometric Multicollinearity Index by

$$GMI\left(X\right)=1-psin\left(X\right).$$

(18)

Lemma 3 (Basic properties).

Let $X=[x_1,\dots ,x_p]\in {\mathbb{R}}^{n\times p}$ have nonzero columns, and let $G=X^{\top }X$ denote its Gram matrix. Then:

i) 

$\mathrm{psin}\left(X\right)\in [0,1]$ and hence $\mathrm{GMI}\left(X\right)\in [0,1]$;

ii) 

$\mathrm{psin}\left(X\right)=1$ (and $\mathrm{GMI}\left(X\right)=0$) if and only if the columns of X are mutually orthogonal;

iii) 

$\mathrm{psin}\left(X\right)=0$ (and $\mathrm{GMI}\left(X\right)=1$) whenever the columns are linearly dependent or $p>n$.

Proof. 

By Hadamard’s inequality,

$$det\left(G\right)\le \prod _{j=1}^p{\parallel x_j\parallel }^2,$$

with equality if and only if the columns of X are mutually orthogonal. Substituting this bound into the polar-sine definition Equation (16) yields

$$0\le \mathrm{psin}\left(X\right)={\displaystyle \frac{\sqrt{det\left(G\right)}}{\parallel x_1\parallel \dots \parallel x_p\parallel }}\le 1,$$

which proves (i). Moreover, $\mathrm{psin}\left(X\right)=1$ holds precisely when equality in Hadamard’s inequality holds, i.e., when the columns are pairwise orthogonal, giving (ii).

If the columns of X are linearly dependent or $p>n$, then G is singular and $det\left(G\right)=0$, so by definition $\mathrm{psin}\left(X\right)=0$ in that case, and (iii) follows immediately. The corresponding statements for $\mathrm{GMI}\left(X\right)$ are then direct consequences of the definition $\mathrm{GMI}\left(X\right)=1-\mathrm{psin}\left(X\right)$.    □

Lemma 4 (Scale invariance).

Let $A=\mathrm{diag}(a_1,\dots ,a_p)$ with each $a_j\ne 0$. Then $\mathrm{psin}\left(XA\right)=\mathrm{psin}\left(X\right)$ and hence $\mathrm{GMI}\left(XA\right)=\mathrm{GMI}\left(X\right)$.

Proof. 

We have ${\left(XA\right)}^{\top }\left(XA\right)=A^{\top }GA$ and $det\left(A^{\top }GA\right)=det{\left(A\right)}^{2}det\left(G\right)$, while $\parallel a_j{x}_j\parallel =|a_j|\parallel x_j\parallel$ for each j. The scale factors cancel in the quotient (17), so $\mathrm{psin}\left(XA\right)=\mathrm{psin}\left(X\right)$, and (18) yields the claim for GMI.    □

Remark 1.

For orthonormal predictors, $\mathrm{psin}\left(X\right)=1$ and $\mathrm{GMI}\left(X\right)=0$. For nearly collinear predictors, $det\left(G\right)\approx 0$, so $\mathrm{psin}\left(X\right)\approx 0$ and $\mathrm{GMI}\left(X\right)\approx 1$. Intermediate angular configurations (e.g., vectors separated by $\approx {45}^{\circ }$) yield values such as $\mathrm{psin}\approx 0.71$ and $\mathrm{GMI}\approx 0.29$, as illustrated in Figure 2. In this sense, GMI is a normalized version of determinant-based multicollinearity measures (such as the generalized variance $det\left(G\right)$ or det of the correlation matrix), rescaled to lie in $[0,1]$ and made explicitly scale-invariant via the polar-sine normalization.

GMI provides a direct geometric diagnostic that complements classical VIF-type measures. Whereas VIF assesses how much the variance of each individual regression coefficient is inflated by linear dependence among the predictors, GMI treats the entire predictor set as a single geometric object and summarizes its angular degeneracy in a single number in $[0,1]$. Because it depends only on the angular configuration of the predictors and is invariant under independent rescaling of the columns, it can be compared meaningfully across differently scaled datasets.

3.2.2. Uniqueness of the Residual Projection

The multivector formulation $I-P=N{N}^{\top }$ expresses the residual projector in terms of any orthonormal basis of the orthogonal complement $\mathcal{C}{\left(X\right)}^{\perp }$. The next result confirms that this projector is independent of the specific basis chosen.

Remark 2 (Independence of the residual projector from the basis).

Let $X\in {\mathbb{R}}^{n\times p}$ have rank $r<n$, and let $N,\tilde{N}\in {\mathbb{R}}^{n\times (n-\rho )}$ have orthonormal columns spanning $\mathcal{C}{\left(X\right)}^{\perp }$. Then there exists an orthogonal $Q\in {\mathbb{R}}^{(n-\rho )\times (n-\rho )}$ such that $\tilde{N}=NQ$, and hence

$$\tilde{N}{\tilde{N}}^{\top }=NQ{Q}^{\top }{N}^{\top }=N{N}^{\top }.$$

In particular, the representation $I-P=N{N}^{\top }$ from Theorem 2 is independent of the orthonormal basis chosen of $\mathcal{C}{\left(X\right)}^{\perp }$: it is simply the (unique) orthogonal projector onto $\mathcal{C}{\left(X\right)}^{\perp }$.

3.3. Generalized Projection in Rank-Deficient Spaces via Null-Spaces and Wedge Products

When the design matrix X is rank-deficient or when $n-\rho >1$, the orthogonal complement $\mathcal{C}{\left(X\right)}^{\perp }$ has dimension greater than one, and a single normal vector—as in the rank-one case—is no longer sufficient to describe the residual direction. In such settings, a full orthonormal basis of $\mathcal{C}{\left(X\right)}^{\perp }$ is required. This subsection explains how standard null-space computations, combined with QR orthonormalization, yield such a basis, and how wedge products provide a coordinate-free way to reason about independence and volume.

Definition 2 (Residual Projection in the Rank-Deficient Case).

Let $X\in {\mathbb{R}}^{n\times p}$ have rank $r<n$, and set $k=n-\rho =dim\mathcal{C}{\left(X\right)}^{\perp }$. If $N\in {\mathbb{R}}^{n\times k}$ has orthonormal columns spanning $\mathcal{C}{\left(X\right)}^{\perp }$, then the residual projector is

$$I-P=N{N}^{\top }.$$

Proposition 5 (Limitation of the cross-product).

The cross-product of Belhaouari [7] produces a single vector orthogonal to $n-1$ independent vectors in ${\mathbb{R}}^n$. This suffices when $dim\mathcal{C}{\left(X\right)}^{\perp }=1$, but for $n-r>1$ it yields only one null direction. Additional directions are needed to span the entire residual space.

Proof. 

If $n-\rho =1$, then $\mathcal{C}{\left(X\right)}^{\perp }$ is one-dimensional, and a single cross-product applied to a basis of $\mathcal{C}\left(X\right)$ recovers the unique (up to sign) normal vector n, giving $I-P=n{n}^{\top }$ as in the rank-one case. When $n-\rho >1$, the orthogonal complement has dimension $k>1$, so multiple linearly independent null directions are required. A single cross-product produces at most one such direction and therefore cannot span a k-dimensional space. Additional vectors must be constructed, for instance, via null-space computations and subsequent orthonormalization.    □

Theorem 6 (Construction of a Residual Basis via null-space and QR).

Let $X\in {\mathbb{R}}^{n\times p}$ have rank r and set $k=n-\rho$ , so that $\mathcal{C}{\left(X\right)}^{\perp }=\mathcal{N}\left(X^{\top }\right)$ has dimension k. Then there exists an orthonormal basis $N\in {\mathbb{R}}^{n\times k}$ of $\mathcal{C}{\left(X\right)}^{\perp }$ such that

$$I-P=N{N}^{\top },$$

where P is the orthogonal projector onto $\mathcal{C}\left(X\right)$. Moreover, such an N can be obtained by the null-space + QR procedure in Algorithm 1.

Proof. 

By construction, the columns of B span $\mathcal{N}\left(X^{\top }\right)$, which is equal to $\mathcal{C}{\left(X\right)}^{\perp }$. If $b_1\wedge \dots \wedge b_k\ne 0$ then the $b_i$’s are linearly independent and form a basis of this k-dimensional subspace. The thin QR factorization $B=QR$ with orthonormal columns in Q then produces $N:=Q$ with $N^{\top }N=I_k$ and $\mathcal{C}\left(N\right)=\mathcal{C}\left(B\right)=\mathcal{C}{\left(X\right)}^{\perp }$. The matrix $N{N}^{\top }$ is therefore the orthogonal projector onto $\mathcal{C}{\left(X\right)}^{\perp }$, so $I-P=N{N}^{\top }$ by the uniqueness of the orthogonal projector onto a fixed subspace.    □

Algorithm 1 Residual Projector via Null-Space and QR (Explicit SVD/RRQR)

Remark 3.

In practice, the null-space basis in Step 1 is typically obtained from a numerically stable factorization such as SVD or a rank-revealing QR. The wedge product need not be computed explicitly; it serves primarily as a geometric way to interpret the independence check. When $dim\mathcal{C}{\left(X\right)}^{\perp }=1$, the construction reduces to the case of rank-one , where a normal vector can be obtained either from the null-space of $X^{\top }$ or by applying a cross-product to a basis of $\mathcal{C}\left(X\right)$.The numerical tolerance used for the determination of rank (e.g., $\rho =\tau ={10}^{-12}$ relative to the largest singular value) follows standard practice and does not affect the conceptual geometric results, which hold exactly at the algebraic level.

Remark 4 (Implementation note: computing $\mathcal{N}\left(X^{\top }\right)$ in floating-point arithmetic).

The instruction “compute a basis of $\mathcal{N}\left(X^{\top }\right)$” requires (i) a numerical rank decision and (ii) an explicit extraction rule. In this work we use either:

(a) SVD. Compute $X^{\top }=U\Sigma V^{\top }$, choose the numerical rank $\rho =\#\{i:{\sigma }_i\ge \tau {\sigma }_1\}$, and set $B=V(:,r+1:n)$.

(b) RRQR. Compute a column-pivoted QR factorization $X^{\top }\Pi =QR$, choose rank ρ from the diagonal of R (e.g., $|R_{ii}|\ge \tau |R_{11}|$), partition $R=\left[\begin{array}{cc}{R}_{11}& R_{12}\\ 0& R_{22}\end{array}\right]$, and set $B=\Pi \left[\begin{array}{c}-R_{11}^{-1}{R}_{12}\\ I_{n-\rho }\end{array}\right]$.

SVD is typically more robust under severe ill-conditioning, while RRQR is often faster when pivoted QR is sufficient. Row-reduction is used only for symbolic/toy demonstrations.

Figure 3 illustrates the residual subspace and two independent null directions before orthonormalization.

3.4. Projection Method Variants and Complexity

Definition 3 (Variants of Residual Projection).

Let $X\in {\mathbb{R}}^{n\times p}$ be a real-valued design matrix of rank $\rho <n$. The residual projection operator $I-P$ onto the orthogonal complement $\mathcal{C}{\left(X\right)}^{\perp }$ can be constructed by several exact procedures, each suited to different regimes of dimension, conditioning, and interpretability.

The following variants highlight that null-space, wedge-based, and determinant-based approaches provide geometrically transparent alternatives to inversion-based formulas in linear regression.

Proposition 7 (Wedge–QR Null-Space Variant).

Let $X\in {\mathbb{R}}^{n\times p}$ have rank ρ, and let $k=n-\rho$. Using the null-space + QR construction of Theorem 6, we obtain an orthonormal residual basis $N\in {\mathbb{R}}^{n\times k}$ spanning $\mathcal{C}{\left(X\right)}^{\perp }$ and hence the projector $I-P=N{N}^{\top }$. If, in addition, we compute a thin QR factorization $X=QR$, then the Gram matrix satisfies

$$G=X^{\top }X=R^{\top }R,det\left(G\right)=det{\left(R\right)}^2,$$

so the squared volume of the parallelotope spanned by the columns of X is given directly by $det{\left(R\right)}^2$. The general complexity of forming N and R is $\mathcal{O}(n{p}^2+nk)$.

Proof. 

The existence of an orthonormal basis N of $\mathcal{C}{\left(X\right)}^{\perp }$ and the identity $I-P=N{N}^{\top }$ follow from Theorem 6. The factorization $X=QR$ implies $G=X^{\top }X=R^{\top }R$, so $det\left(G\right)=det{\left(R\right)}^2$, which yields the stated volume formula. The complexity estimate is inherited from the null-space + QR construction.    □

When geometric interpretations involving volume are necessary-such as evaluating multicollinearity through the Geometric Multicollinearity Index (GMI)—factorizations of the Gram matrix are particularly convenient.

Proposition 8 (Cholesky-Based Volume and Projection).

Let $G=X^{\top }X$ and assume that X has full-column-rank so that $G\succ 0$. Compute the Cholesky factorization $G=L{L}^{\top }$, which yields

$$det\left(G\right)=det{\left(L\right)}^2=\prod _i{L}_{ii}^2.$$

Let N be any orthonormal basis of $\mathcal{C}{\left(X\right)}^{\perp }$ constructed as in Theorem 6. Then $I-P=N{N}^{\top }$ is the orthogonal residual projector. The dominant cost is $\mathcal{O}(p^3+nk)$: $\mathcal{O}\left(p^3\right)$ for the Cholesky factorization and $\mathcal{O}\left(nk\right)$ for forming and applying N.

Proof. 

The positive definiteness of G ensures the existence of a unique Cholesky factor L with positive diagonal entries, and the determinant formula follows from $det\left(G\right)=det{\left(L\right)}^2={\prod }_i{L}_{ii}^2$. The construction of N and the identity $I-P=N{N}^{\top }$ is provided by Theorem 6; any such N produces an orthogonal projector onto $\mathcal{C}{\left(X\right)}^{\perp }$.    □

In the special case where the residual subspace is one-dimensional, the projector admits an especially simple representation.

Proposition 9 (cross-product (Rank-One Case)).

If $dim\left(\mathcal{C}{\left(X\right)}^{\perp }\right)=1$, then

$$I-P=n{n}^{\top },$$

(19)

where $n\in {\mathbb{R}}^n$ satisfies $X^{\top }n=0$ and $\parallel n\parallel =1$. The one-time cost of building n is $\mathcal{O}\left(n^2\right)$, and each subsequent projection $(I-P)Y$ costs $\mathcal{O}\left(n\right)$.

Proof. 

When $\mathcal{C}{\left(X\right)}^{\perp }$ is one-dimensional, any nonzero vector in this space is a scalar multiple of a unit normal n. The orthogonal projector onto the line spanned by n is precisely $n{n}^{\top }$. Symmetry and idempotence follow from $\parallel n\parallel =1$, and orthogonality is ensured by $X^{\top }n=0$.    □

Example 1.

Consider $n=4$, $p=3$, and the residual-codimension $k=n-p=1$. In this codimension-one setting, the classical OLS residual projector based on ${\left(X^{\top }X\right)}^{-1}$ (assuming that X has full column-rank) involves a matrix inversion together with several matrix–vector multiplications, which require on the order of a few hundred floating-point operations in this small example. By contrast, computing a unit normal vector n via the geometric cross-product costs $\mathcal{O}\left(n^2\right)$ operations, and each application of the rank-one projector $n{n}^{\top }$ to a new response Y costs only $\mathcal{O}\left(n\right)$ operations. Once n has been constructed, the projection of additional responses is therefore substantially cheaper than recomputing residuals through ${\left(X^{\top }X\right)}^{-1}$, illustrating the efficiency of geometric formulation in symbolic or resource-limited regimes.

Unless stated otherwise, we use a single Gaussian sketch without power iterations or oversampling; orthonormalization is performed via a standard thin QR factorization.

Algorithm 2: Sketch–QR Baseline (Gaussian Range Finder)

Table 1 compares four exact ways to build the residual projector $(I-P)$ for a given design matrix X. Classical OLS and the Cholesky-based variant both work through ${\left(X^{\top }X\right)}^{-1}$ and treat the residual subspace only implicitly, with a cubic dependence on the number of predictors p and the restriction that $X^{\top }X$ be invertible. In contrast, the Wedge–QR and rank-one cross-product methods construct $(I-P)$ from an explicit basis of $\mathcal{C}{\left(X\right)}^{\perp }$; the former handles general rank-deficient cases, while the latter is tailored to the codimension-one setting and achieves the lowest per-response cost when many responses must be projected.

In summary, Wedge–QR and Cholesky-based constructions are well-suited when one needs both an exact residual projector and access to volume-based quantities such as $det\left(X^{\top }X\right)$ or the GMI. The formulation of a rank-one cross-product is preferable whenever $dim\mathcal{C}{\left(X\right)}^{\perp }=1$, as it produces a simple geometric projector $n{n}^{\top }$ with linear complexity per response and, when $p\ll n$, is usually the cheapest option to repeatedly project many responses. For general rank-deficient designs, a QR/null-space construction offers a numerically safer alternative to explicitly forming and inverting $X^{\top }X$. Together, the four variants in Table 1 provide a practical menu of exact geometric projectors that avoid unnecessary matrix inversion and can be chosen according to the rank of X, the conditioning of $X^{\top }X$ and the available computational budget.

We emphasize that the exact residual projector $I-P$ can be constructed via several algebraically equivalent routes (normal-equations OLS when applicable, rank-one cross products, null-space/QR, wedge-based constructions and Cholesky-based variants), which differ only in numerical assumptions and computational cost; see Table 1.

4. Illustrative Numerical Examples

When the design matrix X is rank-deficient, the normal equations $X^{\top }X\beta =X^{\top }Y$ do not admit a unique solution. In all rank-deficient experiments, we therefore compute the minimum-norm least-squares solution using the Moore–Penrose pseudoinverse, i.e., $\widehat{\beta }=X^{+}Y$, and refer to this baseline as OLS (Moore–Penrose pseudoinverse).

Evaluation protocol. Predictive performance is assessed using K-fold cross-validation with $K=5$, reporting CV MSE and CV $R^2$. Unless stated otherwise, all randomized procedures use a fixed random seed (seed = 42) to ensure reproducibility.

Coefficient-stability summary. Coefficient stability is evaluated via $B=100$ perturbation trials with the design matrix X held fixed. In each trial, the response is perturbed as

$${\tilde{y}}^{\left(b\right)}=y+{\eta }^{\left(b\right)},{\eta }^{\left(b\right)}\sim \mathcal{N}\left(0,{\sigma }_y^{2}I\right),{\sigma }_y=0.01\mathrm{std}\left(y\right),b=1,\dots ,B.$$

The model is refit for each trial to obtain coefficients ${\widehat{\beta }}^{\left(b\right)}$. Let ${\mathrm{SD}}_j$ denote the sample standard deviation of the jth coefficient ${\left\{{\widehat{\beta }}_j^{\left(b\right)}\right\}}_{b=1}^B$ across trials. We summarize coefficient stability by

$$\mathrm{medianSD}\left(\widehat{\beta }\right):={\mathrm{median}}_{j=1,\dots ,p}\left({\mathrm{SD}}_j\right),$$

i.e., the median across coefficients of their across-trial standard deviations.

To validate the geometric projector, we verify agreement between the classical OLS residual $r_{\mathrm{OLS}}=(I-P)y$ and the geometric residual $r_{\mathrm{geo}}=N{N}^{\top }y$ by monitoring $\parallel r_{\mathrm{OLS}}-r_{\mathrm{geo}}{\parallel }_2$ up to numerical tolerance.

Sketch–QR [6] yields an approximate basis $\tilde{Q}$ and projector $\tilde{P}=\tilde{Q}{\tilde{Q}}^{\top }$, producing the approximate residual $\tilde{r}=(I-\tilde{P})y$. Agreement with the exact residual is summarized using cosine similarity $cos(r_{\mathrm{OLS}},\tilde{r})$ and the residual discrepancy $\parallel \tilde{r}-r_{\mathrm{OLS}}{\parallel }_2$ (and, when reported, a principal-angle distortion between $\mathcal{C}\left(X\right)$ and $\mathcal{C}\left(\tilde{Q}\right)$).

4.1. Rank-One Residual Projection in ${\mathbb{R}}^3$

This example explicitly shows how the normality of the cross-product reproduces the ordinary least-squares (OLS) residual in the rank-one case, in line with Proposition 1.

We consider a rank-one residual setting where the orthogonal complement of the column space is one-dimensional. Let us

$$\mathbf{X}=\left[\begin{array}{cc}1& 2\\ 1& 3\\ 1& 5\end{array}\right],\mathbf{Y}=\left[\begin{array}{c}1\\ 2\\ {\displaystyle \frac{11}{2}}\end{array}\right],$$

with $\mathbf{X}\in {\mathbb{R}}^{3\times 2}$ of full-column-rank (so ${\mathbf{X}}^{\top }\mathbf{X}$ is invertible). The ordinary least-squares projection matrix is

$$\mathbf{P}=\mathbf{X}{\left({\mathbf{X}}^{\top }\mathbf{X}\right)}^{-1}{\mathbf{X}}^{\top },{\mathbf{r}}_{\mathrm{OLS}}=(\mathbf{I}-\mathbf{P})\mathbf{Y}.$$

A direct calculation yields

$${\mathbf{r}}_{\mathrm{OLS}}=\left[\begin{array}{c}{\displaystyle \frac{3}{14}}\\ -{\displaystyle \frac{9}{28}}\\ {\displaystyle \frac{3}{28}}\end{array}\right],{\mathbf{r}}_{\mathrm{OLS}}\in \mathcal{C}{\left(\mathbf{X}\right)}^{\perp }.$$

To recover the same residual in purely geometric form, we construct a unit normal vector to the column space of $\mathbf{X}$. Let us

$${\mathbf{x}}_1=\left[\begin{array}{c}1\\ 1\\ 1\end{array}\right],{\mathbf{x}}_2=\left[\begin{array}{c}2\\ 3\\ 5\end{array}\right],{\mathbf{n}}_0={\mathbf{x}}_1\times {\mathbf{x}}_2=\left[\begin{array}{c}2\\ -3\\ 1\end{array}\right],\mathbf{n}={\displaystyle \frac{{\mathbf{n}}_0}{\parallel {\mathbf{n}}_0\parallel }}={\displaystyle \frac{1}{\sqrt{14}}}\left[\begin{array}{c}2\\ -3\\ 1\end{array}\right].$$

Then ${\mathbf{X}}^{\top }\mathbf{n}=0$ and $\parallel \mathbf{n}\parallel =1$, so Proposition 1 applies, and

$$\mathbf{I}-\mathbf{P}=\mathbf{n}{\mathbf{n}}^{\top }.$$

The residual can therefore be written as the rank-one projection

$$\mathbf{r}=(\mathbf{I}-\mathbf{P})\mathbf{Y}=\mathbf{n}{\mathbf{n}}^{\top }\mathbf{Y}=\left({\mathbf{n}}^{\top }\mathbf{Y}\right)\mathbf{n}=\left[\begin{array}{c}{\displaystyle \frac{3}{14}}\\ -{\displaystyle \frac{9}{28}}\\ {\displaystyle \frac{3}{28}}\end{array}\right],$$

which coincides with ${\mathbf{r}}_{\mathrm{OLS}}$ and lies on the line spanned by $\mathbf{n}$.

From a computational point of view, this example highlights the attraction of the geometric formulation in low dimension: the cross-product and the multiplication of a scalar vector are sufficient to obtain the residual, avoiding explicit matrix inversion. Once $\mathbf{n}$ is known, applying the projector $(\mathbf{I}-\mathbf{P})=\mathbf{n}{\mathbf{n}}^{\top }$ to a new response vector reduces to one dot product and one scaled copy of $\mathbf{n}$, i.e., $\mathcal{O}\left(n\right)$ work per residual.

4.2. Multivector Residual Projection in ${\mathbb{R}}^4$

This example illustrates the multivector residual projection via an orthonormal basis of $\mathcal{C}{\left(\mathbf{X}\right)}^{\perp }$ as in Theorem 6, and links it to the Geometric Multicollinearity Index.

We now consider a multivector setting in which the residual subspace has a dimension greater than one. Let us

$$\mathbf{X}=\left[\begin{array}{cc}1& 1\\ 1& 2\\ 1& 3\\ 1& 4\end{array}\right],\mathbf{Y}=\left[\begin{array}{c}1\\ 2\\ 3\\ 5\end{array}\right],\mathbf{X}\in {\mathbb{R}}^{4\times 2},\rho =2.$$

The ordinary least-squares projector is again

$$\mathbf{P}=\mathbf{X}{\left({\mathbf{X}}^{\top }\mathbf{X}\right)}^{-1}{\mathbf{X}}^{\top },{\mathbf{r}}_{\mathrm{OLS}}=(\mathbf{I}-\mathbf{P})\mathbf{Y},$$

and a straightforward computation gives

$${\mathbf{r}}_{\mathrm{OLS}}=\left[\begin{array}{c}{\displaystyle \frac{1}{5}}\\ -{\displaystyle \frac{1}{10}}\\ -{\displaystyle \frac{2}{5}}\\ {\displaystyle \frac{3}{10}}\end{array}\right].$$

Since $\rho =2$, its orthogonal complement has dimension $dim\mathcal{C}{\left(\mathbf{X}\right)}^{\perp }=2$. A basis for the null-space of ${\mathbf{X}}^{\top }$ is

$${\mathbf{n}}_1=\left[\begin{array}{c}1\\ -2\\ 1\\ 0\end{array}\right],{\mathbf{n}}_2=\left[\begin{array}{c}2\\ -3\\ 0\\ 1\end{array}\right],{\mathbf{X}}^{\top }{\mathbf{n}}_i=0,i=1,2.$$

Applying Gram–Schmidt yields an orthonormal basis

$${\mathbf{u}}_1={\displaystyle \frac{1}{\sqrt{6}}}\left[\begin{array}{c}1\\ -2\\ 1\\ 0\end{array}\right],{\mathbf{u}}_2={\displaystyle \frac{1}{\sqrt{30}}}\left[\begin{array}{c}2\\ -1\\ -4\\ 3\end{array}\right],\mathbf{N}=\left[{\mathbf{u}}_1{\mathbf{u}}_2\right]\in {\mathbb{R}}^{4\times 2},$$

with ${\mathbf{N}}^{\top }\mathbf{N}={\mathbf{I}}_2$ and ${\mathbf{X}}^{\top }\mathbf{N}=0$. The general residual projection theorem then gives

$$\mathbf{I}-\mathbf{P}=\mathbf{N}{\mathbf{N}}^{\top },{\mathbf{r}}_{\mathrm{geom}}=\mathbf{N}{\mathbf{N}}^{\top }\mathbf{Y}.$$

Evaluating ${\mathbf{r}}_{\mathrm{geom}}$ for this $\mathbf{X}$ and $\mathbf{Y}$ reproduces the residual OLS:

$${\mathbf{r}}_{\mathrm{geom}}={\mathbf{r}}_{\mathrm{OLS}}=\left[\begin{array}{c}{\displaystyle \frac{1}{5}}\\ -{\displaystyle \frac{1}{10}}\\ -{\displaystyle \frac{2}{5}}\\ {\displaystyle \frac{3}{10}}\end{array}\right],$$

making explicit that the residual lives in the two-dimensional subspace spanned by ${\mathbf{u}}_1$ and ${\mathbf{u}}_2$ and realizing the null-space + QR construction in Theorem 6, in a concrete low-dimensional setting.

From a complexity point of view, building $\mathbf{N}$ on a null-space basis followed by QR (or Gram–Schmidt) is a one-off cost of order $\mathcal{O}(n{p}^2+nk)$ for $\mathbf{X}\in {\mathbb{R}}^{n\times p}$ with residual dimension $k=n-\rho$, while applying the projector $(\mathbf{I}-\mathbf{P})=\mathbf{N}{\mathbf{N}}^{\top }$ to a new response vector requires only $\mathcal{O}\left(nk\right)$ operations.

4.2.0.1. GMI for the ${\mathbb{R}}^4$ Example

For this design matrix $\mathbf{X}=[{\mathbf{x}}_1,{\mathbf{x}}_2]$, the Gram matrix is

$$\mathbf{G}={\mathbf{X}}^{\top }\mathbf{X}=\left[\begin{array}{cc}4& 10\\ 10& 30\end{array}\right],det\left(\mathbf{G}\right)=20.$$

The column norms are $\parallel {\mathbf{x}}_1\parallel =2$ and $\parallel {\mathbf{x}}_2\parallel =\sqrt{30}$, so the polar sine and the Geometric Multicollinearity Index are

$$\mathrm{psin}({\mathbf{x}}_1,{\mathbf{x}}_2)={\displaystyle \frac{\sqrt{det\left({\mathbf{X}}^{\top }\mathbf{X}\right)}}{\parallel {\mathbf{x}}_1\parallel \parallel {\mathbf{x}}_2\parallel }}={\displaystyle \frac{\sqrt{20}}{2\sqrt{30}}}={\displaystyle \frac{\sqrt{6}}{6}},\mathrm{GMI}=1-{\displaystyle \frac{\sqrt{6}}{6}}\approx 0.59.$$

In this small example, the GMI value $\mathrm{GMI}\approx 0.59$ indicates a moderate degree of collinearity between the two regressors; the residual subspace $\mathcal{C}{\left(\mathbf{X}\right)}^{\perp }$ has dimension two (as $n-r=2$), and, compared to the orthonormal case ($\mathrm{GMI}=0$), the fitted coefficients are more sensitive to perturbations of $\mathbf{Y}$.

This example isolates the effect of controlled collinearity on GMI in a minimal symbolic setting.

Consider the family of matrices

$${\mathbf{X}}_{\epsilon }=\left[\begin{array}{cc}1& 0\\ 0& 1\\ \epsilon & \epsilon \end{array}\right],\epsilon \in [0,1].$$

(20)

For every $\epsilon \in [0,1]$ the two columns are linearly independent, so ${\mathbf{X}}_{\epsilon }^{\top }{\mathbf{X}}_{\epsilon }$ is positive definite, and the polar-sine is well-defined. A direct computation gives

$$\mathrm{psin}\left({\mathbf{X}}_{\epsilon }\right)={\displaystyle \frac{\sqrt{1+2{\epsilon }^2}}{1+{\epsilon }^2}},$$

(21)

and hence

$$\mathrm{GMI}\left(\epsilon \right)=1-{\displaystyle \frac{\sqrt{1+2{\epsilon }^2}}{1+{\epsilon }^2}}.$$

(22)

As $\epsilon$ increases, the columns become more collinear, $\mathrm{psin}$ decreases, and $\mathrm{GMI}$ increases, reflecting the growing multicollinearity in a purely geometric way. In families where columns eventually become exactly collinear, the Gram determinant vanishes, the polar sine drops to zero, and $\mathrm{GMI}$ reaches 1, in agreement with the rank-deficient discussion in Section 3.

4.2.0.2. Large-Scale Multivector Residual Projection

To complement the low-dimensional symbolic example above, we consider a larger synthetic design in which the residual subspace has a dimension greater than one and the multivector formulation $I-P=N{N}^{\top }$ is essential.

We generate a rank-deficient design matrix $X\in {\mathbb{R}}^{n\times p}$ with $n=1000$, $p=50$, and $\rho =45$, so that $dim\mathcal{C}{\left(X\right)}^{\perp }=n-\rho =955$. All predictors are standardized, and the response is constructed so that the exact residual $r_{\mathrm{exact}}$ has a prescribed norm, ensuring a controlled comparison between methods.

Table 2 reports residual norms, cosine similarity to the exact residual, and run times for the variants of the Article 1 projector only. The exact constructions (null-space/QR, rank-one cross-product when applicable, and the Moore–Penrose pseudoinverse baseline for OLS) reproduce the same residual up to numerical precision, confirming the multivector identity $I-P=N{N}^{\top }$ in a high-dimensional setting.

Sketch–QR yields an approximate residual, whose agreement with the exact residual is summarized by cosine similarity.Entries marked NaN or N/A indicate methods that are not applicable in this regime (e.g., rank-one cross-product when $dim\mathcal{C}{\left(X\right)}^{\perp }\ne 1$), while FAIL indicates that the required algebraic conditions (e.g., $X^{\top }X\succ 0$) are violated.

This experiment demonstrates that the geometric residual formulation extends naturally from symbolic low-dimensional examples to realistic large-n, rank-deficient designs, without relying on regularization or nonlinear modeling.

Together with the symbolic R⁴ example, this large-scale synthetic experiment confirms that the multivector residual projector $N{N}^{\top }$ is exact in dimensions, while Sketch–QR provides a principled approximation when a reduced basis is desired, setting the stage for the real-data illustrations in Section 4.3.

4.3. Illustrative Real Data Use of GMI

Real-data experiments use two regression benchmarks: Boston Housing and Auto MPG. Both are standardized via centering and scaling before forming X, and they provide realistic multicollinearity patterns suitable for illustrating GMI as a purely geometric, response-independent diagnostic.

This example illustrates how GMI can serve as a purely geometric multicollinearity diagnostic on real data and how it can be interpreted alongside coefficient stability.

We use the Boston Housing dataset because it is a widely used regression benchmark with well-known predictor correlations, making it a convenient setting for illustrating multicollinearity diagnostics and residual geometry on realistic data.

To illustrate the geometric diagnostics on real data, we use the classical Boston Housing dataset, with median house price as the response and a set of standardized socio-economic and structural predictors. The protocol used in this subsection is as follows:

(a)

Standardise all predictors to zero mean and unit variance, and form the design matrix X.

(b)

For selected subsets of predictors, form the corresponding design matrix X, compute the Gram matrix $G=X^{\top }X$, and evaluate the polar sine and GMI using the definitions in (18).

(c)

Report predictive metrics using 5-fold cross-validation (CV) so that MSE and $R^2$ are out-of-sample summaries rather than in-sample fit statistics.

(d)

Fit ordinary least-squares models on these subsets and monitor the variation of the estimated coefficients under small perturbations of Y (e.g., additive Gaussian noise or resampling).

The purpose of this subsection is to demonstrate (i) that the geometric residual projector $(I-P)=N{N}^{\top }$ reproduces the classical OLS residuals on a realistic dataset, and (ii) that GMI provides a compact, response-independent summary of predictor-space degeneracy.

4.3.0.3. Boston Housing and Auto MPG: Increasing Model Size

Real-data experiments use two regression benchmarks: Boston Housing and Auto MPG. Both are standardized through centering and scaling before forming X, and provide realistic multicollinearity patterns suitable to illustrate GMI as a purely geometric and response-independent diagnostic.

Table 3 and Table 4 report results for increasing the size of the model ($p=1$, $p=4$, $p\ge 6$, and in full where applicable). For each subset, we report out-of-sample predictive metrics using 5-fold cross-validation (CV MSE and CV $R^2$), geometric multicollinearity (GMI), and classical diagnostics (max VIF and $\kappa \left(X^{\top }X\right)$), along with a summary of coefficient-stability.

We restrict algorithmic comparisons to the projector variants defined in Table 1 (OLS/Cholesky when applicable, null-space/QR, rank-one cross-product when $\mathrm{codim}=1$, and Sketch–QR).

Table 3Table 4 show that increasing the size of the model improves predictive performance (lower CV MSE, higher CV $R^2$) while typically increasing multicollinearity. In Boston, moving from $p=1$ to the full model reduces CV MSE from $44.02$ to $23.67$ and increases CV $R^2$ from $0.467$ to $0.712$, while GMI increases from 0 to $0.988$ and $\kappa \left(X^{\top }X\right)$ increases from 1 to $96.47$. In Auto MPG the collinearity is even stronger: the full model attains CV MSE $11.31$ and CV $R^2$ $0.810$, but the corresponding diagnostics are GMI $\approx 0.980$, max VIF $\approx 21.84$, and $\kappa \left(X^{\top }X\right)\approx 136.84$.

These results illustrate the intended role of GMI: it is computed purely from X (independent of Y) and tracks familiar multicollinearity measures (VIF and condition number), while the coefficient-stability summary provides a concrete link between predictor-space degeneracy and sensitivity of fitted parameters.

4.3.0.4. Implementation Note (Stable GMI Computation)

When the Gram matrix $G=X^{\top }X$ is poorly conditioned, the evaluation of direct determinants is numerically fragile. We therefore compute the polar sine and GMI in the log-domain with a small jitter $\epsilon >0$:

$$log{\mathrm{psin}}_{\epsilon }\left(G\right)={\displaystyle \frac{1}{2}}logdet(G+\epsilon I)-{\displaystyle \frac{1}{2}}\sum _{i=1}^{p}log(G_{ii}+\epsilon ),{\mathrm{GMI}}_{\epsilon }\left(G\right)=1-exp\left(log{\mathrm{psin}}_{\epsilon }\left(G\right)\right).$$

In implementation, $logdet(G+\epsilon I)$ is evaluated using a numerically stable log-determinant routine (equivalently, a Cholesky factorization when $G+\epsilon I\succ 0$). Unless stated otherwise, we use $\epsilon ={10}^{-12}$.

Figure 4 makes the equivalence of the exact geometric projector and classical OLS residuals explicit: the discrepancy curve for the exact QR-basis construction is at a numerical noise level in both the $p=1$ and $p=4$ models. In contrast, Sketch–QR constructs an approximate projector $\tilde{P}$, so its residual $\tilde{r}$ differs slightly from $r_{\mathrm{OLS}}$; the magnitude of this discrepancy decreases as the size of the sketch s increases. Overall, this visualization complements Table 3Table 4 by separating the exact residual-geometry identities (Article 1 methods) from the approximate Sketch–QR construction while keeping the presentation compact and reproducible.

5. Discussion

This paper presents a rank-aware geometric interpretation of linear regression residuals via orthogonal projectors. In the case of codimension-one, the residual is fully determined by a unit normal $n\in \mathcal{C}{\left(X\right)}^{\perp }$, resulting in $r=(I-P)Y=n{n}^{\top }Y$; in general rank settings, the residual lies in a higher-dimensional orthogonal complement spanned by an orthonormal basis N, with $(I-P)=N{N}^{\top }$. This formulation makes the residual subspace explicit and decouples basis construction from repeated projection.

An immediate consequence is that all exact computational routes—normal-equation OLS, Cholesky-based solvers, and null-space or QR constructions—must yield identical residuals up to numerical tolerance, as they implement the same projector onto $\mathcal{C}{\left(X\right)}^{\perp }$. We confirm this numerically through residual agreement metrics and discrepancy visualizations, showing that differences arise only from conditioning and the computational regime. In contrast, Sketch–QR yields an approximate projector and residual, whose deviation from the exact solution is quantified using cosine similarity and norm-based discrepancies.

We further link residual geometry to predictor-space conditioning through the Geometric Multicollinearity Index (GMI), a scale-invariant, normalized volume measure derived from the polar sine. Across the Boston Housing and Auto MPG benchmarks, higher GMI values consistently align with larger max VIF and $\kappa \left(X^{\top }X\right)$, while coefficient-stability summaries illustrate increased parameter sensitivity under geometric degeneracy.

Although both regression and PCA rely on orthogonal projections, their objectives differ fundamentally: PCA projects predictors to minimize reconstruction error, with residuals remaining in $\mathcal{C}\left(X\right)$, whereas regression projects the response, yielding residuals in $\mathcal{C}{\left(X\right)}^{\perp }$.

Finally, this work focuses on unregularized least squares and exact orthogonal projection. With regularization, the operators are no longer orthogonal and the residual subspace becomes non-Euclidean or data-adaptive, motivating future work on regularized and sketch-aware residual geometry while preserving the geometric diagnostics enabled by GMI.

6. Conclusions

This paper presented a rank-aware geometric interpretation of linear regression residuals. In the case of codimension-one, the residual projector reduces to the rank-one form $(I-P)=n{n}^{\top }$, where n spans $\mathcal{C}{\left(X\right)}^{\perp }$. For general rank, the residual lies in a $k=n-\rho$ dimensional orthogonal complement, with a projector $(I-P)=N{N}^{\top }$ for an orthonormal basis N of $\mathcal{C}{\left(X\right)}^{\perp }$. Although algebraically equivalent to the classical OLS formulation, this representation makes the residual subspace explicit and decouples basis construction from repeated projection.

Building on this viewpoint, we introduced the Geometric Multicollinearity Index (GMI), a scale-invariant diagnostic derived from the polar sine and Gram determinant that quantifies predictor-space degeneracy. Synthetic experiments confirm predictable behavior under controlled perturbations, while real-data benchmarks on Boston Housing and Auto MPG show that increasing model size typically improves predictive performance while increasing multicollinearity; GMI captures this trade-off and aligns with max VIF and $\kappa \left(X^{\top }X\right)$. Across all experiments, exact projector constructions yield identical residuals up to numerical tolerance, whereas Sketch–QR produces a controlled approximation whose deviation is quantified via cosine similarity and residual discrepancy norms.

In general, these results elevate $\mathcal{C}{\left(X\right)}^{\perp }$ to a central geometric object in regression analysis and position GMI as a concise indicator of departure from orthogonal designs. Extensions to sketch-based, large-scale, streaming, and nonlinear settings are deferred to companion work.