Non-Gradient Quaternion Training Matrix Modifications for Colored Images Distillation

1. Introduction

Deep neural networks (NNs) are difficult to train efficiently in the face of the exponential growth of data in modern machine learning (ML). Large-scale image databases have a huge demand for computational resources, including training time that may take days and memory requirements that exceed the standard hardware capacity [1,2]. To create a small but highly informative set of training samples that contains most of the important information of the original large dataset so that any model trained on those distilled images can achieve comparable results to training with the full dataset [3], the data distillation method is one of the most prominent solutions. Such a solution basically creates a small number of training samples, which saves computational time and memory and also makes the machine learning more efficient in resource-limited environments.

Note that gradient-based methods rely on NNs to extract information such as gradients and network parameters from the entire original dataset, which are then transferred back to another NN to produce the distilled images [4]. As a result, the distillation process itself is computationally expensive and time-consuming.

In contrast, the proposed new method is based on a modification of the PCA (M-PCA) and the SVD. Hence, we will denote it as M-PCA-SVD for Grey-scale images and QM-PCA-QSVD (Quaternion Modified PCA and Quaternion SVD) for colored images. Note that this distillation method is non-gradient and operates entirely through closed-form algebraic operations, requiring no NN training during distillation. This makes it significantly faster than gradient-based approaches, while still producing a compact and informative distilled dataset.

The present work directly extends the PCA-based framework designed to distill vectors from a set of vectors [5]. The new method works class-wise and first adapts the M-PCA to handle images by presenting the set of training images of a class as a single matrix. Further, this matrix is decomposed by the SVD. Then the left singular matrix is block-wise rotated, and the modified matrix is reconstructed. At the end, the upper m (user-selected number) rows of the reconstructed matrix are selected and transformed to images, which we call distilled images. Figure 1 presents a step-by-step visual pipeline of our proposed M-PCA-SVD method. For simplicity, we only presented the flowchart for grayscale image distillation, which is logically similar to the quaternion method for colored images. Examples of such images are presented in Figures 2–5.

The main novelty of the present new distillation method is:

• We developed the first method that combines PCA and SVD, along with their quaternion forms, to work together for grey-level and colored image distillation.

Another particular novelty is:

• The presentation of the entire training set as a single quaternion matrix, which is first modified by the M-PCA and, for the second time, by block-rotating the left singular matrix of the SVD for grey-level images and QSVD for color images.

The advantages that come from the novelties are significant savings of runtime at the expense of a small decrease in the classification accuracy (see Table 3). In addition, the new method outperforms all competitors when applied on CIFAR-10 and CIFAR-100.

The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 details the proposed framework, while Section 4 describes the CNNs and image datasets used for experimental validation. Section 5 presents experimental results. Section 6 discusses theoretical properties and implications of the M-PCA-SVD (QM-PCA-QSVD) method, and the paper concludes with Section 7, where future works are presented as well.

2. Related Work

2.1. Gradient-Based Dataset Distillation

Existing dataset distillation methods can be broadly divided into gradient-based and non-gradient-based approaches. Gradient-based methods form the dominant paradigm, involving gradient information derived from an NN used during the distillation process itself. Li et al. [6] proposed Importance-Aware Adaptive Dataset Distillation (IADD), which automatically assigns adaptive importance weights to network parameters during gradient-based distillation, producing robust synthetic datasets with superior cross-architecture generalization. Su et al. [7] developed D⁴M method, which uses a pretrained latent diffusion model with prototype learning and applies gradient-based training-time matching to align distilled and original data distributions at ImageNet scale. Ezekwem and Sirakov [8] extract the $L_1$ norm of the loss function gradient after CNN training to modify images pixel-wise, then apply K-means clustering to select centroids as distilled images, outperforming KIP [9] and trajectory-based methods on CIFAR-10. Liu et al. [10] proposed DREAM, a representative matching strategy that selects only the most representative original images for gradient, distribution, or trajectory matching, reducing distillation iterations by more than eight times without performance drop. Sucholutsky and Schonlau [11] developed a soft-label dataset distillation, assigning to each synthetic sample a distribution of labels rather than a single hard label, achieving over $96\%$ accuracy on MNIST with only ten distilled images per class and extending distillation to text data. Bohdal et al. [12] studied label distillation, optimizing synthetic labels for fixed real images rather than synthesizing images, and showed it improves flexibility and compatibility with diverse network architectures and off-the-shelf optimizers. Li et al. [13] applied gradient-based soft-label distillation to medical imaging, compressing gastric X-ray datasets into tiny anonymous distilled sets to enable privacy-preserving medical data sharing.

Another set of gradient-based methods is those based on the Ridge Regression. Such methods are developed in [9], which presents the fundamental Kernel Inducing Point (KIP) method that learns the support dataset via gradient descent. Further, in [14], the authors developed DP-KIP-ScatterNet, which uses the wavelet features from ScatterNet method instead of those from Convolutional Neural Tangent Kernels.

2.2. Knowledge Distillation and Teacher-Student Models

A related but distinct concept is knowledge distillation, which refers to model compression through teacher-student networks, where a smaller student network is trained to replicate the behavior of a larger pretrained teacher network by learning from its soft output representations at every level [15]. While knowledge distillation compresses the model, dataset distillation compresses the data; the two approaches are complementary and can be combined. Building on this teacher-student paradigm, Tan et al. [16] proposed Influence Matching (Inf-Match), which learns a synthetic dataset whose effect on the converged model parameters matches that of the full dataset, achieving state-of-the-art accuracy on Tiny-ImageNet [17] and scaling to vision-language distillation tasks. Xu et al. [18] addressed distribution shifts in diffusion-based dataset distillation by introducing an $L1$ sparsity regularizer during diffusion training and a synchronous denoising strategy during sampling, achieving state-of-the-art performance with superior computational efficiency.

2.3. Non-Gradient-Based Dataset Distillation

Non-gradient, non-NN-based methods construct distilled datasets through closed-form algebraic operations, without any optimization loop or backpropagation. Sirakov et al. [5] proposed a non-gradient method that applies PCA to the training vectors and uses the derived eigenvectors to distill and augment the original training set. They used nested double distillation and showed that PCA-distilled vectors give a significant boost to classification statistics on four public databases and four classifiers. Sirakov and Ngo [19] extended this algebraic approach to images. They combined Discrete Wavelet Transform with Modified PCA (M-PCA) and sparse representation to demonstrate the effectiveness of purely algebraic pipelines in reducing the size of the dataset while maintaining accuracy. Lowe et al. [20] showed that structured algebraic manipulation of internal data representations across multiple levels of abstraction can produce powerful, generalizable features, without requiring gradient-based optimization, using their Bootleg framework.

3. Proposed Method

In this section, we develop the proposed non-gradient image distillation framework. The method extends the M-PCA developed for vector data in [5] to image databases, supporting distillations from both grey-level and colored images. The pipeline proceeds in the following stages: first, a matrix containing all training images for a class is created. Every row of the matrix is a single image. The matrix is transformed by the M-PCA method. In the next stage, the transformed matrix is decomposed by the SVD method, and the left singular matrix is block-wise rotated. At the end, m (user-selected number) upper rows are mapped back to images and used as distilled.

3.1. Extension of M-PCA to Grey-Level Images

Consider an image training dataset containing c different classes of grey-level images. Denote an arbitrary grey-level image from class i as $I_{u\times w}^j$, where j is the consecutive index of the image and its size is $u\times w=k$. We map each image $I_{u\times w}^j$ to a vector ${\mathbf{v}}_j\in {\mathbb{R}}^k$, and by applying this mapping to all n images in class i we construct the training data matrix:

$${\mathbf{D}}_t={({\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_n)}^T\in {\mathbb{R}}^{n\times k}.$$

(1)

We apply the M-PCA method from [5] to ${\mathbf{D}}_t$ and obtain the modified matrix ${\Delta }_{n\times k}$, whose rows are modified training images belonging to the original k-dimensional feature space.

3.2. SVD-Based Distillation

In the next stage, we apply SVD [21] to the M-PCA-modified matrix ${\Delta }_{n\times k}$ obtained from Equation 1. Then we decompose the matrix as follows :

$${\Delta }_{n\times k}={\mathbf{L}}_{n\times r}{\Sigma }_{r\times r}{\mathbf{R}}_{r\times k}^T,$$

(2)

where $r\le min(n,k)$ is the rank of ${\Delta }_{n\times k}$. We retain the $m<r$ largest singular values by taking the m leftmost columns of ${\mathbf{L}}_{n\times r}$, keeping only the top $m\times m$ block of ${\Sigma }_{r\times r}$, and retaining the top m rows of ${\mathbf{R}}_{r\times k}^T$, yielding the low-rank approximation:

$${\Delta }_{n\times k}^m={\mathbf{L}}_{n\times r}^m{\Sigma }_{m\times r}^m{\left({\mathbf{R}}_{r\times k}^m\right)}^T.$$

(3)

3.3. Block Rotation

We then apply a structured block rotation $\rho$ to the left singular matrix ${\mathbf{L}}_{n\times r}^m$. The rotation matrix is:

$$\rho =\left(\begin{array}{cc}cos\theta & -sin\theta \\ sin\theta & cos\theta \end{array}\right),$$

(4)

and is applied block-wise by partitioning ${\mathbf{L}}_{n\times r}^m$ into consecutive $2\times 2$ sub-matrices. This yields the rotated reconstruction:

$${\left({\Delta }_{n\times k}^m\right)}^{{\rho }_L}={\left({\mathbf{L}}_{n\times r}^m\right)}^{\rho }{\Sigma }_{m\times r}^m{\left({\mathbf{R}}_{r\times k}^m\right)}^T.$$

(5)

Alternatively, the rotation may be applied to the right singular matrix:

$${\left({\Delta }_{n\times k}^m\right)}^{{\rho }_R}={\mathbf{L}}_{n\times r}^m{\Sigma }_{m\times r}^m{\left[{\left({\mathbf{R}}_{r\times k}^m\right)}^T\right]}^{\rho }.$$

(6)

An important property of the proposed method is that rotations applied to the left and right matrices, as well as rotations by different angles, produce distinct reconstructed images:

$${\left({\Delta }_{n\times k}^m\right)}^{{\rho }_L}\ne {\left({\Delta }_{n\times k}^m\right)}^{{\rho }_R},\mathrm{and}{\left({\Delta }_{n\times k}^m\right)}^{{\rho }_{1}L}\ne {\left({\Delta }_{n\times k}^m\right)}^{{\rho }_{2}L}\mathrm{if}{\rho }_1\ne {\rho }_2.$$

(7)

This property represents a key advantage of the M-PCA-SVD framework: a large number of distinct distilled image sets can be generated by varying the rotation angle $\theta$ and the choice of singular matrix, providing the opportunity to distill multiple sets of images from a single dataset without any gradient computation.

The upper m rows of the reconstructed matrix ${\left({\Delta }_{n\times k}^m\right)}^{{\rho }_L}$ or ${\left({\Delta }_{n\times k}^m\right)}^{{\rho }_R}$ form the final set of distilled grey-level images for class i. Note that the number m is user-selected.

Figure 2 and Figure 3 present the distillation results from the grey-level Digit-MNIST and Fashion-MNIST image datasets, respectively. In each figure, row 1 shows a randomly selected original image per class, while row 2 shows 10 images distilled by the M-PCA-SVD method from Digit-MNIST and Fashion-MNIST. A single image was randomly selected from every distilled set. The distilled images are generated entirely through closed-form algebraic operations applied per class, without any gradient computation or use of NN. As can be observed, the distilled images retain the essential structural and distinct features of the classes of the originals, confirming that M-PCA-SVD effectively captures the variation in the training data while significantly reducing the number of samples required for training.

3.4. Extension to Color Images via Quaternion PCA

In the above-presented section, we developed the M-PCA-SVD method to distill grey images. In the present section, we will develop a method for distilling color images. For this purpose, we present the PCA method in the quaternion algebra [22], and model a color image with the help of quaternion [23]:

$$\dot{\mathbf{q}}=q^1+q^2\mathbf{i}+q^3\mathbf{j}+q^4\mathbf{k}.$$

(8)

Assume now that the RGB model of a color image at pixel $(x,y)$ is presented as the following vector function, which we describe with a quaternion:

$$\dot{\mathbf{q}}=\mathbf{C}(\mathbf{x},\mathbf{y})=0+R(x,y)\mathbf{i}+G(x,y)\mathbf{j}+B(x,y)\mathbf{k}=q^R\mathbf{i}+q^G\mathbf{j}+q^B\mathbf{k}.$$

(9)

Further, we define a vector of quaternions $\dot{\mathbf{v}}={({\dot{\mathbf{q}}}_{\mathbf{1}},{\dot{\mathbf{q}}}_{\mathbf{2}},...,{\dot{\mathbf{q}}}_{\mathbf{M}})}^T={\dot{\mathbf{v}}}^{\mathbf{R}}\mathbf{i}+{\dot{\mathbf{v}}}^G\mathbf{j}+{\dot{\mathbf{v}}}^B\mathbf{k}$, where ${\dot{\mathbf{v}}}^{\mathbf{e}}={(q_1^e,q_2^e,...,q_M^e,)}^T$, where $q_m^e\in R$ and $e\in \{R,G,B\}$. Now, a matrix of quaternions (colored images) is created [23] as follows:

$$\dot{\mathbf{I}}(\mathbf{x},\mathbf{y})={({\dot{\mathbf{v}}}_{\mathbf{1}},{\dot{\mathbf{v}}}_{\mathbf{2}},...,{\dot{\mathbf{v}}}_{\mathbf{N}})}^T={\mathbf{V}}^{\mathbf{R}}\mathbf{i}+{\mathbf{V}}^{\mathbf{G}}\mathbf{j}+{\mathbf{V}}^{\mathbf{B}}\mathbf{k},$$

(10)

where ${\dot{\mathbf{v}}}^{\mathbf{e}}\in H^M$, while ${\mathbf{V}}^{\mathbf{e}}={({\mathbf{v}}_{\mathbf{1}}^{\mathbf{e}},{\mathbf{v}}_{\mathbf{2}}^{\mathbf{e}},...,{\mathbf{v}}_{\mathbf{N}}^{\mathbf{e}})}^T$, and $e\in \{R,G,B\}$.

Consider an image database with c different classes of color images and denote an arbitrary color image from class with ${\dot{\mathbf{I}}}_{\mathbf{u}\times \mathbf{w}}^{\mathbf{j}}$i, where j is the consecutive number of the image and $u\times w=k$. Denote the cardinality of the $i-th$ class with $\left|i\right|=n$. Map every image ${\dot{\mathbf{I}}}_{\mathbf{u}\times \mathbf{w}}^{\mathbf{j}}$ to a vector ${\dot{\mathbf{v}}}_{\mathbf{j}}\in R^{k\times 3}$, for $j=1,2,...,n$. Arrange the above quaternion vectors as the quaternion matrix ${({\dot{\mathbf{v}}}_{\mathbf{1}},{\dot{\mathbf{v}}}_{\mathbf{2}},...,{\dot{\mathbf{v}}}_{\mathbf{n}})}^T={\dot{\mathbf{D}}}_{\mathbf{t}}\in R^{n\times k\times 3}$ which yields the following quaternion presentation through real matrices:

$${\dot{\mathbf{D}}}_t={\mathbf{D}}_{\mathbf{t}}^{\mathbf{R}}\mathbf{i}+{\mathbf{D}}_{\mathbf{t}}^{\mathbf{G}}\mathbf{j}+{\mathbf{D}}_{\mathbf{t}}^{\mathbf{B}}\mathbf{l}.$$

(11)

Then, we denote the quaternion vector of the $l-th$ pixel (feature) of all training images of class i with ${\dot{\mathbf{f}}}_{\mathbf{l}}\in {\mathbf{R}}^{\mathbf{n}}$ and represent it as ${\dot{\mathbf{f}}}_{\mathbf{l}}={({\dot{\mathbf{x}}}_{\mathbf{1}\mathbf{l}},.....,{\dot{\mathbf{x}}}_{\mathbf{nl}})}^T$. Further, denote the mean of $\dot{{\mathbf{f}}_{\mathbf{l}}}$ with ${\mu }_l$, and the rank of $\dot{{\mathbf{D}}_{\mathbf{t}}}$ with $r\le min\{n,k\}$. At the first step of the method, we normalize every entry of a quaternion pixel (feature) vector using the quaternion mean of the vector’s entries as shown in Equation 12 below. It implies that the mean of every normalized quaternion pixel vector will become 0.

$${\dot{\mathbf{x}}}_{\mathbf{il}}\leftarrow {\dot{\mathbf{x}}}_{\mathbf{il}}-\dot{{\mu }_l},$$

(12)

In the pixel quaternion vectors normalizing Equation 12, $x_{il}$ is the $i-th$ entry, for $i=1,....,n$, value of the quaternion pixel (features) vector l, where $l=1,....,k$, and $\dot{{\mu }_l}$ is the quaternion mean of the quaternion pixel (features) vector. Further, we calculate the quaternion covariance matrix. Its entries ${\dot{\mathbf{F}}}_{\mathbf{pl}}$ are defined with the equation below, and they measure the statistical relationship between the training quaternion pixel (features) vectors in class i [24] :

$${\dot{\mathbf{F}}}_{\mathbf{pl}}={\displaystyle \frac{1}{n-1}}{\displaystyle \sum _{i=1}^n}({\dot{\mathbf{x}}}_{\mathbf{il}}-{\dot{\mu }}_l){({\dot{\mathbf{x}}}_{\mathbf{ip}}-{\dot{\mu }}_p)}^T.$$

(13)

Equation 13 yields the quaternion covariance matrix of the initial quaternion matrix ${\dot{\mathbf{D}}}_{\mathbf{t}}$:

$$\dot{\mathbf{F}}=\left\{{\dot{\mathbf{F}}}_{\mathbf{pl}}\right\},p=1,...,k,l=1,...,k.$$

(14)

From Equation 14 we determine the quaternion Eigenvectors ${\dot{\mathbf{u}}}_{\mathbf{l}}$ and the Eigenvalues ${\lambda }_l$:

$$\dot{\mathbf{F}}{\dot{\mathbf{u}}}_{\mathbf{l}}=\dot{{\lambda }_l}{\dot{\mathbf{u}}}_{\mathbf{l}},|\dot{\mathbf{F}}-\dot{\lambda }\dot{\mathbf{I}}|=0,(\dot{\mathbf{F}}-\dot{\lambda }\dot{\mathbf{I}}){\dot{\mathbf{u}}}_{\mathbf{l}}=\dot{\mathbf{0}}.$$

(15)

where $\dot{\mathbf{I}}$ denotes $k\times k$ quaternion identity matrix on whose main diagonal entires is located the quaternion $\dot{\mathbf{q}}=1\mathbf{i}+1\mathbf{j}+1\mathbf{k}$. Also, we denote with $\dot{\mathbf{0}}=0\mathbf{i}+0\mathbf{j}+0\mathbf{k}$. Now we create quaternion matrix with the help of the quaternion Eigenvectors ${\dot{\mathbf{u}}}_{\mathbf{l}}\in {\mathbf{R}}^{\mathbf{k}\times \mathbf{3}}$, calculated from Equation 15, arranging them in a decreasing order, from left to right, according to the quaternion Eigenvalues $\dot{{\lambda }_l}={\lambda }_l^R\mathbf{i}+{\lambda }_l^G\mathbf{j}+{\lambda }_l^B\mathbf{k}$, where ${\lambda }^e\in R$, for $e\in R,G,B$:

$$\dot{\mathbf{U}}=[{\dot{\mathbf{u}}}_{\mathbf{1}},{\dot{\mathbf{u}}}_{\mathbf{2}},...,{\dot{\mathbf{u}}}_{\mathbf{r}}].$$

(16)

Note that r is the rank of ${\mathbf{D}}_{\mathbf{t}}^{\mathbf{e}}$ from Equation 11, while ${\lambda }_l^e\in R$ is the $l-th$ Eigenvalue of the real matrix for $e\in \{R,G,B\}$. Further we project ${\dot{\mathbf{D}}}_{\mathbf{t}}$ onto the vector space spanned by the quaternion Eigenvectors $\dot{{\mathbf{u}}_{\mathbf{i}}},i=1,...,r$ [24]:

$$\dot{{\mathbf{y}}_{\mathbf{i}}}={\left({\dot{\mathbf{U}}}^T\dot{{\mathbf{v}}_{\mathbf{i}}}\right)}^T={\dot{{\mathbf{v}}_{\mathbf{i}}}}^T\dot{\mathbf{U}}.$$

(17)

where $\dot{{\mathbf{y}}_{\mathbf{i}}}\in {\mathbf{R}}^{\mathbf{r}\times \mathbf{3}}$ is the new low-dimensional representation of the i-th training quaternion vector $\dot{{\mathbf{v}}_{\mathbf{i}}}\in {\mathbf{R}}^{\mathbf{k}\times \mathbf{3}}$, which is a quaternion vector row in $\dot{{\mathbf{D}}_{\mathbf{t}}}$ with rank $r\le k$.

Note that the larger an Eigenvalue ${\dot{\lambda }}_l$ is, the stronger the corresponding Eigenvector $\dot{{\mathbf{u}}_{\mathbf{l}}}$ is. Thus, we pick from ${\dot{\mathbf{U}}}^{\mathbf{T}}$ the $m<r$ top quaternion Eigenvectors $\dot{{\mathbf{u}}_{\mathbf{l}}}$, place them as vector columns and create the quaternion matrix $\dot{\mathbf{T}}\in {\mathbf{R}}^{\mathbf{k}\times \mathbf{m}\times \mathbf{3}}$ which modifies the original quaternion matrix ${\dot{\mathbf{D}}}_{\mathbf{t}}$ into ${\dot{\mathcal{D}}}_{\mathbf{t}}\in {\mathbf{R}}^{\mathbf{n}\times \mathbf{m}\times \mathbf{3}}$. Hence, we determine the quaternion matrix that contains transformed (modified) quaternion rows, which represent the modified training color images of class i:

$${\dot{\mathbf{D}}}_{\mathbf{t}}\dot{\mathbf{T}}={\dot{\mathcal{D}}}_{\mathbf{t}}.$$

(18)

Further, using the transpose quaternion matrix ${\dot{\mathbf{T}}}^{\mathbf{T}}\in {\mathbf{R}}^{\mathbf{m}\times \mathbf{k}\times \mathbf{3}}$ we calculate :

$${\dot{\mathcal{D}}}_{\mathbf{t}}{\dot{\mathbf{T}}}^{\mathbf{T}}={\dot{\Delta }}_{\mathbf{t}},$$

(19)

such that $\dot{{\Delta }_{\mathbf{t}}}\in {\mathbf{R}}^{\mathbf{n}\times \mathbf{k}\times \mathbf{3}}$, and $\dot{{\Delta }_{\mathbf{t}}}\ne \dot{D_t}$. Hence $\dot{{\Delta }_{\mathbf{t}}}$ contains modified training color images.

3.5. Distillation from the Modified Training Color Images

Recall that in the case of grey level images, we applied the real-valued SVD on the modified by the M-PCA matrix of grey level training images ${\Delta }_{\mathbf{t}}$. In case of color images, the Quaternion M-PCA (QM-PCA), described above, generates the quaternion matrix of the modified color training images $\dot{{\Delta }_{\mathbf{t}}}$. Therefore, we will apply now the quaternion SVD (QSVD) described in [25] to receive:

$$\dot{{\Delta }_{\mathbf{t}}}={\dot{\mathbf{L}}}_{\mathbf{n}\times \mathbf{k}}^{\mathbf{m}}{\Sigma }_{\mathbf{r}\times \mathbf{r}}^{\mathbf{m}}{\left({\left({\dot{\mathbf{R}}}_{\mathbf{r}\times \mathbf{k}}^{\mathbf{m}}\right)}^{⊳}\right)}^{\mathbf{T}}.$$

(20)

In Equation 20, the notation ${\dot{\mathbf{L}}}_{\mathbf{n}\times \mathbf{k}}^{\mathbf{m}}$ describes quaternion left unitary matrix, while ${\left({\left({\dot{\mathbf{R}}}_{\mathbf{r}\times \mathbf{k}}^{\mathbf{m}}\right)}^{⊳}\right)}^{\mathbf{T}}$ is a quaternion right transpose unitary conjugate matrix. Hence, we may present the quaternion left unitary matrix in the form:

$${\dot{\mathbf{L}}}_{\mathbf{n}\times \mathbf{k}}^{\mathbf{m}}={\left({\mathbf{L}}_{\mathbf{n}\times \mathbf{k}}^{\mathbf{m}}\right)}_{\mathbf{i}}^{\mathbf{R}}+{\left({\mathbf{L}}_{\mathbf{n}\times \mathbf{k}}^{\mathbf{m}}\right)}_{\mathbf{j}}^{\mathbf{G}}+{\left({\mathbf{L}}_{\mathbf{n}\times \mathbf{k}}^{\mathbf{m}}\right)}_{\mathbf{k}}^{\mathbf{B}}.$$

(21)

Equation 21 implies that rotation $\rho$ on angle $\theta$ could be applied and described by an equation like Equation 5 in quaternion form:

$${\left({\dot{\Delta }}_{\mathbf{t}}\right)}^{\rho \mathbf{L}}={\left({\dot{\mathbf{L}}}_{\mathbf{n}\times \mathbf{k}}^{\mathbf{m}}\right)}^{\rho }{\Sigma }_{\mathbf{r}\times \mathbf{r}}^{\mathbf{m}}{\left({\left({\dot{\mathbf{R}}}_{\mathbf{r}\times \mathbf{k}}^{\mathbf{m}}\right)}^{⊳}\right)}^{\mathbf{T}}.$$

(22)

The user-selected top m rows of ${\left({\dot{\Delta }}_t\right)}^{\rho \mathbf{L}}$ constitute the set of distilled color images for class i. As in the grey-level case, different rotation angles and choices of singular matrix generate distinct distilled sets, providing a rich family of distilled sets from a single closed-form algebraic pipeline with no gradient computation required.

Figure 4, Figure 5 and Figure 6 present the distillation results for the colored CIFAR-10, CIFAR-100, and BloodMNIST image datasets, respectively. For Figure 4 and Figure 5, row 1 shows a randomly selected original colored image per class, and row 2 shows the visually closest distilled colored image. In Figure 6 shows four randomly selected original images from BloodMNIST are on the left, and on the right are their visually closest distilled images. All color images are processed using QM-PCA to preserve inter-image RGB relationships, followed by QSVD and block-wise rotation at angle $\theta$. Despite the higher complexity of color image data, the distilled images retain recognizable class-level visual structure, demonstrating that the proposed non-gradient, non-CNN algebraic framework generalizes effectively from grey-level to color image distillation.

4. Experimental Setup

In this section, we describe the datasets, network architectures, and training configurations used to evaluate and validate our dataset distillation pipeline.

4.1. Datasets

We validated our method on five benchmark public image datasets that are widely used in the dataset distillation literature:

• Digit-MNIST [26]: A grayscale dataset of handwritten digits (0–9), consisting of 60,000 training images and 10,000 test images, each of size $28\times 28$ pixels, distributed across 10 classes.
• Fashion-MNIST [27]: A grayscale dataset of clothing items (e.g., shirts, shoes, bags), with the same structure as Digit-MNIST: 60,000 training and 10,000 test images at $28\times 28$ pixels across 10 classes. It is considered a bit more challenging benchmark due to greater intra-class visual variation.
• CIFAR-10 [28]: A color image dataset of $32\times 32$ pixels comprising 50,000 training and 10,000 test images distributed across 10 classes (e.g., airplanes, birds, and cats). It’s higher visual complexity and the use of color makes the image database a significantly harder classification problem than the MNIST variants.
• CIFAR-100 [29]: A color image dataset sharing the same image size and overall structure as CIFAR-10, but spanning 100 fine-grained classes with 500 training images and 100 test images per class. Its large class count and limited per-class samples make it the most demanding benchmark considered in this work.
• BloodMNIST [30]: A color image dataset of $28\times 28$ pixels comprising 11,959 training, 1,712 validation, and 3,421 test images distributed across 8 classes (e.g., Basophils, Eosinophils, and Erythroblasts). It consists of microscopic images of human blood cells and is widely used for training and testing machine learning models.

For all datasets, classification accuracy is evaluated on the original, unmodified test set. Once the training is conducted on the entire original training dataset, the classifier is trained exclusively on the distilled images. For the two training strategies, the classification abilities are evaluated on the same unmodified test set.

4.2. CNN Architectures

For our experiments, we used two convolutional neural network (CNN) architectures based on the difficulty of the specific dataset:

• Baseline CNN [31]: A lightweight network used for classifying Digit-MNIST, Fashion-MNIST, and BloodMNIST. It consists of two convolutional layers followed by a ReLU activation function and a max-pooling operation. The output layer is fully connected and maps to the number of target classes. This compact architecture is a standard low-complexity baseline in the dataset distillation literature and is well suited to the $28\times 28$ greyscale inputs of the MNIST-family datasets.
• ResNet50V2 [32]: 50-layer residual network for CIFAR-10 and CIFAR-100 classification. ResNet50V2 adopts a pre-activation residual design, placing batch normalization and activation before each convolution, allowing for more stable gradient flow when training on complex datasets. For all experiments, the backbone was kept fixed, and only the fully connected classification head attached to the network’s output was trained on the distilled images. This protocol isolates the quality of the distilled data from any confounding effect of fine-tuning the deep feature extractor.

4.3. Training Setup

We trained the above models implementing the following settings:

• Framework: All models were implemented and trained in Python using TensorFlow/Keras.
• Images per class (IPC): We distilled 10, 20, and 50 images per class for Digit-MNIST and Fashion-MNIST, and 10 and 50 images per class for CIFAR-10, CIFAR-100, and BloodMNIST.
• Epochs: Each model was trained for multiple epochs that varied depending on the dataset. For Digit-MNIST, Fashion-MNIST, and CIFAR-10, the number of epochs used was 100, 200, 300, 600, and 1000; for CIFAR-100, the number of epochs was 100, 200, and 400; and for BloodMNIST, 100, 200, 400, and 600 epochs were conducted as summarized in Table 1.
• Batch size: Set to 32 for all experiments.
• Loss function: Categorical cross-entropy was used since all tasks are multiclass classification problems:

  $$\mathcal{L}=-{\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{\widehat{c}=1}^c}{y}_{i,\widehat{c}}log\left({\widehat{y}}_{i,\widehat{c}}\right),$$

  (23)

  where n is the number of training samples, c is the number of classes, and $y_{i,\widehat{c}}$ is the ground truth label of the i-th input sample for class $\widehat{c}$. Specifically,

  $$y_{i,\widehat{c}}=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}\mathrm{sample}i\mathrm{belongs}\mathrm{to}\mathrm{class}\widehat{c},\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

  (24)

  and ${\widehat{y}}_{i,\widehat{c}}$ is the predicted probability for the i-th sample to belong to class $\widehat{c}$.
• Optimizer: The Adam optimizer was implemented by the two classifiers selected by us with a fixed learning rate of $\alpha =0.001$. No learning rate decay was applied, ensuring consistent and comparable results across all models and datasets.

All experiments were conducted using TensorFlow/Keras with GPU acceleration. The key parameters, such as, learning rate, batch size, and number of distilled images per class — were determined empirically through preliminary validation to achieve a balance between classification accuracy and computational efficiency.

The accuracy (AC) was used to evaluate the classification performance and was calculated as follows:

$$\mathrm{AC}={\displaystyle \frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}},$$

(25)

where TP, TN, FP, and FN are True Positives, True Negatives, False Positives, and False Negatives, respectively. In all experiments, the classifier is evaluated on the original unmodified test set of the corresponding dataset. Therefore, the reported accuracy is a measure of generalisation to unseen real images and not the distilled subset.

Table 1 provides a concise overview of the experimental configuration for each dataset.

5. Experimental Results

In the present section, we show experimental results to validate the theoretical developments and novelties of the present study. For this purpose, we used the above-described image databases, CNNs, and setups.

In Table 2, we show the accuracy we receive for various IPCs and epochs. We used 10, 20, and 50 IPC for each dataset and the Baseline NN for all experiments in this table. One may observe that for digit-MNIST, with 10 and 20 IPC, the Baseline CNN obtained its highest results for 300 epochs. Increasing the epochs further did not increase the accuracy due to overfitting of the model. However, for 50 IPC, we received the highest accuracy for the maximum (1000) number of epochs. For Fashion-MNIST, we received the best accuracy for 10, 20, and 50 IPC, respectively, for epochs 1000, 300, and 200.

Table 3 presents the training time and classification accuracy of the Baseline NN trained on the full original dataset and the distilled images for Digit-MNIST and Fashion-MNIST. Training on the original data gives a slightly higher accuracy, but the difference in training time is significant. For Digit-MNIST, training time with the original dataset, is approximately $4\times$ than training time with the distilled images. For Fashion-MNIST, the gap is even larger at approximately $5\times$. These results demonstrate that the proposed M-PCA-SVD distillation method achieves a highly favorable trade-off, which is a modest reduction in accuracy of about $2\%$ and $5.5\%$ respectively translates into savings of more than $75\%$ in training time (for Digit-MNIST) and a dramatic reduction from 60,000 original images to just 500 distilled images, highlighting the practical advantage of the proposed non-gradient framework in resource-constrained environments. Please note that for Fashion-MNIST, the saved time is even higher - 80% (Table 3).

Table 4 presents the classification accuracy of ResNet50V2 trained on original and distilled images from CIFAR-10 and CIFAR-100, and the Baseline NN trained on original and distilled images from BloodMNIST. For CIFAR-10, training with the distilled images consistently and significantly outperforms the original images across all epoch settings for both IPC=10 and IPC=50. This achievement suggests that the QM-PCA-QSVD distillation process does not just compress the data but also actively enhances the discriminative characteristics of the training set. For CIFAR-100, the classification after training with the distilled images also outperforms classification after training with the original images across all epoch settings for both IPC=10 and IPC=50, demonstrating that the method scales effectively to the more challenging 100-class setting. For BloodMNIST, the distilled images outperform training with the original images at IPC=10 and achieve competitive and slightly superior results at IPC=50, with the distilled set reaching $85.47\%$ compared to $83.98\%$ for the originals, demonstrating that the proposed framework generalizes well to medical image datasets.

Figure 7a presents the confusion matrix for CIFAR-10, where the ResNet50V2 classifier trained on QM-PCA-QSVD distilled images achieves strong diagonal accuracy, indicating correct classification across all ten classes. The most challenging classes are cat and deer, which show the highest misclassification (the lowest recognition rate as shown in Figure 7a). Note that the model was tested on the entire test set of 10,000 images. Figure 7 b shows the training and validation accuracy curves over 400 epochs, where both curves become steady and stabilize around $75\%$, confirming that the model trained on distilled images generalizes well without overfitting.

Table 5 presents a detailed comparison of the proposed M-PCA-SVD method with state-of-the-art dataset distillation methods on five benchmark datasets. The results obtained on the grey-level databases are mixed. The competing methods use complex NN-based optimization pipelines specifically tuned for these benchmarks, while our method does not need any optimization specific to the image databases, as the NN-based methods do. However, for the more complicated color image data sets CIFAR-10 and CIFAR-100, the proposed QM-PCA-QSVD method is clearly superior to all competitors. On CIFAR-10, our method outperforms all listed methods under both IPC settings, including the recent gradient-based IADD [6] and the factorized methods such as KFS and DREAM. On CIFAR-100 at IPC=50, our method outperforms all competitors, including DREAM ($52.6\%$) and IADD ($49.0\%$) by achieving $56.27\%$. These results confirm that the proposed non-gradient algebraic framework significantly outperforms gradient-NN-based state-of-the-art methods on the more challenging color image benchmarks, while requiring no NN training during the distillation process itself.

5.1. Ablation Study

For the performance of each stage in the introduced QM-PCA-QSVD method, we have conducted an ablation study on the Blood-MNIST dataset using the Baseline CNN classifier with IPC=50. We compared the CNN performance according to the following: (1) Training with 50 randomly selected original images per class; (2) Training with IPC=50, where we applied QM-PCA alone for image distillation without applying the QSVD; (3) Training with IPC=50 applying our proposed method QM-PCA and QSVD block rotation at $\theta ={300}^{\circ }$.

Table 6 gives the summary of the classification accuracy achieved in each configuration. We received 81.26% when trained the Baseline CNN on 50 randomly selected original images from the training set. We received an improved accuracy of 84.16% when we trained the CNN with 50 IPC distilled by QM-PCA only. It shows QPCA-based distillation can extract more discriminative image features and generate a compact distilled set that performs better than training on the same number of original images. On the next stage, we added the QSVD block rotation part and trained our CNN with 50 randomly selected distilled images per class and received an accuracy of 85.67% for the distillation process. This validates that structured rotation of the left singular matrix adds additional useful information to the distilled images and improves the classification accuracy.

The results confirm two main factors of our proposed method. Firstly, the QM-PCA stage is sufficient to outperform the original images. Secondly, the addition of the QSVD block rotation step ensures a consistent and meaningful increase in the accuracy over QM-PCA, which proves it to be an essential component in the pipeline. Remarkably, both distillation approaches yield better classification results than training on an equal number of random original images, highlighting the value of our non-gradient framework. This optimization happens because M-PCA and QM-PCA extract Eigenvectors with maximum Eigenvalues, while SVD and QSVD capture the vectors with the largest singular values.

Another comparison to support the significance of the new QM-PCA-QSVD method is with the method presented by [33]. Note that this method augmented the training set three times, bringing the training images from 11,959 to almost 36,000. Using this training set, the authors reported an accuracy of 76.2% with an NN and 76.8% with a support vector machine (SVM) after a grid search. Recall that the new QM-PCA-QSVD achieved an accuracy of 85.47% when training the Baseline CNN with a total of 400 images.

6. Discussion

An important observation that can be drawn from Table 2 and Table 4 is that increasing the number of epochs when training with distilled images does not always lead to an increase in classification accuracy. This behavior suggests that the distilled images, generated through the proposed closed-form algebraic pipeline, produce a compact and representative summary of the original data, which is sufficient for the classifier to converge relatively early. Using the small distilled set, beyond a certain number of epochs, the model may begin to overfit rather than continuing to improve, which explains the varying accuracy trends observed across different distilled IPC settings and datasets. This highlights the importance of epoch selection when training with distilled data and suggests that early stopping or monitoring based on validation is advisable in practice.

Table 3 demonstrates the computational effectiveness of the proposed M-PCA-SVD method. Training with the distilled images requires only a fraction of the time needed to train on the full original dataset, speeding up the process to approximately $4\times$ and $5\times$ for Digit-MNIST and Fashion-MNIST, respectively. This substantial saving in computer time and memory comes at the cost of a modest reduction in classification accuracy, ranging from approximately $2\%$ to at most $5.5\%$. In many practical applications, particularly in limited-resource or time-sensitive environments, this trade-off is highly favorable, as the proposed method delivers near-original-data performance while dramatically reducing the computational burden of both the distillation process and the subsequent classifier training.

Table 5 provides a broader perspective by comparing the proposed QM-PCA-QSVD method against a wide range of state-of-the-art dataset distillation approaches. On the grey-level Digit-MNIST and Fashion-MNIST databases, the results by the new method are mixed. It achieves competitive accuracy, particularly at higher IPC values, but follows some gradient-based and factorized competitors on these simpler datasets. This is not surprising, as those methods utilize sophisticated NN-based optimization pipelines specifically designed and tuned for such benchmarks, often requiring thousands of GPU hours to complete the distillation process. In contrast, the proposed method requires no NN training during distillation whatsoever.

On the more complex and challenging color image datasets, CIFAR-10 and CIFAR-100, the proposed QM-PCA-QSVD method tells a different story. The distilled color images train the classifiers in a way that the classification accuracy obtained by our method outperforms all listed competitors at both IPC settings on CIFAR-10, and achieves the best reported result on CIFAR-100 at IPC=50. This outcome suggests that the algebraic structure of the QM-PCA-QSVD pipeline, particularly the QM-PCA-based inter-image color preservation and the QSVD block rotation capture discriminative features that are especially well-suited to complex color image classification tasks and do so more effectively than gradient-based approaches that rely on process-level imitation rather than algebraic data transformation.

7. Conclusions

In this paper, we propose a novel non-gradient, non-NN dataset distillation framework based on Modified Principal Component Analysis (M-PCA) and Singular Value Decomposition (SVD) with block rotation, extended to color images through their quaternion forms (QM-PCA and QSVD). The method was evaluated on five benchmark datasets: Digit-MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and BloodMNIST and compared against a wide range of state-of-the-art gradient-based and non-gradient distillation methods.

The main novelties of the proposed method are twofold. First, we developed the first dataset distillation method that combines PCA and SVD, as well as their quaternion counterparts, to work together in a unified pipeline for both grey-level and color image distillation. Second, we introduced a novel representation of the entire training set as a single quaternion matrix, which is modified through matrix operations (M- PCA and QM-PCA) and block-wise rotation of the left singular matrix of the SVD or QSVD, generating a compact and diverse set of distilled images through a purely algebraic closed-form operation.

The advantages that arise from these novelties are significant. The proposed method requires no NN training during the distillation process itself, no gradient computation, and no iterative optimization loop, making it substantially faster than the competing gradient-based approaches. As demonstrated in Table 3, training with distilled images reduces computational time by a factor of $4\times$ to $5\times$ compared to training on the full original dataset, with only a modest reduction in classification accuracy of $2\%$ to $5.5\%$. This trade-off is particularly valuable for deep NNs trained on very large image databases of tens of thousands of images, where the savings in computer time and memory can be decisive in resource-constrained environments.

Further, recall that both PCA and SVD are applied per class. From the n original training images of the class, a matrix $n\times k$ is designed where k denotes the number of features. Then the computational complexity of PCA is $\mathcal{O}(n{k}^2+k^3)$ [34] and that of SVD is $\mathcal{O}\left(n^{2}k\right)$ [35]. Since both methods are closed-form and require no iterative optimization, the distillation time of the proposed M-PCA-SVD method is $max\left(\mathcal{O}(n{k}^2+k^3),\mathcal{O}\left(n^{2}k\right)\right)$, which is substantially smaller than that of gradient-based and neural network-driven distillation methods.

Furthermore, as shown in Table 5, the proposed QM-PCA-QSVD method outperforms all the reported state-of-the-art competitors on the more challenging color image benchmarks CIFAR-10 and CIFAR-100, demonstrating that a purely algebraic data transformation can produce distilled datasets that are not only compact and fast to produce but are also superior in their ability to train accurate classifiers on complex visual recognition tasks.

Future work will explore the extension of the proposed framework to larger and higher-resolution datasets, as well as the investigation of alternative rotation strategies according to the right singular matrix and various angles $\Theta$, to further improve distillation quality toward improving classification statistics.