An Empirical Comparative Study of Classical Dimensionality Reduction Methods: MDS, Isomap, and LLE

1. Introduction

In high-dimensional data analysis, dimensionality reduction is a fundamental technique that enables effective data visualization, noise filtering, and computational efficiency. Feature Extraction Algorithms (FEAs) serve as a cornerstone in this process, transforming raw data into lower-dimensional representations while preserving essential structures. Among them, manifold-based FEAs have gained significant attention due to their ability to uncover nonlinear geometric structures inherent in complex datasets. These methods seek to preserve intrinsic relationships between data points, rather than relying on simple linear projections [1,2].

Within this paradigm, Multidimensional Scaling (MDS), Isometric Mapping (Isomap), and Locally Linear Embedding (LLE) represent three pivotal approaches that have shaped the evolution of manifold learning and nonlinear dimensionality reduction. MDS, initially developed for psychometrics and data visualization, provides a distance-preserving embedding. Isomap extends MDS by incorporating geodesic distances, making it effective for nonlinear structures. LLE, on the other hand, focuses on local geometric properties to reconstruct a globally consistent low-dimensional representation. These methods have been widely applied in various domains, including computer vision, bioinformatics, and robotics, demonstrating their enduring impact on data-driven research [3,4,5]. Despite their popularity, the choice among these methods is often guided by heuristic preferences rather than systematic performance evidence. Many prior studies focus either on theoretical derivations or on narrow experimental settings, leaving a gap in broad, quantitative comparisons across diverse datasets.

In this work, we conduct an empirical comparative study of MDS, Isomap, and LLE and their variants, evaluating them under controlled experiments across synthetic manifolds, low-dimensional tabular datasets, and high-dimensional image data. Despite the emergence of deep learning-based feature extraction techniques, MDS, Isomap, and LLE remain relevant due to their interpretability, efficiency, and adaptability in different scenarios. Understanding their theoretical foundations, practical applications, and comparative advantages is essential for researchers and practitioners seeking suitable dimensionality reduction techniques for their data. The key contributions are as follows:

• Algorithmic Workflows, Applications, and Complexity Analysis We summarize the core mathematical principles and computational processes of MDS, Isomap, and LLE. We highlight their respective application domains and discuss their computational complexity in large-scale settings.
• Conceptual and Empirical Comparisons We compare reconstruction error, runtime, and visualization quality, revealing the trade-offs between global and local structure preservation. We analyze their strengths, limitations, and suitability for various data structures, providing both theoretical insights and empirical validation.
• Adaptability in the Modern Era We examine the relevance of MDS, Isomap, and LLE in the context of large-scale data, deep learning advancements, and real-world applications. We discuss possible extensions and how these methods can be integrated into contemporary machine learning workflows.

2. Notation

The majority of manifold-based finite element analyses (FEAs) are nonlinear, unsupervised [6], and rely on neighborhood graphs [7]. These methods primarily concentrate on nonlinear mappings, assuming that the data, when projected into a lower-dimensional space, lies on a densely sampled manifold that needs to be unfolded [8,9]. Among these techniques, MDS, Isomap, and LLE are considered the classics. Table 1 presents the notations used in the paper.

3. Multi-Dimensional Scaling

Multidimensional Scaling (MDS) was introduced initially in [10] and stands as one of the pioneering techniques for dimensionality reduction [11]. It serves multiple purposes, including subspace learning, dimensionality reduction, and feature extraction. The core principle of MDS is to maintain the similarity [12] or dissimilarity/distances [13] between points in the low-dimensional space [2,14]. This approach enables MDS to capture the global structure of the data by fitting it locally [15]. MDS can be classified into three main types: classical MDS, metric MDS, and non-metric MDS [11,16], each of which will be discussed in this section. These categories yield distinct embedding results [17].

3.1. Classical Multidimensional Scaling

The goal of classical MDS is to preserve the similarity of the data points in the embedding space as it was in the input space [10]. One way to measure similarity is to calculate the inner product. Consequently, the difference between the similarities in the input and embedding spaces can be minimized by:

$$\underset{{\left\{{\mathbf{y}}_i\right\}}_{i=1}^n}{\mathrm{minimize}}{c}_1:=\sum _{i=1}^n\sum _{j=1}^n{(x_i^{\top }{x}_j-y_i^{\top }{y}_j)}^2,$$

(1)

and we can extend it to a matrix form as:

$$\underset{Y}{\mathrm{minimize}}{c}_1=\left|\right|X^{\top }X-Y^{\top }Y{\left|\right|}_F^2$$

(2)

and where ${\parallel ·\parallel }_F$ denotes the Frobenius norm, and $X^{\top }X$ and $Y^{\top }Y$ are the Gram matrices of the original data X and the embedded data Y, respectively. We can then decompose the two Grame matrices using eigenvalue decomposition and get:

$$\begin{array}{cc}\hfill X^{\top }X& =V\Delta V^{\top },\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill Y^{\top }Y& =Q\Psi Q^{\top },\hfill \end{array}$$

(4)

where $\Delta$ and $\Psi$ are obtained by squaring the singular values, after utilizing a Singular Value Decomposition (SVD) and by taking the right singular vectors of X and Y as V and Q, respectively. After mathematical proof and derivation[12], we can then obtain Y using:

$$Y={\Delta }^{{\displaystyle \frac{1}{2}}}{V}^{\top }.$$

(5)

Finally, we can truncate Y to $Y\in {\mathbb{R}}^{p\times n}$ , we can select the first p rows, which can result in a p-dimensional embedding of the n points.

3.2. Generalized Classical MDS (Kernel Classical MDS)

In the generalized setting, classical MDS is extended by allowing any valid kernel function to define the similarity matrix. Notably, Isomap can be seen as a special case of this approach, where geodesic distances are used instead of Euclidean distances [11].

As we get the Gram matrix in Section 3.1 we can give the Gram matrix as:

$$X^{\top }X=-{\displaystyle \frac{1}{2}}HDH,$$

(6)

where D is the distance matrix with a squared Euclidean distance as its elements. Then by extending with a general kernel matrix, rather than the double-centered Gram matrix, we can have[11]:

$$K=-{\displaystyle \frac{1}{2}}HDH$$

(7)

where $K\in {\mathbb{R}}^{n\times n}$ is the kernel matrix. for embedding X using classical MDS, the eigenvalue decomposition of the kernel matrix K is obtained in a similar way as Eq.(3):

$$K=V\Delta V^{\top }$$

(8)

Finally, we can get Y just as in Eq.(5), after a proper truncation and optimal selection, we can get top k dimention embedding of n points. We show the brief algorithm of generalized classical MDS in Algorithm .

Algorithm 1:MDS

Require:  $X\in R^{n\times d}$ Ensure:  $Y\in R^{n\times k}$ 1: Compute the dissimilarity matrix $d^X$ of X 2: Compute K by utilizing the centering matrix H 3: Apply Eigen decomposition to K to have top k Eigen values and corresponding vectors 4: Obtain Y using diagonal matrix K and top k Eigen vectors

Classical MDS with the Euclidean distance is equivalent to Principal Component Analysis (PCA). Moreover, the generalized classical MDS is equivalent to kernel PCA. This is a useful proposition we can use to further analyze the different extraction properties of two classical methods [18].

3.3. Metric Multidimensional Scaling

In classical Multidimensional Scaling (MDS), the focus is on maintaining the similarities between points in the embedding space. However, subsequent methods, such as Metric MDS, shifted the objective to preserving distances instead of similarities [19,20] within the embedding space [13]. Metric MDS aims to minimize the discrepancies in distances between points in both the input and embedding spaces [21]. The cost function used in Metric MDS is commonly known as the stress function [22,23]. This approach is termed "metric MDS" because it employs a distance metric for its optimization process.

The goal of metric MDS is to change the cost function to preserve the distances, rather than the similarities in the embedding space. In addition, metric MDS seeks to directly preserve pairwise distances in the embedded space, typically minimizing a stress function such as Kruskal’s stress. The objective function of metric MDS could be optimized by:

$$\begin{array}{cc}\hfill & \underset{{\left\{y_i\right\}}_{i=1}^n}{\mathrm{minimize}}{c}_2:=\hfill \\ & {\left({\displaystyle \frac{{\sum }_{i=1}^n{\sum }_{j=1,j<i}^n{\left(d_x(x_i,x_j)-d_y(y_i,y_j)\right)}^2}{{\sum }_{i=1}^n{\sum }_{j=1,j<i}^n{d}_x^2(x_i,x_j)}}\right)}^{{\displaystyle \frac{1}{2}}}\hfill \end{array}$$

(9)

where the molecular part represents the normalization factors. Another well-known dimensionality reduction method Sammon Mapping [24], which is a special case of metric MDS.

3.4. Non-metric Multidimensional Scaling

Non-Metric Multidimensional Scaling (NMDS) is a variant of MDS that focuses on preserving the rank order of distances rather than their exact values [25,26]. Unlike metric MDS, which seeks to maintain precise pairwise distances, NMDS is more flexible and can handle non-Euclidean, nonlinear, or ordinal similarity measures. Instead of preserving absolute distances, it focuses on preserving the relative order of distances, making it robust to nonlinearly transformed similarity measures. This makes it particularly useful when the underlying data structure is unknown or when only qualitative comparisons (e.g., “A is more similar to B than to C”) are available. NMDS iteratively adjusts the low-dimensional representation to ensure that the ranking of distances in the embedded space closely matches the ranking in the original space, often optimizing a stress function such as Kruskal’s stress [27]. The objective can be generally written as:

$$\begin{array}{cc}\hfill & \underset{{\left\{y_i\right\}}_{i=1}^n}{\mathrm{minimize}}{c}_3:\hfill \\ & ={\left({\displaystyle \frac{{\sum }_{i=1}^n{\sum }_{j=1,j<i}^n{\left(d_x(x_i,x_j)-f\left(d_y(y_i,y_j)\right)\right)}^2}{{\sum }_{i=1}^n{\sum }_{j=1,j<i}^n{d}_x^2(x_i,x_j)}}\right)}^{{\displaystyle \frac{1}{2}}}\hfill \end{array}$$

(10)

Because of its robustness to distortions in absolute distances, NMDS is widely used in psychology, sociology, ecology, and market research, where perceptual or subjective similarity judgments are common.

4. Isometric Mapping

Isometric Mapping (Isomap) is a nonlinear dimensionality reduction technique designed to address the limitations of traditional methods like PCA and MDS, which struggle with preserving the intrinsic geometry of data lying on a nonlinear manifold [8,28]. Unlike these classical approaches that rely on Euclidean distances, Isomap reconstructs the manifold structure by maintaining geodesic distances between data points in the lower-dimensional space [2,14,29].

To achieve this, Isomap begins by constructing a neighborhood graph, where each data point is linked to its closest neighbors based on predefined criteria. It then estimates geodesic distances between all points by computing the shortest paths within this graph using algorithms such as Dijkstra’s or Floyd-Warshall [30]. The final step involves applying classical MDS to the computed geodesic distance matrix, ensuring that the low-dimensional representation retains the essential relationships present in the original high-dimensional space. Geodesic distance refers to the shortest path between two points along the surface or structure of a given space, rather than through the surrounding ambient space. In the context of manifold learning techniques like Isomap, geodesic distance is used to approximate the true intrinsic distance between data points that lie on a nonlinear manifold [31]. A simple comparison of Euclidian distance and geodeisic distance is shown in Figure 1.

The algorithm of Isomap is summarized in Algorithm 2.

Algorithm 2:Isomap

Require:  $X\in R^{n\times d}$ Ensure:  $Y\in R^{n\times k}$ 1: Develop the neighborhood graph of X 2: Calculate the geodesic distance and build $d_G$ 3: Apply MDS to $d_G$ to obtain new space Y

While Isomap has proven effective in capturing nonlinear structures, it is not without drawbacks. One major issue is its sensitivity to the quality of the neighborhood graph, which may introduce errors if incorrect connections are formed, leading to distortions in the final embedding.[32] Additionally, Isomap struggles with complex manifolds that contain discontinuities or non-convex regions, where geodesic distances may become unreliable. Despite these challenges, Isomap remains widely used in various scientific and engineering applications, including face recognition, motion analysis, and medical imaging, due to its ability to reveal meaningful low-dimensional representations of high-dimensional data [33].

5. Locally Linear Embedding

Locally Linear Embedding (LLE) is a widely recognized nonlinear spectral dimensionality reduction technique, primarily designed for manifold learning and feature extraction [34,35,36]. Unlike traditional linear techniques such as PCA or MDS, LLE preserves the local geometric structure of high-dimensional data while mapping it to a lower-dimensional space [37]. This is achieved by approximating each data point as a weighted linear combination of its nearest neighbors in the original space and ensuring that these relationships remain consistent in the embedded space [2,38].

The algorithm follows a three-step process: (1) identifying the K nearest neighbors for each data point, (2) computing the optimal reconstruction weights that best express each point as a linear combination of its neighbors, and (3) finding a low-dimensional embedding that preserves these relationships [15,39]. By focusing on local neighborhood structures rather than global variance, LLE effectively captures the intrinsic geometry of nonlinear manifolds, making it a powerful tool for dimensionality reduction [40].

Given a dataset of N points ${\left\{x_i\right\}}_{i=1}^N$ in a D-dimensional space, LLE first constructs a neighborhood graph by identifying the K nearest neighbors of each point based on Euclidean distance. Then, it computes the reconstruction weights $w_{ij}$ by approximating each data point as a weighted linear combination of its neighbors, minimizing the reconstruction error:

$$\sum _{i=1}^N{∥x_i-\sum _{j\in \mathcal{N}\left(i\right)}{w}_{ij}{x}_j∥}^2$$

(11)

subject to the constraint:

$$\sum _{j\in \mathcal{N}\left(i\right)}{w}_{ij}=1.$$

(12)

These weights are obtained by solving a constrained least-squares problem. Finally, LLE determines the low-dimensional embeddings ${\left\{y_i\right\}}_{i=1}^N$ by minimizing:

$$\sum _{i=1}^N{∥y_i-\sum _{j\in \mathcal{N}\left(i\right)}{w}_{ij}{y}_j∥}^2.$$

(13)

This step corresponds to finding the eigenvectors associated with the smallest $d+1$ nonzero eigenvalues of the matrix:

$$M={(I-W)}^T(I-W),$$

(14)

where W is the weight matrix containing $w_{ij}$. The final low-dimensional representation is given by the eigenvectors corresponding to the smallest d nonzero eigenvalues. LLE is widely used in manifold learning as it effectively preserves local geometric structures without assuming global linearity.

The algorithm of LLE is summarized in Algorithm 3.

Algorithm 3:LLE

Require:  $X\in R^{n\times d}$ Ensure:  $Y\in R^{n\times k}$ 1: Find c-nearest neighbors for each data point of X 2: Calculate local weights W that linearly best rebuild data $\left(X^{\prime }\right)$ from its neighbors 3: Map $X^{\prime }$ to Y on k-dimensions using same weights from 2 by minimizing the cost

LLE has been widely applied across various domains, including image processing, bioinformatics, and network analysis [41,42]. In particular, it has shown effectiveness in medical diagnosis applications, where high-dimensional patient data is embedded into a meaningful lower-dimensional representation for classification and visualization [43]. Furthermore, LLE has been successfully used for analyzing astronomical spectra, demonstrating its versatility in handling complex scientific data [44]. However, despite its advantages, LLE has some limitations, including sensitivity to noise and difficulties in handling non-convex manifolds [35].

The advantage of LLE is its proficiency in managing nonlinear data and sparse matrices. Consequently, it may demand less computational time and space compared to other feature extraction methods. Overall, LLE offers a unique approach to nonlinear dimensionality reduction by leveraging local linear reconstructions to uncover meaningful global embeddings. Its ability to maintain local relationships while reducing dimensionality makes it a valuable technique in manifold learning and data-driven applications [36,37].

6. Contemporary Relevance and Applications

In contemporary applications, Multi-Dimensional Scaling (MDS), Isomap, and Locally Linear Embedding (LLE) are vital for analyzing high-dimensional data. While MDS excels at preserving pairwise dissimilarities in linear spaces, it struggles with nonlinear relationships. Isomap overcomes this by incorporating geodesic distances, making it ideal for data on nonlinear manifolds [34]. LLE further enhances this by maintaining local geometric structures, allowing it to model complex manifolds effectively [35]. These techniques are widely used in fields such as image processing, medical diagnostics, and bioinformatics, offering powerful tools for dimensionality reduction and data visualization. In addition, some of these methods are still active in modern machine learning systems, and has been widely used in multiple fields like quantum physics and multi-class data augmentation of wind [45,46]. A summary of development and applications of MDS, Isomap, and LLE is shown in Table 2.

7. Experiments

7.1. Datasets

We use four different datasets, encompassing synthetic and real-world data, numerical data and images, as well as both regression and classification tasks, to provide a comprehensive comparison of MDS, Isomap, and LLE. The datasets are summarized in Table 3.

7.2. Methods

In the context of MDS, we examine both metric MDS and non-metric MDS, along with Isomap, which can also be regarded as a special case of kernel classical MDS. In addition to standard LLE, we consider two variants: HLLE [76] and MLLE [76]. Hessian LLE (HLLE) improves the robustness of LLE by using Hessian-based constraints to better preserve local geometric structures. Modified LLE (MLLE) tackles the problem of short-circuiting in manifolds by adjusting the reconstruction weights, ensuring more accurate modeling of complex data structures. The visualization of the six methods are shown in Section 7.3.

Aside from the methods based on MDS, Isomap and LLE, we also utilize several classical methods for visualization experiments, including PCA [77], LTSA [78], Kernel PCA [77], Laplacian Eigenmaps [79], and t-SNE [80]. The overall visualization results are shown in Appendix (Appendix A).

Table 4. Runtime and Error Comparisons of Dimensionality Reduction Methods Across Datasets

Method	SwissRoll		Iris		AutoMPG		MNIST
	Runtime (s)	Error	Runtime (s)	Error	Runtime (s)	Error	Runtime (s)	Error
Metric MDS	6.28	5.9e+06	0.16	1.6e+02	1.55	2.3e+06	0.59	4.1e+10
Non-metric MDS	2.12	2.3e+04	0.04	5.1e+02	0.30	3.5e+03	0.14	2.0e+03
Isomap	0.35	1.0e+01	0.02	1.2e-01	0.13	1.4e+03	0.04	1.0e+06
Standard LLE	0.09	2.2e-07	0.01	2.9e-05	0.08	1.1e-03	1.01	2.0e-03
Hessian LLE	0.18	4.8e-06	0.03	1.1e-01	0.21	1.4e-01	6.49	2.4e+01
Modified LLE	0.14	2.2e-06	0.03	2.5e-01	0.20	6.2e-01	6.97	1.7e+02

Table 4. Runtime and Error Comparisons of Dimensionality Reduction Methods Across Datasets

Method	SwissRoll		Iris		AutoMPG		MNIST
	Runtime (s)	Error	Runtime (s)	Error	Runtime (s)	Error	Runtime (s)	Error
Metric MDS	6.28	5.9e+06	0.16	1.6e+02	1.55	2.3e+06	0.59	4.1e+10
Non-metric MDS	2.12	2.3e+04	0.04	5.1e+02	0.30	3.5e+03	0.14	2.0e+03
Isomap	0.35	1.0e+01	0.02	1.2e-01	0.13	1.4e+03	0.04	1.0e+06
Standard LLE	0.09	2.2e-07	0.01	2.9e-05	0.08	1.1e-03	1.01	2.0e-03
Hessian LLE	0.18	4.8e-06	0.03	1.1e-01	0.21	1.4e-01	6.49	2.4e+01
Modified LLE	0.14	2.2e-06	0.03	2.5e-01	0.20	6.2e-01	6.97	1.7e+02

7.3. Visualization Results

We show the visualization results of different datasets from Figure 2 to Figure 5.

Figure 2. SwissRoll visualization results.

Figure 2. SwissRoll visualization results.

Figure 3. Iris visualization results.

Figure 3. Iris visualization results.

Figure 4. AutoMPG visualization results.

Figure 4. AutoMPG visualization results.

Figure 5. MNIST visualization results.

Figure 5. MNIST visualization results.

The SwissRoll dataset reveals striking differences between the techniques. In terms of error performance, Standard LLE dramatically outperforms other methods with an error ($2.2\times {10}^{-7}$) that is approximately 10 orders of magnitude lower than Metric MDS ($5.9\times {10}^6$). Standard LLE is also the fastest method (0.09s), while Metric MDS is significantly slower (6.28s). Regarding visual structure, LLE variants preserve the intrinsic manifold structure, displaying a coherent circular or curved pattern, whereas Non-metric MDS fails to capture meaningful structure. The superior performance of LLE methods suggests this dataset has strong local geometric properties that are well-preserved by neighborhood-based approaches.

The well-known Iris dataset presents different characteristics compared to the synthetic manifold. Standard LLE again achieves the lowest error ($2.9\times {10}^{-5}$), but Isomap (0.02s) and Non-metric MDS (0.04s) offer competitive runtime performance. In terms of clustering quality, LLE variants and Isomap effectively separate the three classes (likely corresponding to the three Iris species), while Non-metric MDS produces poor separation. Hessian and Modified LLE produce more linear arrangements of the data points, potentially revealing the known characteristic that one Iris species is linearly separable from the other two. The moderate dimensionality of the Iris dataset (4 features) likely contributes to the overall good performance across most methods.

The AutoMPG dataset reveals interesting patterns related to fuel efficiency. Standard LLE maintains its error advantage ($1.1\times {10}^{-3}$), but the gap narrows compared to other datasets. Modified LLE and Metric MDS produce the most visually interpretable projections, clearly showing the gradient from high to low MPG values. The curved structures revealed by several methods (particularly Modified LLE’s U-shape) suggest that vehicle efficiency follows non-linear relationships in the feature space. Interestingly, Metric MDS produces a visually coherent projection despite its high error ($2.3\times {10}^6$), highlighting that mathematical error doesn’t always correlate with visual interpretability.

The high-dimensional MNIST handwritten digit dataset (784 dimensions) presents unique challenges for dimensionality reduction. LLE variants become significantly slower (6.49s and 6.97s for Hessian and Modified LLE, respectively) compared to other methods and other datasets. Metric MDS produces an extremely high error ($4.1\times {10}^{10}$), while Standard LLE maintains a remarkably low error ($2.0\times {10}^{-3}$). Standard LLE produces the most effective separation of digit classes, though several methods show similar digits (e.g., 1 and 7) clustering together. Isomap offers the best balance of speed (0.04s) and reasonable structure preservation for this high-dimensional dataset.

7.4. Temporal Complexity Analysis

In terms of computational complexity, MDS is generally the most computationally expensive among the three methods, as it requires calculating a distance matrix and performing eigenvalue decomposition, which can become slow for large datasets. Isomap involves additional steps such as constructing a neighborhood graph and calculating the shortest paths between points, making it more complex than MDS, especially as the dataset grows. LLE, while similar to Isomap in that it requires a nearest-neighbor search and eigen-decomposition, tends to be more efficient in handling sparse data structures. However, it can still be computationally expensive due to the need to solve for reconstruction weights. Overall, Isomap and LLE are more complex than MDS, with Isomap requiring more memory and processing due to the shortest-path calculations, while LLE can be more efficient for sparse data but still demands significant computation for larger datasets. We show the complexity comparison of MDS, Isomap, and LLE in Table 5.

Overall, we make a general comparison of MDS, Isomap, and LLE in Table 5.

Table 6. General Comparison of MDS, Isomap, and LLE.

	Goal	Supervision	Linearity	Data Charac.	Iteration	Topology
MDS	Preserve Euclidean pairwise distances	Unsuper.	Nonlinear	Perform better with relational data	Yes	Manifold
Isomap	Preserve geodesic pairwise distances	Unsuper./Super.	Nonlinear	Handle sparse and noisy data	No	Manifold
LLE	Preserve local properties	Unsuper.	Nonlinear	Handle sparse and nonlinear data	Yes	Manifold

Table 6. General Comparison of MDS, Isomap, and LLE.

	Goal	Supervision	Linearity	Data Charac.	Iteration	Topology
MDS	Preserve Euclidean pairwise distances	Unsuper.	Nonlinear	Perform better with relational data	Yes	Manifold
Isomap	Preserve geodesic pairwise distances	Unsuper./Super.	Nonlinear	Handle sparse and noisy data	No	Manifold
LLE	Preserve local properties	Unsuper.	Nonlinear	Handle sparse and nonlinear data	Yes	Manifold

8. Adaptibility in the Modern Era

With the advancement of deep learning technologies, an increasing number of traditional methods are being replaced by deep learning in the modern era. In this rapidly evolving technological landscape, it is worth exploring how traditional methods like MDS, Isomap, and LLE can be adapted, refined, or integrated into contemporary approaches.

Selecting an appropriate dimensionality reduction technique for high-dimensional data, such as images, requires careful consideration of multiple factors including data characteristics, computational resources, preservation requirements, and the intended downstream tasks. Classical methods like MDS, Isomap, and LLE, alongside modern deep learning approaches [81], each present distinct advantages and limitations.

The intrinsic dimensionality and structure of the data should guide initial method selection. MDS techniques emphasize distance preservation and work well when global distances are meaningful, with metric MDS preserving Euclidean distances and non-metric MDS focusing on ordinal relationships. For image data, however, Euclidean distances often fail to capture perceptual similarity, limiting the effectiveness of standard MDS. When images are believed to lie on or near a manifold, manifold learning approaches become more appropriate. Isomap extends MDS by approximating geodesic distances along the manifold through neighborhood graphs, making it suitable for data with curved but smooth underlying structure. LLE and its variants (Hessian, Modified) focus on preserving local relationships by reconstructing each point from its neighbors, often yielding better results for image data where local pixel relationships are more meaningful than global distances.

For high-dimensional image datasets, computational considerations become paramount. Standard implementations of MDS and Isomap scale poorly. Several strategies can mitigate these limitations: landmark points [82,83]; incremental variants compute embeddings for new points without recalculating the entire solution [84]; and spectral approximation techniques can improve efficiency [85,86]. For extremely large image datasets (e.g., millions of images), sampling representative subsets before applying classical methods often becomes necessary.

Deep learning methods, such as Variational Autoencoders (VAEs) [87] and Generative adversarial networks (GANs) [88], offer compelling alternatives for high-dimensional image data. VAEs combine the reconstruction capability of autoencoders with probabilistic modeling, creating smooth, continuous latent spaces that facilitate generation and interpolation. Their non-linear transformations can capture complex relationships in image data that linear methods miss. Similarly, GANs incorporate discriminator networks to match latent distributions to prior distributions, often yielding more structured latent spaces. These deep learning approaches scale well with data dimensionality, though they typically require substantial data volumes and training time.

For practical implementation with image data, several preprocessing steps can significantly improve results. Downsampling high-resolution images reduces dimensionality while preserving essential structural information. Feature extraction using pre-trained convolutional neural networks (e.g., extracting activations from intermediate layers of VGG networks[89] or ResNet [90]) can provide semantically meaningful representations before applying classical dimensionality reduction. Color space transformations (RGB to Lab or HSV) can also yield representations where distances better align with perceptual similarity.

Hybrid approaches often yield superior results for image data. For instance, applying t-SNE or UMAP to features extracted from pre-trained neural networks combines the perceptual understanding of deep networks with the visualization strength of manifold learning. Similarly, using classical methods to initialize latent spaces for deep learning models can accelerate convergence and improve results. Evaluation criteria should match the intended application. For visualization, qualitative assessment of cluster separation and structure preservation may suffice. For downstream tasks like classification, performance metrics on the reduced representation provide objective evaluation. Reconstruction error offers a direct measure of information preservation, while neighborhood preservation metrics (e.g., trustworthiness and continuity) evaluate how well local relationships are maintained.

No single dimensionality reduction technique universally outperforms others across all high-dimensional data scenarios. For image data specifically, deep learning methods generally excel at capturing complex patterns and generating new samples, while manifold learning approaches like Isomap and LLE often provide more interpretable visualizations of data structure. Practical applications frequently benefit from combining approaches—using deep networks for feature extraction followed by classical methods for final dimensionality reduction and visualization. This integrated strategy leverages the complementary strengths of different approaches while mitigating their individual limitations.

9. Conclusions

In this thesis, we compared three dimensionality reduction techniques: Multidimensional Scaling (MDS), Isometric Mapping (Isomap), and Locally Linear Embedding (LLE). Each method has its own strengths and weaknesses, making them suitable for different types of datasets.

• MDS works well for linear datasets, preserving global pairwise distances, but struggles with non-linear structures and can be computationally expensive.
• Isomap extends MDS by incorporating geodesic distances, making it suitable for non-linear manifolds, though it can be computationally intensive and sensitive to noisy data.
• LLE excels at preserving local structures in non-linear datasets, but requires careful tuning and is sensitive to noise in high-dimensional spaces.

Our findings suggest that the choice of method should be guided by the dataset’s intrinsic structure and the specific objectives of the analysis. MDS is preferable when global distance preservation is crucial, Isomap is effective for nonlinear manifolds, and LLE excels at capturing local nonlinear relationships. In most cases, nonlinear FEAs performed better than linear FEAs. In conclusion, the choice of method depends on the dataset’s structure and the task’s objectives. Each method possesses distinct advantages, and future research may explore hybrid approaches that harness their strengths or integrate them with deep learning techniques to enhance scalability and robustness.