Theoretical Foundations and Practical Applications in Signal Processing and Machine Learning

1. Introduction

The analysis and processing of multidimensional data have become central to many fields, including signal processing, machine learning, neuroscience, and computational sciences. As data complexity increases, traditional vector- and matrix-based methods often fail to capture the underlying structure efficiently. Tensor decomposition, a mathematical framework for representing high-order data structures, has emerged as a powerful tool for uncovering latent patterns, reducing dimensionality, and improving computational efficiency in various applications [1]. Tensors, which generalize matrices to higher dimensions, naturally arise in a wide range of domains. For instance, in signal processing, multidimensional signals such as hyperspectral images, multi-antenna wireless signals, and electroencephalography (EEG) data can be more effectively represented using tensor models [2]. In machine learning, tensors are employed in deep learning, data fusion, and knowledge graph analysis, where multi-way interactions are essential for capturing meaningful relationships [3]. The ability of tensor decompositions to leverage multilinear algebra allows for richer representations of structured data, overcoming some of the limitations of conventional matrix factorization methods. Several prominent tensor decomposition techniques exist, each with its strengths and applications. The CANDECOMP/PARAFAC (CP) decomposition factorizes a tensor into a sum of rank-one components, facilitating interpretability and uniqueness under certain conditions. The Tucker decomposition provides a more flexible representation by decomposing a tensor into a core tensor and factor matrices, enabling dimensionality reduction and feature extraction [4]. More recent developments, such as Tensor Train (TT) and Tensor Ring (TR) decompositions, have introduced scalable methods for high-dimensional data compression while maintaining computational feasibility. Beyond theoretical advancements, tensor decomposition methods have found widespread use in real-world problems. In signal processing, they contribute to source separation, multi-channel filtering, and compressed sensing [5]. In machine learning, tensor factorization is leveraged for recommender systems, graph embedding, and deep neural network compression. Additionally, scientific applications, such as chemometrics, neuroscience, and quantum physics, have benefited from tensor-based models that enable efficient representation and analysis of complex data [6]. Despite their advantages, tensor decomposition techniques also present computational challenges. The curse of dimensionality, non-convex optimization, and scalability issues pose difficulties in practical implementations. Recent research has focused on developing efficient algorithms, low-rank approximations, and randomized methods to mitigate these limitations and make tensor decompositions applicable to large-scale datasets. This paper provides a comprehensive overview of tensor decomposition methods, exploring their mathematical foundations, computational strategies, and applications in signal processing and machine learning. We discuss key theoretical properties, algorithmic developments, and emerging trends that drive ongoing research in this area [7]. By highlighting both the strengths and challenges of tensor methods, we aim to provide insights into their potential for advancing data-driven disciplines and enabling more efficient analysis of high-dimensional data structures.

2. Fundamentals of Tensor Decomposition

2.1. Definition and Notation

A tensor is a multi-dimensional generalization of a matrix, often represented as a multi-way array. Formally, an N-th order (or N-way) tensor $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N}$ consists of elements indexed by N dimensions, also known as modes. When $N=1$, the tensor reduces to a vector, and when $N=2$, it corresponds to a matrix. For $N\ge 3$, the structure extends into higher dimensions [8]. The notation for tensors follows standard conventions:

• Scalars are denoted by lowercase letters (e.g., x) [9].
• Vectors are denoted by bold lowercase letters (e.g., $\mathbf{x}$) [10].
• Matrices are represented by bold uppercase letters (e.g., $\mathbf{X}$) [11].
• Tensors are denoted by calligraphic uppercase letters (e.g., $\mathcal{X}$).

A key operation in tensor analysis is the unfolding (or matricization) of a tensor, which reshapes it into a matrix along a specified mode [12]. The mode-n unfolding of a tensor $\mathcal{X}$ is denoted as ${\mathbf{X}}_{\left(n\right)}$ and rearranges the tensor fibers along mode n into a matrix [13].

2.2. Basic Tensor Operations

Several fundamental operations are essential in tensor decomposition:

• Outer Product: Given vectors $\mathbf{a}\in {\mathbb{R}}^{I_1}$, $\mathbf{b}\in {\mathbb{R}}^{I_2}$, and $\mathbf{c}\in {\mathbb{R}}^{I_3}$, their outer product forms a rank-one tensor:

  $$\mathcal{X}=\mathbf{a}\circ \mathbf{b}\circ \mathbf{c},$$

  (1)

  where each element is given by $x_{i_1,i_2,i_3}=a_{i_1}{b}_{i_2}{c}_{i_3}$.
• Mode-n Product: The multiplication of a tensor $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N}$ with a matrix $\mathbf{U}\in {\mathbb{R}}^{J\times I_n}$ along mode n is defined as:

  $$\mathcal{Y}=\mathcal{X}{\times }_n\mathbf{U},$$

  (2)

  where the resulting tensor $\mathcal{Y}$ has dimensions $I_1\times \dots \times I_{n-1}\times J\times I_{n+1}\times \dots \times I_N$ [14].
• Frobenius Norm: The norm of a tensor is given by [15]:

  $${\parallel \mathcal{X}\parallel }_F=\sqrt{\sum _{i_1,i_2,\dots ,i_N}{x}_{i_1,i_2,\dots ,i_N}^2}.$$

  (3)

2.3. Low-Rank Representation and Importance in Applications

Many real-world tensor datasets exhibit low-rank structures, meaning they can be approximated using a small number of factors [16]. This property is crucial for efficient storage, computational feasibility, and interpretability. Low-rank tensor decompositions are widely used in applications such as:

• Signal Processing: Source separation, blind deconvolution, and multi-way filtering [17].
• Machine Learning: Dimensionality reduction, data fusion, and knowledge discovery [18].
• Computer Vision: Image compression, multi-view learning, and feature extraction [19].
• Biomedical Engineering: Brain imaging, genomic analysis, and medical signal processing.

These fundamental operations and properties set the stage for the tensor decomposition techniques discussed in the following sections [20].

3. Tensor Decomposition Methods

Tensor decomposition methods provide a framework for analyzing and factorizing high-dimensional data into meaningful low-rank components [21]. These decompositions extend classical matrix factorizations, such as singular value decomposition (SVD), to higher-order data structures, enabling efficient representation, compression, and interpretation of complex datasets. In this section, we present a comprehensive overview of major tensor decomposition techniques, including CANDECOMP/PARAFAC (CP) decomposition, Tucker decomposition, Tensor Train decomposition, and more recent approaches [22].

3.1. CANDECOMP/PARAFAC (CP) Decomposition

The CANDECOMP/PARAFAC (CP) decomposition [23,24] is one of the most fundamental tensor factorization techniques [25]. It decomposes an N-th order tensor $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N}$ into a sum of rank-one tensors:

$$\mathcal{X}\approx \sum _{r=1}^R{\mathbf{a}}_r^{\left(1\right)}\circ {\mathbf{a}}_r^{\left(2\right)}\circ \dots \circ {\mathbf{a}}_r^{\left(N\right)},$$

(4)

where each ${\mathbf{a}}_r^{\left(n\right)}\in {\mathbb{R}}^{I_n}$ is a factor matrix along mode n, and R represents the rank of the decomposition [26].

3.1.1. Properties and Uniqueness

One of the key advantages of CP decomposition is its uniqueness under mild conditions [27]. Unlike matrix factorizations, which often require additional constraints to ensure unique solutions, CP decomposition is unique (up to permutation and scaling) under the Kruskal condition:

$$k_1+k_2+\dots +k_N\ge 2R+(N-1),$$

(5)

where $k_n$ represents the Kruskal rank of the factor matrices [28].

3.1.2. Algorithms for CP Decomposition

Several algorithms have been developed to compute CP decomposition:

• Alternating Least Squares (ALS): The most commonly used method, which iteratively updates one factor matrix at a time while keeping others fixed.
• Gradient-Based Methods: These include stochastic gradient descent (SGD) and conjugate gradient techniques to improve convergence [29].
• Randomized and Approximate Methods: Tensor sketching and randomized SVD are used to accelerate CP decomposition for large-scale data [30].

3.1.3. Applications

CP decomposition has found extensive applications in:

• Signal Processing: Blind source separation, multi-sensor data fusion, and channel estimation [31].
• Machine Learning: Topic modeling, recommendation systems, and deep learning compression.
• Neuroscience: EEG and fMRI data analysis for identifying brain activity patterns.

3.2. Tucker Decomposition

The Tucker decomposition [32] is a more general factorization method that decomposes a tensor into a core tensor and factor matrices:

$$\mathcal{X}\approx \mathcal{G}{\times }_1{\mathbf{A}}^{\left(1\right)}{\times }_2{\mathbf{A}}^{\left(2\right)}\dots {\times }_N{\mathbf{A}}^{\left(N\right)},$$

(6)

where $\mathcal{G}\in {\mathbb{R}}^{J_1\times J_2\times \dots \times J_N}$ is the core tensor and each ${\mathbf{A}}^{\left(n\right)}\in {\mathbb{R}}^{I_n\times J_n}$ is a factor matrix.

3.2.1. Advantages Over CP Decomposition

Unlike CP decomposition, which enforces strict rank constraints, Tucker decomposition provides more flexibility by allowing a reduced-dimensional core tensor. This leads to:

• Better interpretability in applications like dimensionality reduction.
• Improved compression capabilities in image and video processing.
• Efficient representation of large-scale tensors with controlled rank [33].

3.2.2. Computation and Algorithms

Tucker decomposition is typically computed using:

• Higher-Order SVD (HOSVD): An extension of matrix SVD applied sequentially along each mode [34].
• Higher-Order Orthogonal Iteration (HOOI): An iterative refinement approach that improves the factorization accuracy.

3.2.3. Applications

Tucker decomposition is widely used in:

• Data Compression: Image and video compression through low-rank approximation [35].
• Multimodal Learning: Combining multiple data sources for improved feature extraction [36].
• Bioinformatics: Protein structure analysis and gene expression modeling [37].

3.3. Tensor Train Decomposition

For very high-dimensional tensors, the Tensor Train (TT) decomposition [38] provides an efficient representation that overcomes the curse of dimensionality. TT decomposition expresses a tensor as a sequence of low-rank tensor contractions:

$$\mathcal{X}(i_1,i_2,\dots ,i_N)={\mathbf{G}}^{\left(1\right)}\left(i_1\right){\mathbf{G}}^{\left(2\right)}\left(i_2\right)\dots {\mathbf{G}}^{\left(N\right)}\left(i_N\right),$$

(7)

where each ${\mathbf{G}}^{\left(n\right)}\left(i_n\right)$ is a small core tensor, drastically reducing storage complexity.

3.3.1. Advantages

• Scalability: TT decomposition scales logarithmically with tensor order, making it suitable for extremely high-dimensional data.
• Efficient Computation: Operations like tensor contraction and matrix-vector multiplication become computationally feasible.

3.3.2. Applications

Tensor Train decomposition is used in:

• Quantum Physics: Modeling high-dimensional quantum states [39].
• Neural Networks: Compression of large neural network weight matrices [40].
• Large-Scale Optimization: Solving high-dimensional PDEs and tensor regression problems [41].

3.4. Other Advanced Decomposition Methods

3.4.1. Tensor Ring Decomposition

Tensor Ring (TR) decomposition [42] generalizes Tensor Train by allowing circular dependencies among core tensors, offering even more flexible compression.

3.4.2. Sparse and Nonnegative Tensor Decompositions

In cases where tensors contain sparse or nonnegative values, specialized methods like:

• Sparse CP/Tucker Decomposition: Enforcing sparsity constraints for efficient representation [43].
• Nonnegative Tensor Factorization (NTF): Ensuring interpretability in applications like topic modeling [23].

3.5. Comparison of Tensor Decomposition Techniques

The choice of tensor decomposition method depends on the application requirements. Table 1 summarizes key properties.

This section has provided an extensive overview of tensor decomposition techniques, setting the stage for their applications in signal processing and machine learning.

4. Theoretical Advancements and Open Problems in Tensor Decomposition

While tensor decomposition has proven to be an invaluable tool in numerous applications, several theoretical challenges remain. The mathematical foundations of tensor factorization are still evolving, with ongoing research in areas such as uniqueness conditions, rank estimation, computational complexity, and robustness to noise [44]. This section explores key theoretical advancements in tensor decomposition and highlights open problems that warrant further investigation.

4.1. Uniqueness and Identifiability of Tensor Decomposition

One of the most fundamental questions in tensor analysis is the uniqueness of decomposition—whether a given tensor has a unique factorization. Unlike matrix decomposition, where Singular Value Decomposition (SVD) guarantees a unique representation up to orthogonal transformations, tensor decomposition does not always have a unique solution.

4.1.1. Uniqueness of CP Decomposition

The uniqueness of CP decomposition depends on the rank and structure of the tensor [45]. A key result in this domain is Kruskal’s theorem, which states that CP decomposition is unique if the sum of the Kruskal ranks of the factor matrices satisfies:

$$k_A+k_B+k_C\ge 2R+2.$$

(8)

However, determining the Kruskal rank of a matrix is itself a challenging problem. Recent research has explored alternative uniqueness conditions based on algebraic geometry, statistical independence, and sparsity constraints [46].

4.1.2. Uniqueness of Tucker Decomposition

Unlike CP decomposition, Tucker decomposition is not unique due to the rotational freedom of the core tensor. This non-uniqueness has implications for interpretability and compression applications. Researchers have proposed constraints such as orthogonality and sparsity to enforce uniqueness in Tucker factorization [47].

4.2. Tensor Rank Estimation and Low-Rank Approximation

Determining the rank of a tensor is a crucial yet difficult problem. Unlike matrices, where rank is well-defined and efficiently computable, the rank of a tensor is not straightforward to determine [48].

4.2.1. Challenges in Determining Tensor Rank

Tensor rank exhibits several counterintuitive properties:

• The rank of a tensor is not necessarily equal to the maximum of its matrix unfoldings [49].
• There exist tensors with arbitrarily high ranks despite having low-dimensional components.
• Tensor rank can be sensitive to small perturbations in the data [50].

Given these challenges, researchers have explored surrogate measures such as the nuclear norm and Schatten-p norms to approximate tensor rank efficiently.

4.2.2. Low-Rank Tensor Approximation

Low-rank approximation is essential in practical applications where full tensor decomposition is computationally infeasible [51]. Key approaches include:

• Truncated CP and Tucker Decomposition: Retains only the most significant components [52].
• Randomized Tensor Factorization: Uses random projections to estimate low-rank components [53].
• Sparse and Structured Approximations: Imposes sparsity constraints to enhance interpretability.

4.3. Computational Hardness and Approximation Guarantees

The computational complexity of tensor decomposition is significantly higher than that of matrix decomposition. Several fundamental hardness results have been established:

• Computing the rank of a given tensor is NP-hard for tensors of order three or higher.
• Finding the best rank-R CP decomposition is NP-hard in general [54].
• Tucker decomposition involves solving large-scale SVD problems, which can be computationally prohibitive.

To address these challenges, researchers have developed approximation algorithms that trade accuracy for efficiency. Probabilistic and heuristic methods, such as Alternating Least Squares (ALS) with regularization, have been widely adopted despite lacking theoretical guarantees on convergence speed and global optimality.

4.4. Robustness and Stability in Noisy Environments

Real-world tensor data often contains noise, missing values, and adversarial perturbations. Developing robust tensor decomposition methods is crucial for reliable data analysis.

4.4.1. Tensor Decomposition with Missing Data

Handling missing data in tensors is a fundamental challenge. Strategies for missing data imputation include:

• Low-Rank Tensor Completion: Uses nuclear norm minimization to estimate missing entries.
• Bayesian Tensor Factorization: Incorporates probabilistic priors to model uncertainty.
• Graph-Based Completion: Leverages relational structures in the data to infer missing values [55].

4.4.2. Robust Tensor Decomposition in Noisy Settings

Noisy tensor data arises in applications such as hyperspectral imaging and biomedical signal processing. To enhance robustness, researchers have explored:

• Sparse and Robust CP Decomposition: Incorporates ${\ell }_1$ norm regularization to suppress outliers [19].
• Total Variation (TV) Regularization: Enforces smoothness in tensor factorization for denoising applications.
• Bayesian Nonparametric Models: Uses hierarchical priors to adaptively model noise distributions.

4.4.3. Adversarial Robustness in Machine Learning Applications

Recent studies have shown that tensor-based machine learning models can be vulnerable to adversarial attacks. Open research directions include:

• Defending Against Tensor-Based Adversarial Attacks: Developing regularization techniques to enhance model security.
• Certifiable Robustness of Tensor Factorization: Establishing theoretical guarantees on robustness under perturbations [56].
• Adversarial Training for Tensor Networks: Enhancing resilience of tensor models to adversarial manipulations.

4.5. Theoretical Connections Between Tensor Methods and Other Fields

Tensor decomposition shares deep mathematical connections with several areas of research. These connections provide new insights into tensor factorization and its applications [57].

4.5.1. Tensors and Algebraic Geometry

Algebraic geometry has been instrumental in understanding the structure of tensor decompositions [58]. Key areas of intersection include:

• Tensor Rank Bounds: Using algebraic varieties to establish rank constraints [59].
• Secant Varieties and Decomposability: Studying geometric conditions for unique decomposability.
• Homotopy Methods for Tensor Factorization: Leveraging algebraic topology for efficient decomposition algorithms.

4.5.2. Tensors and Quantum Information Theory

Tensor networks play a crucial role in quantum computing and quantum information theory. Connections between tensor methods and quantum mechanics include:

• Tensor Network Representations of Quantum States: Efficiently encoding quantum many-body systems [60].
• Entanglement Entropy and Tensor Ranks: Analyzing the complexity of quantum entanglement using tensor factorizations [61].
• Quantum Algorithms for Tensor Factorization: Exploring quantum-inspired methods for high-dimensional tensor decomposition.

4.6. Conclusion

Despite significant progress in tensor decomposition, several open theoretical challenges remain. Understanding uniqueness conditions, developing efficient rank estimation techniques, and improving robustness in noisy settings are critical directions for future research. Furthermore, interdisciplinary connections between tensors, algebraic geometry, and quantum mechanics offer promising avenues for advancing both theoretical and applied tensor analysis [62]. Addressing these challenges will not only refine the mathematical foundations of tensor decomposition but also unlock new applications in artificial intelligence, scientific computing, and beyond.

5. Applications of Tensor Decomposition in Signal Processing and Machine Learning

Tensor decomposition techniques have found widespread applications across various domains, particularly in signal processing and machine learning. The ability of tensors to capture multi-way interactions makes them invaluable in representing and analyzing structured data. This section explores key application areas, highlighting how tensor methods contribute to performance improvements and computational efficiency.

5.1. Applications in Signal Processing

Signal processing often involves handling high-dimensional data, such as multi-sensor signals, speech recordings, biomedical signals, and image sequences [63]. Tensor decomposition plays a crucial role in several signal processing tasks, including source separation, array signal processing, and compressed sensing [64].

5.1.1. Blind Source Separation

Blind Source Separation (BSS) aims to recover underlying source signals from observed mixtures without prior knowledge of the mixing process. Traditional matrix-based methods, such as Independent Component Analysis (ICA), often assume statistical independence of sources [65]. However, tensor decomposition provides a more general framework by leveraging the multi-way structure of data.

• CP Decomposition for BSS: The CP decomposition is particularly useful in scenarios where the observed signal is modeled as a sum of rank-one components, each corresponding to an independent source [66].
• Tucker Decomposition for Multimodal Data: When multiple sources are recorded through different modalities (e.g., EEG and fMRI in neuroscience), Tucker decomposition enables joint analysis of multimodal data [67].

Applications of tensor-based BSS include:

• EEG and fMRI Analysis: Identifying independent brain activity sources.
• Speech Processing: Separating overlapping speech signals in audio recordings.
• Wireless Communications: Decoupling multiple transmitted signals in MIMO systems [68].

5.1.2. Array Signal Processing

Array signal processing involves analyzing signals collected by an array of sensors, such as radar, sonar, and antenna arrays in wireless communications [69]. Tensors naturally represent multi-dimensional array data, making decomposition techniques useful in:

• Direction of Arrival (DOA) Estimation: Tensor-based subspace methods improve the accuracy of DOA estimation in multi-antenna systems.
• Beamforming: Tensor decompositions help design optimal beamforming weights for interference suppression [70].
• Channel Estimation: In MIMO communication, tensor methods improve the estimation of channel state information (CSI) [71].

5.1.3. Compressed Sensing and Sparse Signal Recovery

Compressed sensing exploits the sparsity of signals to enable accurate reconstruction from fewer measurements. Tensor decomposition enhances compressed sensing by leveraging multi-way sparsity patterns, leading to more efficient signal recovery. Applications include:

• Medical Imaging: Accelerating MRI and CT image reconstruction [72].
• Wireless Communications: Efficient spectrum sensing in cognitive radio networks.
• Astronomical Imaging: Recovering high-resolution images from sparse telescope data [73].

5.2. Applications in Machine Learning

In machine learning, tensor decompositions serve as powerful tools for dimensionality reduction, feature extraction, deep learning compression, and graph-based learning. Below, we explore some major applications [74].

5.2.1. Dimensionality Reduction and Feature Extraction

Many machine learning tasks involve high-dimensional data, which can lead to issues such as overfitting and increased computational complexity. Tensor decomposition helps in reducing dimensionality while preserving critical information.

• Tucker Decomposition for Feature Selection: By extracting low-dimensional representations of data, Tucker decomposition improves classification and clustering performance.
• CP Decomposition in Natural Language Processing (NLP): Tensor factorization helps in word embedding models and topic modeling.

Examples of applications include:

• Face Recognition: Tensor-based feature extraction improves accuracy in facial recognition systems.
• Text Mining: Tensor-based topic models reveal latent structures in large text corpora.
• Recommender Systems: Tensor decomposition enhances collaborative filtering by capturing higher-order user-item interactions.

5.2.2. Deep Learning Compression and Acceleration

Deep neural networks (DNNs) often suffer from high memory and computational costs [75]. Tensor decomposition provides a structured approach to compressing and accelerating deep learning models [76].

• Tensor Train Decomposition for Model Compression: TT decomposition reduces the number of parameters in fully connected layers, making deep learning models more efficient [77].
• CP and Tucker Decompositions for Convolutional Neural Networks (CNNs): These methods help decompose convolutional filters, reducing computational complexity in CNNs [78].

Tensor-based deep learning acceleration is widely used in:

• Edge and Mobile AI: Deploying efficient neural networks on resource-constrained devices.
• Autonomous Vehicles: Optimizing deep learning models for real-time object detection.
• Medical Diagnosis: Improving efficiency in AI-driven medical image analysis [79].

5.2.3. Knowledge Graph Completion and Graph Learning

Knowledge graphs, which represent relationships between entities in large-scale databases, benefit significantly from tensor-based models. Tensor decomposition enables:

• Link Prediction: Identifying missing relationships in knowledge graphs.
• Graph Embedding: Representing nodes in a low-dimensional space for improved clustering and classification.

Applications include:

• Biomedical Research: Discovering new drug interactions and gene-disease associations.
• Social Network Analysis: Detecting hidden patterns in social media interactions.
• Recommendation Systems: Enhancing personalized recommendations based on multi-relational data.

5.3. Comparison of Tensor Decomposition Applications

Table 2 summarizes the key applications of different tensor decomposition techniques.

5.4. Challenges and Future Directions

Despite their advantages, tensor decomposition methods face several challenges:

• Scalability: Handling large-scale tensors efficiently remains a challenge [80].
• Computational Complexity: Many tensor decomposition algorithms are iterative and require careful optimization.
• Interpretability: While tensor methods provide compact representations, interpreting the results in real-world applications can be complex.

Future research directions include:

• Efficient Parallel and Distributed Algorithms: Leveraging GPU and cloud computing for large-scale tensor computations [81].
• Hybrid Models: Integrating tensor decomposition with deep learning for enhanced performance [82].
• Robustness and Generalization: Developing tensor methods that are robust to noise and missing data [83].

This section has highlighted the broad spectrum of applications of tensor decomposition in signal processing and machine learning, showcasing its impact in real-world problems.

6. Computational Challenges and Optimization Techniques for Tensor Decomposition

Although tensor decomposition provides powerful tools for analyzing multi-way data, its computational complexity and scalability present significant challenges [84,85]. As the size and order of tensors grow, traditional algorithms become impractical due to high memory requirements and increased computational costs [86]. This section discusses key computational challenges associated with tensor decomposition and explores optimization techniques, including efficient algorithms, parallelization strategies, and low-rank approximations.

6.1. Computational Complexity of Tensor Decomposition

Tensor decomposition algorithms often require solving large-scale optimization problems, making their computational complexity a critical concern. Below, we analyze the computational cost of common tensor decompositions [55].

6.1.1. CP Decomposition Complexity

The CANDECOMP/PARAFAC (CP) decomposition approximates an N-way tensor as a sum of rank-one components [87]. The Alternating Least Squares (ALS) algorithm, commonly used for CP decomposition, updates each factor matrix iteratively [88]. The computational complexity per iteration is:

$$O\left(R\sum _{n=1}^N{I}_n\prod _{m\ne n}{I}_m\right),$$

(9)

where R is the rank of decomposition, and $I_n$ is the dimension along mode n. As the tensor order increases, the cost grows exponentially, leading to the well-known curse of dimensionality [89].

6.1.2. Tucker Decomposition Complexity

The Tucker decomposition requires computing a core tensor along with factor matrices. The standard approach uses the Higher-Order Singular Value Decomposition (HOSVD), which involves computing the SVD of mode-n unfolding matrices. The computational complexity is:

$$O\left(\sum _{n=1}^N{I}_n^3\prod _{m\ne n}{I}_m\right).$$

(10)

Since each unfolding matrix can be large, computing Tucker decomposition for high-order tensors can be prohibitive.

6.1.3. Tensor Train and Tensor Ring Complexity

Tensor Train (TT) decomposition provides a scalable alternative for high-order tensors by factorizing them into a sequence of smaller core tensors [90]. The computational complexity of TT decomposition is:

$$O\left(N{R}^{2}I\right),$$

(11)

where R is the TT-rank, and I is the average dimension size [91]. TT decomposition avoids the exponential growth in memory and computation but requires careful rank selection. Tensor Ring (TR) decomposition further generalizes TT by allowing circular dependencies among core tensors. While TR decomposition offers increased flexibility, it also introduces additional optimization complexity [92].

6.2. Memory Constraints and Storage Optimization

High-dimensional tensors require significant storage, making memory-efficient representations essential. Strategies for reducing memory footprint include:

• Sparse Tensor Storage: Instead of storing all elements, sparse representations store only nonzero values and their indices, significantly reducing memory usage [93].
• Compressed Formats: Using quantization and low-bit representations to store tensor elements efficiently.
• Distributed Storage: Splitting tensors across multiple processing units in distributed computing environments.

6.3. Optimization Techniques for Tensor Decomposition

Given the high computational cost, various optimization techniques have been developed to accelerate tensor decomposition [94]. We discuss key strategies below.

6.3.1. Alternating Least Squares (ALS) and Variants

The Alternating Least Squares (ALS) method is a widely used optimization technique for CP and Tucker decomposition. ALS iteratively updates one factor matrix at a time while fixing the others. Despite its simplicity, ALS may suffer from slow convergence and local minima issues. Variants of ALS include:

• Regularized ALS: Adds L2 regularization to prevent overfitting and improve generalization.
• Stochastic ALS: Uses stochastic gradient updates to accelerate convergence.
• Randomized ALS: Incorporates randomization techniques to reduce computational cost [95].

6.3.2. Gradient-Based Optimization

Gradient-based optimization methods, such as Stochastic Gradient Descent (SGD) and Adam optimizer, have been successfully applied to tensor decomposition [96]. These methods are particularly useful when dealing with large-scale tensors.

• SGD for CP and Tucker: Updates factor matrices using mini-batches to improve scalability.
• Second-Order Optimization: Uses Newton’s method and conjugate gradient techniques to accelerate convergence.
• Momentum-Based Optimization: Incorporates momentum to avoid oscillations in the optimization landscape [97].

6.3.3. Randomized and Approximate Methods

Randomized algorithms provide efficient approximations to tensor decompositions with lower computational cost. These methods include:

• Randomized SVD for Tucker: Computes low-rank approximations using random projections.
• Sketching Methods: Uses tensor sketching techniques to reduce dimensionality before applying decomposition [98].
• Probabilistic Tensor Factorization: Applies Bayesian inference to estimate tensor components in an approximate manner.

6.3.4. Parallel and Distributed Computing

As tensor dimensions increase, parallel and distributed implementations become essential for scalability. Approaches include:

• GPU Acceleration: Uses CUDA-based tensor operations to parallelize matrix multiplications.
• Multi-Core Processing: Splits tensor factorization tasks across multiple CPU cores.
• Distributed Tensor Decomposition: Implements factorization on large-scale clusters using frameworks like TensorFlow and Spark.

6.4. Tensor Decomposition in Large-Scale and Streaming Data

Modern applications often involve streaming data, where tensors evolve over time [99]. Traditional batch decomposition techniques may not be efficient in such scenarios. Instead, dynamic and online tensor decomposition methods are required [100].

6.4.1. Incremental and Online Tensor Decomposition

Online tensor decomposition updates factor matrices incrementally as new data arrives. Methods include:

• Online CP Decomposition: Maintains a low-rank tensor approximation that evolves with streaming data [101].
• Incremental Tucker Decomposition: Updates core tensors and factor matrices without recomputing the entire decomposition.

6.4.2. Tensor Decomposition in High-Performance Computing

High-performance computing (HPC) techniques are essential for tensor decomposition in large-scale datasets. Techniques include:

• Supercomputing Implementations: Running tensor decomposition on high-performance clusters [102].
• Hybrid CPU-GPU Models: Combining CPU and GPU processing for improved efficiency [103].
• Cloud-Based Tensor Factorization: Leveraging cloud computing platforms for distributed tensor computations [104].

6.5. Future Directions in Tensor Computation Optimization

As tensor decomposition continues to evolve, several research challenges remain:

• Scalable Algorithms: Developing more efficient algorithms that can handle petabyte-scale tensors.
• Adaptive Rank Selection: Automating the selection of optimal tensor ranks for decomposition.
• Robustness in Noisy Environments: Enhancing tensor methods to handle missing and corrupted data.
• Quantum Tensor Computation: Exploring quantum algorithms for tensor decomposition to achieve exponential speedups [105].

This section has discussed the computational challenges of tensor decomposition and presented optimization techniques that enable large-scale and efficient implementations. The next section explores emerging applications and the future impact of tensor methods in artificial intelligence and data science.

7. Emerging Applications and Future Directions of Tensor Decomposition

As data-driven technologies continue to evolve, tensor decomposition is emerging as a powerful tool in cutting-edge applications across artificial intelligence, computational science, and engineering. The ability to analyze multi-dimensional data efficiently makes tensor methods invaluable in domains such as quantum computing, neuroscience, personalized medicine, and deep learning optimization [106]. This section explores emerging applications of tensor decomposition and discusses future research directions that will shape the next generation of multi-way data analysis.

7.1. Tensor Decomposition in Artificial Intelligence and Machine Learning

Tensor methods have gained significant attention in artificial intelligence (AI) due to their ability to model multi-relational data [107]. Beyond their traditional role in feature extraction and dimensionality reduction, tensors are now being used in areas such as explainable AI, federated learning, and continual learning [108].

7.1.1. Tensor-Based Deep Learning Architectures

Deep learning models, particularly convolutional neural networks (CNNs) and transformer-based architectures, suffer from high memory and computational costs. Tensor decomposition offers solutions for improving efficiency:

• Tensor Compression in Transformers: Large language models (LLMs) such as GPT and BERT can be compressed using tensor train (TT) and tensor ring (TR) decomposition, reducing the number of parameters while preserving accuracy.
• Factorized Convolutional Layers: CP and Tucker decomposition can replace standard convolutional filters in CNNs, leading to faster inference and reduced model size.
• Low-Rank Attention Mechanisms: Tensor-based attention models improve efficiency in vision transformers (ViTs) and self-attention networks [109].

7.1.2. Federated Learning and Distributed AI

Federated learning enables machine learning models to be trained across multiple decentralized devices while preserving data privacy [110]. Tensor methods enhance federated learning in the following ways:

• Efficient Model Aggregation: Tensor factorization reduces the dimensionality of model updates, leading to faster communication in federated learning settings [111].
• Privacy-Preserving AI: Tensor-based representations enable secure and compressed data exchanges in privacy-sensitive applications such as healthcare and finance.

7.1.3. Explainable AI (XAI)

Tensor decomposition enhances interpretability in AI models by providing compact representations of high-dimensional decision spaces. Applications in XAI include:

• Interpretable Neural Networks: Tensor factorization helps visualize and analyze neural network activations to understand model decisions [112].
• Bias Detection in AI: Tensor analysis can uncover hidden biases in machine learning models by analyzing multi-modal data distributions.

7.2. Tensor Decomposition in Neuroscience and Biomedical Engineering

Neuroscience and biomedical applications generate complex, multi-modal data that require advanced analysis techniques [113]. Tensor decomposition is playing a transformative role in brain imaging, genomics, and personalized medicine [114].

7.2.1. Neuroimaging and Brain Signal Analysis

Tensor methods are widely used in analyzing functional MRI (fMRI), electroencephalography (EEG), and magnetoencephalography (MEG) data. Applications include:

• Identifying Brain Networks: CP and Tucker decomposition are used to extract latent brain connectivity patterns from fMRI data.
• EEG Signal Classification: Tensor-based models improve classification accuracy in brain-computer interfaces (BCIs).
• Neurological Disease Diagnosis: Tensor factorization aids in detecting early markers of neurodegenerative diseases such as Alzheimer’s and Parkinson’s.

7.2.2. Personalized Medicine and Genomic Data Analysis

Genomics and personalized medicine require high-dimensional data analysis techniques to uncover genetic patterns and optimize treatments. Tensor decomposition contributes to:

• Drug Discovery: Factorizing drug-response tensors helps in predicting personalized treatment outcomes [115].
• Multi-Omics Integration: Tensor methods combine genetic, transcriptomic, and proteomic data for a comprehensive understanding of diseases.
• Cancer Biomarker Identification: Decomposing patient gene expression tensors aids in identifying biomarkers for precision oncology.

7.3. Tensor Methods in Scientific Computing and Engineering

Scientific computing applications often involve high-dimensional simulations and complex datasets that benefit from tensor-based representations [116].

7.3.1. Computational Chemistry and Quantum Physics

Tensor decomposition is increasingly used in quantum mechanics and computational chemistry:

• Quantum State Representation: Tensor network models provide efficient representations of quantum many-body states.
• Density Matrix Factorization: Low-rank tensor approximations help reduce computational complexity in quantum chemistry simulations.
• Quantum Machine Learning: Tensor-based learning methods optimize quantum circuit designs [117].

7.3.2. Climate Science and Geospatial Data Analysis

Tensor decomposition is helping climate scientists analyze multi-modal geospatial datasets:

• Weather Prediction: Tensor factorization improves climate models by identifying temporal-spatial patterns.
• Remote Sensing: Decomposing hyperspectral satellite images aids in land cover classification and environmental monitoring.
• Disaster Forecasting: Tensor-based anomaly detection identifies patterns in extreme weather events such as hurricanes and wildfires [118].

7.4. Challenges and Future Directions in Tensor Decomposition Research

Despite its growing adoption, tensor decomposition faces several challenges that need to be addressed to fully realize its potential.

7.4.1. Scalability and Computational Efficiency

• Developing faster, scalable algorithms that can handle massive tensors efficiently [119].
• Integrating tensor decomposition with distributed computing frameworks for cloud-based implementations [120].
• Exploring hardware acceleration (e.g., GPU, TPU, and quantum computing) to improve computational speed [121].

7.4.2. Automated and Adaptive Tensor Factorization

Selecting optimal tensor rank is a critical issue that affects accuracy and efficiency. Future research should focus on:

• Adaptive rank estimation techniques to dynamically adjust tensor decomposition models.
• Bayesian and probabilistic tensor methods for uncertainty quantification in factorization.
• Learning-based approaches that use neural networks to optimize tensor decomposition.

7.4.3. Robustness and Generalization in Real-World Data

• Enhancing robustness to noise and missing data in tensor decomposition applications.
• Developing generalizable tensor models that work across different domains and datasets.
• Investigating adversarial robustness of tensor-based models in security-sensitive applications.

7.4.4. Integration with Emerging AI Paradigms

Tensor decomposition is expected to integrate with emerging AI trends:

• Hybrid AI Models: Combining tensor factorization with graph neural networks (GNNs) and transformers [122].
• Neurosymbolic AI: Leveraging tensor methods to improve symbolic reasoning in AI [123].
• Edge AI and IoT: Deploying tensor-optimized AI models on resource-constrained edge devices [55,124].

7.5. Conclusions

Tensor decomposition is rapidly transforming a wide range of scientific and engineering disciplines. From accelerating deep learning models to improving climate predictions, the ability to analyze multi-way data efficiently makes tensor methods indispensable in modern AI and computational science. As research progresses, further advancements in scalability, robustness, and real-world applicability will shape the future of tensor-based machine learning and data analysis [125].

8. Conclusions

Tensor decomposition has emerged as a fundamental mathematical tool with far-reaching implications across signal processing, machine learning, scientific computing, and artificial intelligence. By extending classical matrix factorizations to higher-order data structures, tensor methods enable efficient representation, compression, and analysis of complex multi-dimensional datasets. This paper has provided an in-depth discussion on the mathematical foundations of tensor decomposition, explored its diverse applications, examined recent theoretical advancements, and outlined open challenges that remain in the field.

8.1. Key Takeaways

The study of tensor decomposition has led to several key insights:

• Mathematical Formulations: Different tensor factorization techniques, including CP, Tucker, TT, and TR decompositions, provide versatile frameworks for analyzing multi-way data [126].
• Computational Trade-offs: While tensor methods offer powerful insights, computational complexity remains a major hurdle, necessitating efficient approximation algorithms.
• Emerging Applications: Tensors play a critical role in modern AI, deep learning, biomedical engineering, quantum computing, and geospatial data analysis [127].
• Theoretical Advancements: Ongoing research in uniqueness conditions, rank estimation, and robustness to noise continues to refine the mathematical understanding of tensors.

8.2. Challenges and Future Directions

Despite its widespread utility, tensor decomposition faces several unresolved challenges:

• Scalability and Efficiency: Handling large-scale tensors requires advanced parallel computing techniques and optimized hardware implementations.
• Automated Model Selection: Determining optimal tensor rank and structure remains an open problem, necessitating adaptive and probabilistic approaches.
• Robustness and Interpretability: Ensuring the reliability of tensor-based models in noisy and adversarial settings is crucial for real-world deployment.
• Interdisciplinary Integration: Bridging the gap between tensor methods and fields such as quantum computing, algebraic geometry, and neuroscience presents exciting research opportunities [128].

8.3. Final Remarks

Tensor decomposition continues to revolutionize data-driven research and technological advancements. As computing power and machine learning algorithms evolve, tensor-based methods will become increasingly integral to processing high-dimensional data efficiently. By addressing current limitations and exploring interdisciplinary collaborations, researchers can further enhance the capabilities of tensor decomposition, unlocking new insights across scientific and engineering domains.

With the rapid growth of data-intensive applications, the future of tensor decomposition lies in the development of more scalable, interpretable, and automated frameworks. As theoretical and computational advancements continue to unfold, tensor methods will remain at the forefront of next-generation AI, machine learning, and scientific discovery.