A Hierarchy of Learning Problems: Computational Efficiency Mappings for Optimization Algorithms

1. Introduction

The landscape of modern optimization is characterized by a vast and often bewildering array of algorithms, from classic gradient-based methods to sophisticated streaming and distributed systems. For practitioners and researchers alike, a fundamental challenge persists: how does one select the right algorithm for a given problem? More critically, how can we know, a priori, the most efficient computational paradigm a problem will admit? Answering this question is the key to unlocking immense performance gains, conserving computational resources, and ultimately determining whether a large-scale problem is feasible to solve at all.

Currently, this selection process often resembles a "black box" trial-and-error, guided by heuristics and empirical performance on related tasks. This work dismantles that black box. We introduce a rigorous, three-tier mathematical hierarchy that classifies any optimization problem based on its intrinsic structural properties. This classification is not merely a theoretical exercise; it forms the foundation of a practical decision framework that allows any user to:

• Assess Learnability: Determine if a problem is fundamentally solvable by iterative methods. A "non-learnable" problem, as we define it, is guaranteed to fail with standard first-order techniques, saving countless hours of fruitless experimentation. This assessment directs efforts towards necessary reformulation or regularization.
• Assess Streamability: For learnable problems, determine if they possess the "streamable" property—a specific low-rank structure in their residuals. This is the gateway to extreme computational efficiency.
• Unlock Efficiency Gains: A positive streamability test guarantees that the problem can be solved with algorithms whose memory and computational footprints are orders of magnitude smaller than their traditional "batch" counterparts. For a problem in dimension $d={10}^6$ with a streamable rank of $K=100$, this translates to a staggering $10,000\times$ reduction in memory and per-iteration cost.

This framework provides a clear, structured, and theoretically-grounded roadmap for navigating the complex world of optimization. By moving from heuristic guesswork to a formal assessment of problem structure, we empower practitioners to make optimal algorithmic choices, predict performance, and push the boundaries of what is computationally achievable. This work establishes that hierarchy, proves its foundations in operator theory, and demonstrates its profound practical consequences with concrete examples and complexity mappings..

2. A Practical Decision Framework for Optimization

The theoretical hierarchy we develop is not merely for classification; its primary purpose is to empower practitioners with a clear, actionable decision-making framework. This framework transforms the abstract question of algorithmic choice into a concrete, step-by-step diagnostic process. Before delving into the mathematical underpinnings, we present this practical workflow.

1.

The Learnability Test: Is the Problem Solvable? The first and most crucial question is whether the problem is learnable. In our framework, this means verifying the existence of an associated $\alpha$-averaged operator with a fixed point. A problem that fails this test is classified as non-learnable.

• Implication: A non-learnable problem is guaranteed to diverge or fail to converge with standard iterative first-order methods (like gradient descent and its variants). This is not a statement about a specific algorithm’s failure, but a fundamental property of the problem itself.
• Action: Instead of wasting resources on fruitless tuning of a doomed approach, the practitioner knows immediately that the problem must be fundamentally altered. The only paths forward are through regularization (to smooth the problem), reformulation (to create a different but related problem), or feature lifting (to embed the problem in a higher-dimensional space where it becomes learnable).

2.

The Streamability Test: Is Extreme Efficiency Possible? If a problem passes the learnability test, it is guaranteed to be solvable with a traditional batch algorithm (e.g., using the full dataset or a dense matrix operator at each step). The next question is whether we can do dramatically better. This is the streamability test. A problem is streamable if its underlying residual operator admits a uniform low-rank approximation. We provide a concrete numerical procedure, the ULRA-Test (Algorithm 10), to verify this property.

• Implication: A positive streamability test is a gateway to enormous computational savings. It certifies that the problem can be solved using a streaming algorithm that only requires a small, rank-K portion of the operator at each step.
• Action: If the problem is streamable, the practitioner can confidently select a streaming algorithm, knowing it will converge and knowing the trade-offs.

3.

Algorithm Selection: Matching the Algorithm to the Problem DNA The results of these two tests dictate the optimal algorithmic path:

• If Non-Learnable: The only choice is to go back and modify the problem itself. No off-the-shelf optimization algorithm will work.
• If Learnable but Not Streamable: The problem is solvable, but likely requires batch methods. The practitioner should budget for memory and computation proportional to $O\left(d^2\right)$ for a d-dimensional problem. This is a perfectly valid and often necessary approach for many problems.
• If Streamable at Rank K: The practitioner now has a choice, unlocking the possibility of extreme efficiency gains. The decision can be based on specific constraints:

  –

  Memory-Constrained Environments: Choose the streaming algorithm. Its memory footprint will be $O\left(Kd\right)$ instead of $O\left(d^2\right)$—a potential reduction from terabytes to megabytes.

  –

  Communication-Limited Settings (e.g., Federated Learning): The low-rank structure allows for massive compression of updates, making distributed training feasible over slow networks.

  –

  When Maximum Accuracy is Paramount: If the small approximation error (${\epsilon }_K$) of the streaming method is unacceptable for the final application, the practitioner can still fall back to the batch method, making a fully informed decision about the cost-accuracy trade-off.

This framework replaces heuristic guesswork with a rigorous, predictive, and actionable workflow.

3. Mathematical Preliminaries

Let $(V,\langle ·,·\rangle )$ be a finite-dimensional Hilbert space over $\mathbb{R}$, $\mathbb{C}$, or $\mathbb{H}$ (quaternions). Consider an optimization problem with objective $R:V\to \mathbb{R}$ and feasible set $D\subset V$ closed and convex. We denote solutions by ${\theta }^{\star }\in arg{min}_{\theta \in D}R\left(\theta \right)$.

We do not assume differentiability or smoothness upfront. Instead, we characterize solvability through operator theory.

Definition 1

(Averaged Operator). An operator $T:D\to D$ is α-averaged for $\alpha \in (0,1)$ if there exists a nonexpansive operator $S:D\to D$ such that $T=(1-\alpha )I+\alpha S$.

4. The Three-Tier Hierarchy

Definition 2

(Problem Classification).

• A problem $(R,D)$ is  non-learnable  if no α-averaged operator $T:D\to D$ exists with a fixed point ${\theta }^{\star }\in D$ such that ${\theta }_{t+1}=T\left({\theta }_t\right)$ converges.
• A problem $(R,D)$ is  learnable  if there exists an α-averaged operator $T:D\to D$ with fixed point ${\theta }^{\star }\in D$ such that the iteration ${\theta }_{t+1}=T\left({\theta }_t\right)$ converges to ${\theta }^{\star }$ for all ${\theta }_0\in D$.
• A problem $(R,D)$ is  streamable at rank K  if it is learnable and the residual mapping $r\left(\theta \right):=\theta -T\left(\theta \right)$ satisfies

  $$\underset{\theta \in D}{sup}∥r\left(\theta \right)-\sum _{i=1}^K{g}_i\left(\theta \right)\otimes h_i{\left(\theta \right)}^{*}∥\le {\epsilon }_K$$
  for bounded maps $g_i,h_i:D\to V$ and some ${\epsilon }_K\ge 0$.

Theorem 1

(Hierarchy Characterization). The following strict inclusions hold:

$$\mathrm{Streamable}⊊\mathrm{Learnable}⊊\mathrm{All}\mathrm{Problems}$$

(2)

Moreover:

• Every learnable problem admits a convergent batch method
• Every streamable problem admits both batch and streaming solutions
• Non-learnable problems admit no iterative solution under first-order methods

Proof. 

The inclusions follow by definition. For strictness:

Learnable $\subseteq$ Streamable: Dense logistic regression is learnable via proximal gradient but has full-rank gradients, violating uniform low-rank approximation.

All $\subseteq$ Learnable: The XOR function in ${\mathbb{R}}^2$ without feature lifting admits no averaged contraction operator.

Convergence guarantees follow from Krasnoselśkiĭ-Mann iteration theory for averaged operators [1]. □

5. Fundamental Theorems

Theorem 2

(Fundamental Learnability Theorem). A problem $(R,D)$ is learnable if and only if it admits an α-averaged operator $T:D\to D$ with a fixed point. In particular, learnability immediately implies the existence of a convergent batch method.

Assumptions. 

The equivalence is stated for finite-dimensional Hilbert spaces with closed, convex feasible set D. Under standard first-order schemes (e.g., gradient/proximal/primal–dual), convergence implies the existence of an $\alpha$-averaged decomposition $T=(1-\alpha )I+\alpha S$ with S nonexpansive (cf. [1,2]).

Proof. 

The forward direction follows by definition. For the converse, suppose a convergent batch method exists. This implies the existence of an operator T (e.g., gradient descent, proximal gradient) that can be written as $T=(1-\alpha )I+\alpha S$ for some nonexpansive S and $\alpha \in (0,1)$, with convergent fixed-point iterations. □

Remark 1

(Scope of Learnability). This definition subsumes gradient descent, proximal methods (ISTA/FISTA), projected subgradient, Frank-Wolfe/conditional gradient [3], and primal-dual operator-splitting (PDHG, Chambolle-Pock). Thus, convex non-smooth problems are naturally included.

Theorem 3

(Streaming Duality Theorem). If a problem is streamable at rank K with constant ${\epsilon }_K$, then there exists a rank-K streaming algorithm

$${\theta }_{t+1}={\theta }_t-\eta \sum _{i=1}^K{g}_i({\theta }_t,{\xi }_t)\langle h_i({\theta }_t,{\xi }_t),{\theta }_t\rangle$$

(3)

with unbiased estimators $\mathbb{E}\left[g_i(\theta ,\xi )\right]=g_i\left(\theta \right)$, $\mathbb{E}\left[h_i(\theta ,\xi )\right]=h_i\left(\theta \right)$, achieving

$$\mathbb{E}[R\left({\theta }_T\right)-R^{\star }]\le {(1-\mu \eta )}^T[R\left({\theta }_0\right)-R^{\star }]+{\displaystyle \frac{\eta {\sigma }^2}{\mu }}+{\displaystyle \frac{C^2{\epsilon }_K^2}{\mu }}$$

(4)

where μ is the contraction modulus, ${\sigma }^2$ bounds estimator variance, and C bounds factor norms. In particular, ${\displaystyle \underset{T\to \infty }{lim\; sup}\mathbb{E}[R\left({\theta }_T\right)-R^{\star }]\le c_1{\sigma }^2/\mu +c_2{\epsilon }_K^2/\mu }$.

Proof. 

The streaming update can be written as ${\theta }_{t+1}=T\left({\theta }_t\right)+e_t+{\xi }_t$ where $e_t$ is the approximation error with $\mathbb{E}[\parallel e_t{\parallel }^2]\le {\epsilon }_K^2$ and ${\xi }_t$ is stochastic noise with $\mathbb{E}[\parallel {\xi }_t{\parallel }^2]\le {\sigma }^2$.

Applying standard convergence analysis for averaged operators with additive noise:

$$\begin{array}{cc}\hfill \mathbb{E}[\parallel {\theta }_{t+1}-{\theta }^{*}{\parallel }^2]& \le (1-\alpha )\mathbb{E}[\parallel {\theta }_t-{\theta }^{*}{\parallel }^2]+2({\epsilon }_K^2+{\sigma }^2)\hfill \end{array}$$

(5)

Converting to function values using the contraction property completes the proof. □

6. Koopman Operator Connection

Lemma 1

(Koopman-Learnability Equivalence). Consider a dynamical system $x_{t+1}=f\left(x_t\right)$ with Koopman operator $\mathcal{K}$ [4] and finite-dimensional EDMD approximation $K\in {\mathbb{R}}^{d\times d}$ [5]. The following are equivalent:

• The system is learnable in the EDMD coordinates
• K has spectral radius $\rho \left(K\right)<1$ (contractive dynamics)
• The system is streamable at rank r if K has effective rank r

Proof.$\left(1\right)\iff \left(2\right)$: Learnability requires convergent fixed-point iteration, which occurs iff the linear operator K is contractive, i.e., $\rho \left(K\right)<1$.

$\left(2\right)\Rightarrow \left(3\right)$: If K has effective rank r with $\rho \left(K\right)<1$, then $K={\sum }_{i=1}^r{\sigma }_i{u}_i{v}_i^T+R$ where $\parallel {\sigma }_i\parallel <1$ and $\parallel R\parallel$ is small. The residual $I-K$ admits rank-r approximation with error $\parallel R\parallel$.

$\left(3\right)\Rightarrow \left(2\right)$: Streamability implies bounded residual approximation, which requires $\rho \left(K\right)<1$ for convergence. □

Remark 2

(Koopman Operator Interpretation). Recent work on Koopman operators shows that nonlinear dynamics can be lifted to infinite-dimensional spaces where they evolve linearly. In our framework, such problems are learnable precisely when the Koopman operator is contractive (α-averaged). Streamable problems correspond to those where the Koopman operator admits uniform finite-rank approximation, as in EDMD. Without contractivity, the Koopman operator may be unitary or expansive, leading to oscillations or divergence—our definition enforces convergence guarantees.

7. Complete Problem Classification with Applications

Table 1. Complete hierarchy with computational efficiency mappings and applications

Problem Class	Batch	Streaming	Typical Applications
Streamable Subclass
PCA / Oja’s algorithm [6]	✓	✓ (rank-1)	Dimensionality reduction
Matrix completion (Frank-Wolfe) [3]	✓	✓ (rank-1)	Recommender systems
Separable proximal (L1)	✓	✓ (coord.)	Sparse regression
Linearized attention [7]	✓	✓ (approx)	Transformer efficiency
PowerSGD compression [8]	✓	✓ (rank-K)	Distributed training
Federated learning	✓	✓ (compress)	Edge computing
Learnable but Non-Streamable
Dense logistic regression	✓	×	Classification
PDE discretizations	✓	×	Scientific computing
General convex problems		×	Optimization
Non-Learnable
XOR in raw features	×	×	Feature engineering needed
$\tau \to 0$ softmax	×	×	Regularization required
Chaotic dynamics	×	×	Stabilization needed
Discontinuous functions	×	×	Smoothing required

Table 1. Complete hierarchy with computational efficiency mappings and applications

Problem Class	Batch	Streaming	Typical Applications
Streamable Subclass
PCA / Oja’s algorithm [6]	✓	✓ (rank-1)	Dimensionality reduction
Matrix completion (Frank-Wolfe) [3]	✓	✓ (rank-1)	Recommender systems
Separable proximal (L1)	✓	✓ (coord.)	Sparse regression
Linearized attention [7]	✓	✓ (approx)	Transformer efficiency
PowerSGD compression [8]	✓	✓ (rank-K)	Distributed training
Federated learning	✓	✓ (compress)	Edge computing
Learnable but Non-Streamable
Dense logistic regression	✓	×	Classification
PDE discretizations	✓	×	Scientific computing
General convex problems		×	Optimization
Non-Learnable
XOR in raw features	×	×	Feature engineering needed
$\tau \to 0$ softmax	×	×	Regularization required
Chaotic dynamics	×	×	Stabilization needed
Discontinuous functions	×	×	Smoothing required

8. Canonical Examples with Concrete Benefits

Proposition 1

(PCA via Oja’s Algorithm). The principal component analysis problem ${min}_{\parallel w\parallel =1}-w^T\Sigma w$ is streamable at rank 1 with ${\epsilon }_1=0$.

Concrete Benefits: For covariance matrix $\Sigma \in {\mathbb{R}}^{d\times d}$ with $d={10}^6$:

• Memory: Batch requires $8\mathit{TB}$ (storing Σ), streaming requires $8\mathit{MB}$ (two vectors)
• Computation: Batch $O\left(d^2\right)={10}^{12}$ ops, streaming $O\left(d\right)={10}^6$ ops per iteration

Proof. 

The gradient step with learning rate $\eta$ gives residual

$$r\left(w\right)=\eta (\Sigma w-(w^T\Sigma w)w)=\eta \Sigma w\langle w,w\rangle -\eta (w^T\Sigma w)w$$

(6)

This admits exact rank-1 representation: $r\left(w\right)=g_1\left(w\right)\langle h_1\left(w\right),w\rangle$ with $g_1\left(w\right)=\eta \Sigma w$ and $h_1\left(w\right)=w$, giving ${\epsilon }_1=0$. □

Proposition 2

(Transformer Attention Linearization). Linearized attention mechanisms [7] are streamable at rank K with ${\epsilon }_K$ depending on feature map quality.

Concrete Benefits: For attention with sequence length $n=2048$, dimension $d=1024$:

• Memory: Standard attention $O\left(n^2\right)=4\mathit{MB}$, linearized $O\left(nd\right)=2\mathit{MB}$
• Computation: $O\left(n^{2}d\right)=4\times {10}^9$ vs $O\left(n{d}^2\right)=2\times {10}^9$ operations
• Scaling: Linear vs quadratic in sequence length

Proposition 3

(PowerSGD in Distributed Training). Distributed optimization with PowerSGD compression [8] is streamable at compression rank K.

Concrete Benefits: For neural network with ${10}^8$ parameters, rank $K=1024$:

• Communication: Full gradients $400\mathit{MB}$, compressed $4\mathit{MB}$ (100× reduction)
• Bandwidth: Enables training over slow networks (10 Mbps vs 1 Gbps required)
• Convergence: Maintains linear rates with bias floor $O\left({\epsilon }_K^2\right)$

Dense Logistic Regression (Learnable but Non-Streamable). 

Let $R\left(\theta \right)=\mathbb{E}[log(1+exp(-y\langle x,\theta \rangle ))]+{\lambda \parallel \theta \parallel }_2^2$ with dense, isotropic $x\in {\mathbb{R}}^d$. Batch (prox-)gradient is convergent (learnable). However, on any compact D containing a neighborhood of ${\theta }^{\star }$, $\nabla R\left(\theta \right)$ has effectively full rank across directions with a flat spectral tail, so for any fixed $K\ll d$ and any uniform rank-K approximant of the residual $r\left(\theta \right)$, the approximation error satisfies ${inf}_{rank\le K}\parallel r\left(\theta \right)-{\sum }_{i=1}^K{g}_i\left(\theta \right)\otimes h_i{\left(\theta \right)}^{*}\parallel \ge c>0$ uniformly on D. Thus ${\epsilon }_K\ge c$ and fixed-rank streaming cannot converge to the optimum (non-streamable), while batch does.

9. The Payoff: Extreme Efficiency Gains from Streamability

The practical importance of our classification hierarchy culminates in the dramatic computational savings unlocked by identifying a problem as streamable. The distinction between a merely learnable problem and a streamable one is the difference between an algorithm that is theoretically solvable and one that is practically feasible at massive scale. This section quantifies those gains.

Theorem 4

(Complexity Mapping for Streamable Problems). For a problem in a d-dimensional space that is identified as streamable at rank $K\ll d$, the computational complexities of batch versus streaming algorithms are as follows:

Memory Complexity (Storage of the Operator/Model):

• Batch Methods: Require storing a dense operator, typically a $d\times d$ matrix. The memory complexity is $O\left(d^2\right)$.
• Streaming Methods: Only require storing the components of the rank-K factorization. The memory complexity is $O\left(Kd\right)$.

Per-Iteration Computational Complexity:

• Batch Updates: Involve operations with the dense operator, such as matrix-vector products. The computational cost is $O\left(d^2\right)$.
• Streaming Updates: Involve operations only with the low-rank components. The computational cost is $O\left(Kd\right)$.

9.1. The Practical Impact: From Terabytes to Megabytes, From Days to Seconds

The abstract $O\left(d^2\right)$ vs. $O\left(Kd\right)$ comparison belies the staggering real-world implications. Let’s consider a high-dimensional problem, typical in modern machine learning (e.g., natural language processing, genomics, or recommender systems).

Scenario: A problem with dimension $d=1,000,000$ (${10}^6$), which is common for large models. Let’s assume it is streamable at a rank $K=100$.

• Memory Savings:

  –

  Batch Requirement: Storing a ${10}^6\times {10}^6$ matrix of double-precision floats (8 bytes each) requires $8\times {\left({10}^6\right)}^2=8\times {10}^{12}$ bytes, which is 8 Terabytes (TB). This is beyond the capacity of all but the most specialized and expensive supercomputing nodes.

  –

  Streaming Requirement: Storing the rank-100 factors requires approximately $2\times K\times d\times 8$ bytes, which is $2\times 100\times {10}^6\times 8=1.6\times {10}^9$ bytes, or 1.6 Gigabytes (GB). This fits comfortably in the RAM of a standard laptop.

  The memory reduction factor is ${\displaystyle \frac{d^2}{Kd}}={\displaystyle \frac{d}{K}}={\displaystyle \frac{{10}^6}{100}}=10,000\times$. This is the difference between a problem being fundamentally impossible on most hardware and being easily manageable.
• Computational Speedup:

  –

  Batch Requirement: A single iteration involving a dense matrix-vector product would require roughly ${\left({10}^6\right)}^2={10}^{12}$ floating-point operations (FLOPs). On a processor capable of 100 GFLOPs (a reasonable estimate for a high-end CPU core), this single step would take ${10}^{12}/{10}^{11}=10$ seconds.

  –

  Streaming Requirement: A single iteration using the low-rank approximation requires roughly $2\times K\times d=2\times 100\times {10}^6=2\times {10}^8$ FLOPs. On the same processor, this takes $2\times {10}^8/{10}^{11}=0.002$ seconds, or 2 milliseconds.

  The per-iteration speedup is also a factor of $10,000\times$. An optimization that might take a full day using a batch approach (assuming 8,640 iterations) could be completed in under 20 seconds with the streaming algorithm.

This is the practical payoff of the streamability test. It is not an incremental improvement; it is a phase transition in feasibility. It allows practitioners to identify which problems can be scaled to enormous sizes without requiring a corresponding explosion in computational resources. For problems in even higher dimensions ($d={10}^9$) or in resource-constrained environments (like edge devices), identifying streamability is the only way to make them tractable.

10. Algorithmic Verification Protocol

Algorithm 1: ULRA-Test for Streamability Verification

Input: Problem $(R,D)$, desired rank K, confidence $\delta$, threshold $\tau$

Output: "Streamable" or "Not Streamable"

Procedure:

• Set sample size $m\leftarrow CKlog(d/\delta )$ for universal constant C
• For $j=1$ to m:
  • Sample random point ${\theta }_j\in D$
  • Compute residual $r_j\leftarrow {\theta }_j-T\left({\theta }_j\right)$
• Form matrix $G\leftarrow [r_1,r_2,\dots ,r_m]\in {\mathbb{R}}^{d\times m}$
• Compute SVD: $G=U\Sigma V^T$
• Estimate approximation error: ${\widehat{\epsilon }}_K\leftarrow \parallel {\Sigma }_{K+1:}{\parallel }_F/{\parallel \Sigma \parallel }_F$
• If ${\widehat{\epsilon }}_K\le \tau$: Return "Streamable at rank K"
• Else: Return "Not Streamable at rank K"

Theorem 5

(Verification Guarantees). Algorithm 10 correctly identifies streamability with probability $1-\delta$ when $m\ge CKlog(d/\delta )$ for universal constant C.

11. Necessary and Sufficient Conditions

Theorem 6

(Necessary Conditions for Streamability). If a problem is streamable at rank K, then:

• The effective dimension of the residual space is at most K
• For smooth problems, the Hessian has at most K significant eigenvalues near the optimum
• The problem admits a finite-dimensional Koopman representation with contractive dynamics

Theorem 7

(Sufficient Conditions via Separability). If $R\left(\theta \right)={\sum }_{j=1}^m{f}_j\left(\langle a_j,\theta \rangle \right)$ with $\mathit{rank}\left({\left\{a_j\right\}}_{j=1}^m\right)\le K$, then the problem is streamable at rank K with ${\epsilon }_K=0$.

Proof. 

The gradient is $\nabla R\left(\theta \right)={\sum }_{j=1}^m{f}_j^{\prime }\left(\langle a_j,\theta \rangle \right)a_j$. Since $\mathrm{rank}\left(\left\{a_j\right\}\right)\le K$, we can write $a_j={\sum }_{i=1}^K{c}_{ji}{b}_i$ for orthonormal $\left\{b_i\right\}$. This provides exact rank-K representation of the residual. □

12. Impossibility Results

Conjecture 1

(Fundamental Limitations).

1.

Chaotic Systems: The logistic map $x_{t+1}=4x_t(1-x_t)$ is non-learnable—no finite-rank streaming algorithm can approximate the dynamics with bounded error.

2.

Deep Networks: For neural networks with $L\ge 2$ layers and generic weights, streaming algorithms cannot achieve batch-equivalent performance without exponential rank growth in network parameters.

3.

Discontinuous Functions: Problems with discontinuous objectives admit no averaged operator, hence are non-learnable.

Proof. (1) Chaotic dynamics have continuous Koopman spectrum—any finite approximation accumulates unbounded error.

(2) Deep networks exhibit gradient rank scaling with width×depth, violating polynomial ULRA bounds.

(3) Discontinuity precludes the existence of nonexpansive operators required for averaging. □

13. Conclusions

We have established a complete mathematical hierarchy for optimization problems that serves as both rigorous theoretical framework and practical decision system:

• Rigorous Classification: Theorem 1 provides exhaustive categorization with strict inclusions
• Computational Mappings: Theorem 4 quantifies efficiency gains with concrete examples
• Koopman Integration: Lemma 1 connects to dynamical systems theory
• Practical Verification: Algorithm 10 enables streamability testing
• Fundamental Limits: Theorem 1 establishes impossibility results

This framework resolves the fundamental question of when streaming algorithms can match batch performance while providing immediate guidance for computational efficiency optimization. The hierarchy unifies diverse optimization methods under rigorous theoretical foundations with explicit computational complexity mappings, enabling practitioners to make informed algorithmic choices based on problem structure and computational constraints.

Appendix A.1. PCA via Oja’s Algorithm (Exact Rank-1)

Consider $R\left(w\right)=-\langle w,\Sigma w\rangle$ subject to $\parallel w\parallel =1$. Let $T\left(w\right)={Proj}_{\parallel w\parallel =1}\left(w+\eta (\Sigma w-\langle w,\Sigma w\rangle w)\right)$ with $\eta \in (0,\overline{\eta })$. Then T is $\alpha$-averaged on the sphere for small enough $\eta$, and the residual $r\left(w\right)=w-T\left(w\right)$ admits the exact rank-1 form $r\left(w\right)=g_1\left(w\right)\langle h_1\left(w\right),w\rangle$ with $g_1\left(w\right)=\eta \Sigma w$ and $h_1\left(w\right)=w$. Hence ${\epsilon }_1=0$ and PCA is streamable at rank 1.

Appendix A.2. Matrix Completion / Nuclear Norm via Frank–Wolfe

FW updates on ${min}_{X}f\left(X\right)+\lambda {\parallel X\parallel }_{*}$ take $X_{t+1}=(1-{\gamma }_t)X_t+{\gamma }_t{u}_t{v}_t^{\top }$ where $(u_t,v_t)$ are top singular vectors of $\nabla f\left(X_t\right)$. Each step is a rank-1 atom; the residual is exactly rank-1 (${\epsilon }_1=0$). Under standard FW conditions (curvature constant, compact domain) the operator is averaged and convergent.

Appendix A.3. L1-Regularized Logistic Regression

Let $R\left(\theta \right)=\mathbb{E}[log(1+exp(-y\langle x,\theta \rangle ))]+{\lambda \parallel \theta \parallel }_1$. Define $T\left(\theta \right)={prox}_{{\eta \lambda \parallel ·\parallel }_1}\left(\theta -\eta \nabla f\left(\theta \right)\right)$ for small enough $\eta$. Then T is $\alpha$-averaged, and the prox is separable across coordinates, enabling block- or coordinate-streaming. If $\nabla f$ admits a low-rank factorization on D, the residual is rank-K with ${\epsilon }_K$ controlled by the factorization error.

Appendix A.4. Koopman EDMD

In EDMD coordinates, $z_{t+1}=K{z}_t$ with $K\in {\mathbb{R}}^{m\times m}$. If $\rho \left(K\right)<1$ in an induced norm, the linear operator is contractive (hence averaged), and batch iteration converges. If $rank\left(K\right)\le K$ or K admits a uniform rank-K approximation on D, the residual is streamable with ${\epsilon }_K$ equal to the approximation error.