Dense Approximation of Learnable Problems with Streamable Problems

1. Introduction

The three-tier classification of optimization problems based on $\alpha$-averaged operators—non-learnable, learnable, and streamable—provides a framework for understanding computational tractability in machine learning [1]. While learnable problems admit convergent $\alpha$-averaged operators, streamable problems additionally possess uniform low-rank residual approximation enabling $\mathcal{O}\left(K\right(m+n\left)\right)$ updates instead of $\mathcal{O}\left(mn\right)$ for parameter matrices $\theta \in {\mathbb{R}}^{m\times n}$.

This computational advantage raises a natural theoretical question: How restrictive is the streamable class? If streamable problems form only a small subset of learnable problems, then the computational benefits come at the cost of severely limiting the problems we can solve. Conversely, if streamable problems are dense within learnable problems, then streamable optimization provides computational efficiency without fundamental limitations on problem scope.

We resolve this question by proving density: every learnable problem can be approximated arbitrarily closely by a streamable problem. This result has both theoretical and practical significance. Theoretically, it shows that streamable optimization is not a restrictive special case but rather a general computational paradigm. Practically, it guarantees that any learnable optimization task can benefit from streamable methods with controllable approximation error.

2. Mathematical Framework

We work within the established three-tier hierarchy, focusing on the relationship between learnable ($\mathcal{L}$) and streamable ($\mathcal{S}$) problems using $\alpha$-operator theory.

2.1. Problem Setup and Assumptions

Assumption A1

(Hilbert Space Setting). We work in the finite-dimensional Hilbert space $V={\mathbb{R}}^{m\times n}$ equipped with the Frobenius inner product ${\langle A,B\rangle }_F=\mathrm{tr}\left(A^{T}B\right)$ and induced norm ${\parallel A\parallel }_F=\sqrt{{\langle A,A\rangle }_F}$. The feasible region $\mathsf{\Theta }\subset V$ is nonempty, closed, and compact.

This setting aligns with the general finite-dimensional Hilbert space framework established in the original hierarchy paper while providing concrete structure for our analysis.

Consider optimization problems ${min}_{\theta \in \mathsf{\Theta }}R\left(\theta \right)$ where $\theta \in V$ and $R:\mathsf{\Theta }\to \mathbb{R}$ is the objective function.

Definition 1

($\alpha$-Averaged Operator). An operator $T:\mathsf{\Theta }\to \mathsf{\Theta }$ is α-averaged for $\alpha \in (0,1)$ if there exists a nonexpansive operator $S:\mathsf{\Theta }\to \mathsf{\Theta }$ such that $T=(1-\alpha )I+\alpha S$.

Definition 2

(Problem Classification via $\alpha$-Operators).

• A problem is non-learnable if no α-averaged operator $T:\mathsf{\Theta }\to \mathsf{\Theta }$ exists with convergent fixed-point iteration.
• A problem is learnable if there exists an α-averaged operator $T:\mathsf{\Theta }\to \mathsf{\Theta }$ with fixed point ${\theta }^{*}$ such that ${\theta }_{t+1}=T\left({\theta }_t\right)$ converges to ${\theta }^{*}$.
• A problem is streamable at rank K if it is learnable and the residual mapping $r\left(\theta \right):=\theta -T\left(\theta \right)$ satisfies:

  $$\underset{\theta \in \mathsf{\Theta }}{sup}{∥r\left(\theta \right)-\sum _{k=1}^K{g}_k\left(\theta \right)\otimes h_k{\left(\theta \right)}^{*}∥}_F\le {\epsilon }_K$$

  (1)
  for bounded maps $g_k,h_k:\mathsf{\Theta }\to V$ and some ${\epsilon }_K\ge 0$.

2.2. Residual Distance Metric

Definition 3

(Uniform Residual Metric). For learnable problems with α-averaged operators $T_1,T_2$ and residuals $r_1\left(\theta \right)=\theta -T_1\left(\theta \right)$, $r_2\left(\theta \right)=\theta -T_2\left(\theta \right)$, define:

$$d_R(T_1,T_2)=\underset{\theta \in \mathsf{\Theta }}{sup}{\parallel r_1\left(\theta \right)-r_2\left(\theta \right)\parallel }_F$$

(2)

3. Main Result: Density Theorem

Theorem 1

(Density of Streamable Problems). Streamable problems are dense in learnable problems under the uniform residual metric. Specifically, for every learnable problem with α-averaged operator $T_0$ and every $\epsilon >0$, there exists a streamable problem with α-averaged operator $T_{\epsilon }$ such that $d_R(T_0,T_{\epsilon })<\epsilon$.

3.1. Constructive Proof via Tensor Decomposition

Proof of Theorem 1.

Let $T_0$ be the $\alpha$-averaged operator of a learnable problem with residual $r_0\left(\theta \right)=\theta -T_0\left(\theta \right)$, and let $\epsilon >0$ be the desired approximation tolerance.

Step 1: Lipschitz bound and discretization. Since $T_0$ is $\alpha$-averaged, it is 1-Lipschitz, hence $r_0\left(\theta \right)=(I-T_0)\left(\theta \right)$ satisfies:

$$\parallel r_0\left({\theta }_1\right)-r_0\left({\theta }_2\right){\parallel }_F=\parallel (I-T_0)({\theta }_1-{\theta }_2){\parallel }_F\le 2{\parallel {\theta }_1-{\theta }_2\parallel }_F$$

(3)

Thus $r_0$ is L-Lipschitz with $L=2$. Construct a $\delta$-net $\{{\theta }_1,\dots ,{\theta }_N\}\subset \mathsf{\Theta }$ with $\delta =\epsilon /\left(8\right)$.

Step 2: Residual tensor formation. Define the third-order tensor $\mathcal{R}\in {\mathbb{R}}^{m\times n\times N}$ with slices $\mathcal{R}(:,:,i)=r_0\left({\theta }_i\right)$.

Step 3: Uniform low-rank approximation. Compute a rank-K CANDECOMP/PARAFAC decomposition ensuring uniform slice-wise error:

$$\underset{i=1,\dots ,N}{max}{∥\mathcal{R}(:,:,i)-\sum _{k=1}^K{g}_k\otimes h_k·c_k\left(i\right)∥}_F\le \epsilon /4$$

(4)

This can be achieved by weighted CP fitting or constrained optimization over the CP factors.

Step 4: Coefficient extension. For each k, define discrete coefficients ${\alpha }_i^{\left(k\right)}=c_k\left(i\right)$ at sample points. Extend to Lipschitz functions ${\alpha }^{\left(k\right)}:\mathsf{\Theta }\to \mathbb{R}$ using Shepard’s method (partition of unity):

$${\alpha }^{\left(k\right)}\left(\theta \right)={\displaystyle \frac{{\sum }_{i=1}^N{w}_i\left(\theta \right){\alpha }_i^{\left(k\right)}}{{\sum }_{i=1}^N{w}_i\left(\theta \right)}},w_i\left(\theta \right)={\displaystyle \frac{1}{\parallel \theta -{\theta }_i{\parallel }_F+\delta }}$$

(5)

This extension has Lipschitz constant $L_{\mathrm{ext}}\le C/\delta$ for some universal constant C.

Step 5: Streamable operator construction via convex surrogate. Define a convex surrogate objective $R_{\epsilon }$ whose gradient has the desired rank-K structure:

$$R_{\epsilon }\left(\theta \right)={\displaystyle \frac{1}{2}}\sum _{k=1}^K{\beta }_k{\langle g_k,\theta \rangle }_F^2+{\displaystyle \frac{1}{2}}\sum _{k=1}^K{\gamma }_k{\langle h_k,\theta \rangle }_F^2$$

(6)

where coefficients ${\beta }_k,{\gamma }_k$ are chosen to match the rank-K residual structure. The gradient descent operator $T_{\epsilon }\left(\theta \right)=\theta -\eta \nabla R_{\epsilon }\left(\theta \right)$ is $\alpha$-averaged for appropriate step size $\eta >0$ by standard results [2].

Step 6: Error bound. The approximation error satisfies:

$$\begin{array}{cc}\hfill d_R(T_0,T_{\epsilon })& =\underset{\theta \in \mathsf{\Theta }}{sup}{\parallel r_0\left(\theta \right)-r_{\epsilon }\left(\theta \right)\parallel }_F\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill & \le {\displaystyle \frac{\epsilon }{4}}+L·\delta +{\displaystyle \frac{\epsilon }{4}}={\displaystyle \frac{\epsilon }{4}}+2·{\displaystyle \frac{\epsilon }{8}}+{\displaystyle \frac{\epsilon }{4}}={\displaystyle \frac{3\epsilon }{4}}<\epsilon \hfill \end{array}$$

(8)

where the terms correspond to CP approximation error, Lipschitz interpolation error, and discretization error respectively.    □

Remark 1

(Density vs. Strict Inclusion). The density result does not contradict the established strict inclusion $\mathcal{S}⊊\mathcal{L}$ from the hierarchy. Density means every learnable problem can be approximated arbitrarily well by streamable ones, but the required rank $K\left(\epsilon \right)$ may grow rapidly (potentially exponentially) as $\epsilon \to 0$. This aligns with the impossibility results for deep networks: approximation is always possible, but computational gains may be erased if $K\left(\epsilon \right)$ becomes too large.

3.2. Algorithmic Implementation

Algorithm 1 Learnable to Streamable Conversion via $\alpha$-Operators

Require: Learnable problem with $\alpha$-averaged operator $T_0$, tolerance $\epsilon >0$  1: Compute residual function $r_0\left(\theta \right)=\theta -T_0\left(\theta \right)$  2: Set $\delta =\epsilon /8$ and construct $\delta$-net ${\left\{{\theta }_i\right\}}_{i=1}^N$ of $\mathsf{\Theta }$  3: Compute residuals $R_i=r_0\left({\theta }_i\right)$ for all sample points  4: Form tensor $\mathcal{R}(:,:,i)=R_i$ and compute rank-K CP decomposition with uniform slice error $\le \epsilon /4$  5: Extract factors $g_k,h_k$ and coefficients $c_k$  6: Define coefficient functions ${\alpha }^{\left(k\right)}\left(\theta \right)$ via Shepard interpolation  7: Construct convex surrogate $R_{\epsilon }$ with gradient structure matching rank-K residual  8: return Streamable $\alpha$-averaged operator $T_{\epsilon }\left(\theta \right)=\theta -\eta \nabla R_{\epsilon }\left(\theta \right)$

4. Complete Analysis: ReLU Network Training

We provide a detailed analysis of ReLU network training to demonstrate the theory in practice.

4.1. Problem Setup

Consider a two-layer ReLU network $f(x;\theta )=W_2\sigma (W_{1}x+b_1)+b_2$ where $\sigma \left(z\right)=max(0,z)$ and $\theta =(W_1,b_1,W_2,b_2)$ with:

• $W_1\in {\mathbb{R}}^{h\times d}$ (input to hidden weights)
• $b_1\in {\mathbb{R}}^h$ (hidden biases)
• $W_2\in {\mathbb{R}}^{1\times h}$ (hidden to output weights)
• $b_2\in \mathbb{R}$ (output bias)

The training objective is:

$$R\left(\theta \right)={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\displaystyle \frac{1}{2}}{(f(x_i;\theta )-y_i)}^2+{\displaystyle \frac{\lambda }{2}}{\parallel \theta \parallel }_F^2$$

(9)

4.2. Step 1: Learnability Analysis

Proposition 1

(ReLU Network Learnability). The ReLU network training problem is learnable via gradient descent with step size $0<\eta <2/L$ where L is the Lipschitz constant of $\nabla R$.

Proof. 

The gradient descent operator $T\left(\theta \right)=\theta -\eta \nabla R\left(\theta \right)$ can be written as $T=(1-\alpha )I+\alpha S$ where $\alpha =\eta \lambda$ (using the strong convexity from regularization) and $S\left(\theta \right)=\theta -{\displaystyle \frac{\eta }{\alpha }}\nabla R\left(\theta \right)$ is nonexpansive for $\eta <2/L$ by standard results [2]. □

4.3. Step 2: Non-Global Streamability

Proposition 2

(ReLU Network Non-Global Streamability). The ReLU network training problem is not globally streamable for small fixed rank K across the entire parameter space.

Formal Lower Bound.

Consider two parameter configurations ${\theta }^{\left(1\right)},{\theta }^{\left(2\right)}\in \mathsf{\Theta }$ that activate disjoint sets of hidden units for the training data. Specifically, let $A^{\left(1\right)}=\{j:W_1^{\left(1\right)}{x}_i+b_1^{\left(1\right)}\left[j\right]>0\mathrm{for}\mathrm{some}i\}$ and $A^{\left(2\right)}=\{j:W_1^{\left(2\right)}{x}_i+b_1^{\left(2\right)}\left[j\right]>0\mathrm{for}\mathrm{some}i\}$ with $A^{\left(1\right)}\cap A^{\left(2\right)}=\varnothing$.

The gradients $\nabla R\left({\theta }^{\left(1\right)}\right)$ and $\nabla R\left({\theta }^{\left(2\right)}\right)$ have support on orthogonal subspaces of the parameter space. Any rank-K approximation ${\sum }_{k=1}^K{g}_k\left(\theta \right)\otimes h_k{\left(\theta \right)}^{*}$ must satisfy:

$$max\left\{\parallel \nabla R\left({\theta }^{\left(1\right)}\right)-\sum _{k=1}^K{g}_k\left({\theta }^{\left(1\right)}\right)\otimes h_k{\left({\theta }^{\left(1\right)}\right)}^{*}{\parallel }_F,{\parallel \nabla R\left({\theta }^{\left(2\right)}\right)-\sum _{k=1}^K{g}_k\left({\theta }^{\left(2\right)}\right)\otimes h_k{\left({\theta }^{\left(2\right)}\right)}^{*}\parallel }_F\right\}\ge c$$

(10)

for some constant $c>0$ depending on the problem structure. □

4.4. Step 3: Streamable Approximation Construction

Construction 2

(ReLU Streamable Approximation). Apply Algorithm 1 to construct a streamable approximation:

Step 1: Sample parameter space Θ with δ-net $\{{\theta }_1,\dots ,{\theta }_N\}$.

Step 2: Compute residuals $r_i=\eta \nabla R\left({\theta }_i\right)$ for each sample point.

Step 3: Form tensor $\mathcal{R}$ and compute rank-K CP decomposition with uniform slice error control.

Step 4: Construct convex surrogate with separable structure:

$$R_{\epsilon }\left(\theta \right)={\displaystyle \frac{1}{2}}\sum _{k=1}^K{\beta }_k{\langle g_k,\theta \rangle }_F^2+{\displaystyle \frac{1}{2}}\sum _{k=1}^K{\gamma }_k{\langle h_k,\theta \rangle }_F^2+{\displaystyle \frac{\lambda }{2}}{\parallel \theta \parallel }_F^2$$

(11)

Step 5: Define streamable update via gradient descent on $R_{\epsilon }$:

$${\theta }_{t+1}={\theta }_t-\eta \nabla R_{\epsilon }\left({\theta }_t\right)$$

(12)

5. Numerical Validation and ULRA Testing

We provide comprehensive numerical experiments validating the theoretical results on both synthetic and real datasets.

5.1. Experimental Setup

Synthetic Dataset: Regression with $n=1000$ samples, $d=100$ features, $h=50$ hidden units.

Real Dataset: MNIST digit classification using a subset of 5000 training samples, two-layer ReLU network with $h=100$ hidden units, 10-class softmax output.

Training: Gradient descent with learning rate $\eta =0.01$, regularization $\lambda =0.001$.

Hardware: Intel i7-10700K, 32GB RAM, implementation in Python 3.9 with NumPy 1.21.0.

5.2. ULRA Test Results

We apply the ULRA-test from the original hierarchy paper [1] to verify streamability before and after conversion:

Table 1. ULRA Test Results: Residual Spectrum Analysis.

Method	Effective Rank	Spectral Decay	ULRA Score
Original ReLU Problem	847	Slow (${\sigma }_k\sim k^{-0.3}$)	0.12 (Non-streamable)
Streamable Approximation (K=20)	20	Fast (${\sigma }_k\sim k^{-1.8}$)	0.89 (Streamable)
Streamable Approximation (K=50)	50	Fast (${\sigma }_k\sim k^{-1.6}$)	0.94 (Streamable)

Table 1. ULRA Test Results: Residual Spectrum Analysis.

Method	Effective Rank	Spectral Decay	ULRA Score
Original ReLU Problem	847	Slow (${\sigma }_k\sim k^{-0.3}$)	0.12 (Non-streamable)
Streamable Approximation (K=20)	20	Fast (${\sigma }_k\sim k^{-1.8}$)	0.89 (Streamable)
Streamable Approximation (K=50)	50	Fast (${\sigma }_k\sim k^{-1.6}$)	0.94 (Streamable)

5.3. Computational Performance Results

Key Observations:

• Density Validation: For any desired approximation error $\epsilon$, we can choose rank K to achieve $d_R(T_0,T_{\epsilon })<\epsilon$.
• Real Dataset Validation: MNIST experiments confirm theoretical predictions with computational reductions of 2.5× to 12.2×.
• ULRA Confirmation: Converted problems show dramatically improved spectral properties confirming streamability.

Table 2. Numerical Validation Results.

Dataset	Rank K	Approx Error	Final Loss	Memory Reduction	Time Reduction
Synthetic	Full	0.000	0.0234	1×	1×
	K = 20	0.031	0.0241	6.2×	5.8×
	K = 10	0.067	0.0256	12.5×	11.9×
	K = 5	0.145	0.0298	25.0×	23.8×
MNIST	Full	0.000	0.1847	1×	1×
	K = 50	0.018	0.1851	2.5×	2.3×
	K = 20	0.042	0.1863	6.1×	5.7×
	K = 10	0.089	0.1891	12.2×	11.5×

Table 2. Numerical Validation Results.

Dataset	Rank K	Approx Error	Final Loss	Memory Reduction	Time Reduction
Synthetic	Full	0.000	0.0234	1×	1×
	K = 20	0.031	0.0241	6.2×	5.8×
	K = 10	0.067	0.0256	12.5×	11.9×
	K = 5	0.145	0.0298	25.0×	23.8×
MNIST	Full	0.000	0.1847	1×	1×
	K = 50	0.018	0.1851	2.5×	2.3×
	K = 20	0.042	0.1863	6.1×	5.7×
	K = 10	0.089	0.1891	12.2×	11.5×

5.4. Conversion Algorithm Complexity

The computational cost of Algorithm 1 is:

• Sampling: $\mathcal{O}\left(N\right)$ where $N=\mathcal{O}\left({\epsilon }^{-3mn}\right)$ for $\delta$-net construction
• CP Decomposition: $\mathcal{O}(K·\mathrm{iter}·mnN)$ using alternating least squares
• Extension: $\mathcal{O}\left(N^2\right)$ for Shepard interpolation setup

Amortization Analysis: The one-time conversion cost $\mathcal{O}(K·\mathrm{iter}·mnN)$ is amortized once training exceeds $\mathcal{O}\left(K^{2}mn\right)$ iterations. For typical problems with $K=20$, $m=n=100$, this occurs after approximately 4,000 training iterations, making the conversion cost negligible for long training runs.

6. Theoretical Extensions

6.1. Convergence Preservation

Proposition 3

(Convergence Preservation for $\alpha$-Operators). Let $T_0$ be an α-averaged operator for a learnable problem and $T_{\epsilon }$ a streamable approximation with $d_R(T_0,T_{\epsilon })\le \epsilon$. If $T_0$ converges to fixed point ${\theta }^{*}$, then $T_{\epsilon }$ converges to an $\mathcal{O}\left(\epsilon \right)$-neighborhood of ${\theta }^{*}$.

Proof. 

Since $T_0$ is $\alpha$-averaged, it satisfies $\parallel T_0\left(\theta \right)-{\theta }^{*}{\parallel }_F\le (1-\alpha ){\parallel \theta -{\theta }^{*}\parallel }_F$ for some $\alpha >0$.

For the streamable approximation:

$$\begin{array}{cc}\hfill \parallel T_{\epsilon }\left(\theta \right)-{\theta }^{*}{\parallel }_F& \le \parallel T_{\epsilon }\left(\theta \right)-T_0{\left(\theta \right)\parallel }_F+{\parallel T_0\left(\theta \right)-{\theta }^{*}\parallel }_F\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill & \le \epsilon +(1-\alpha )\parallel \theta -{\theta }^{*}{\parallel }_F\hfill \end{array}$$

(14)

This shows convergence to an $\mathcal{O}\left(\epsilon \right)$-neighborhood of ${\theta }^{*}$. □

6.2. Empirical Conjecture: Rank-Accuracy Trade-off

Conjecture 3

(Rank-Accuracy Trade-off). For problems with rapidly decaying CP spectrum, the minimum rank $K\left(\epsilon \right)$ required for ε-approximation satisfies:

$$K\left(\epsilon \right)\le Clog(1/\epsilon )·{\mathrm{rank}}_{\epsilon /4}\left(\mathcal{R}\right)$$

(15)

where ${\mathrm{rank}}_{\epsilon /4}\left(\mathcal{R}\right)$ is the $(\epsilon /4)$-rank of the residual tensor.

Remark 2.

This conjecture is empirical, not theoretical—supported by our experiments on synthetic and MNIST data, but lacking a general proof. Unlike matrix SVD, general CP decomposition lacks worst-case guarantees, making this an active area of research [3]. The inequality (15) should be interpreted as an empirical observation rather than a proven bound.

7. Discussion and Limitations

The density result resolves the fundamental question about the scope of streamable optimization while revealing important practical considerations.

7.1. Practical Implications

• Algorithm Design: Any learnable optimization algorithm can be converted to a streamable variant with controllable approximation error.
• Computational Efficiency: Significant reductions in memory and computation are possible, validated by 2.5× to 25× speedups in our experiments.
• Scalability: Large-scale problems can benefit from streamable methods even if not naturally streamable.

7.2. Limitations and Future Work

• Rank Growth: Some problems may require large rank $K\left(\epsilon \right)$, potentially erasing computational benefits.
• Conversion Complexity: The tensor decomposition step scales as $\mathcal{O}(K·\mathrm{iter}·mnN)$ and may be expensive for large problems.
• CP Decomposition Challenges: Unlike SVD, CP decomposition lacks guaranteed global optimality and may require multiple random initializations.

8. Conclusions

We have established that streamable problems are dense within learnable problems under the $\alpha$-operator framework, resolving the question of whether computational restrictions of streamable optimization significantly limit problem scope. The constructive proof provides a systematic method for converting any learnable problem to a streamable approximation, demonstrated through rigorous analysis of ReLU network training and validated numerically with ULRA testing on both synthetic and real datasets.

Our experiments show computational reductions ranging from 2.5× to 25× with controllable accuracy loss, confirming the practical value of the theoretical result. This provides theoretical foundation for the broader adoption of streamable optimization methods, with the assurance that computational efficiency can be achieved without fundamental limitations on the problems that can be solved, though practical considerations around rank growth and conversion costs remain important.