The Structural Chasm: Why No Positional Encoding Can Bridge the SSM–Transformer Gap

1. Introduction

State space models (SSMs) such as Mamba [1] and Transformers are the two dominant paradigms for sequence modeling, yet they embody fundamentally different inductive biases: SSMs maintain a compressed recurrent state that is updated sequentially, while Transformers employ attention to perform global, pairwise comparisons across the context. Both achieve strong empirical performance on overlapping tasks, raising a natural question: when is each architecture fundamentally better?

Existing theoretical work approaches this question from two angles. The algebraic perspective studies what each architecture can compute, showing that diagonal SSMs can only simulate commutative (Abelian) automata [2,3], while Transformers handle broader classes including non-commutative state tracking. The computational perspective studies efficiency-expressivity tradeoffs for hybrid models [4]. Neither perspective, however, addresses the most practically relevant question: given a finite dataset from a structured sequential process, which architecture is a better statistical estimator—i.e., which extracts more information per observation?

We answer this question through the lens of Bayesian decision theory. Our framework studies sequential estimation in Linear Gaussian State Space Models (LG-SSMs), motivated by in-context learning (ICL): a model receives a sequence of observations and must predict the next one, without explicit knowledge of the underlying system parameters. This setting provides a rigorous testbed for comparing the statistical efficiency of different architectures.

Figure 1. Bidirectional Statistical Separation. (Left) The Transformer/ERM paradigm pools observations into an empirical loss, ignoring temporal structure. (Right) The selective SSM paradigm updates a belief state via adaptive filtering. Theorem 2 shows the right paradigm achieves strictly lower risk for temporally correlated tasks; Theorem 3 shows the left paradigm is superior for static estimation.

Figure 1. Bidirectional Statistical Separation. (Left) The Transformer/ERM paradigm pools observations into an empirical loss, ignoring temporal structure. (Right) The selective SSM paradigm updates a belief state via adaptive filtering. Theorem 2 shows the right paradigm achieves strictly lower risk for temporally correlated tasks; Theorem 3 shows the left paradigm is superior for static estimation.

Our contributions are fivefold:

(1) Structural chasm theorem. We prove that the two architectures inhabit fundamentally different computational paradigms—stateful recursion versus pairwise comparison—and that no positional encoding scheme can bridge the gap in the constant-parameter regime (Theorem 1). We provide a concrete non-commutative product tracking task where a 4.7K-parameter SSM outperforms a 50.7K-parameter Transformer at long sequences, confirming that the gap is structural, not merely parametric.

(2) Forward separation (SSM > Transformer). We prove that meta-trained selective SSMs achieve Bayes-optimal sequential prediction by learning Kalman filtering (Lemma 2, Theorem 2). For tasks with AR(1) correlated observation noise, any permutation-invariant predictor—including attention-based Transformers—requires at least $1/(1-{\alpha }^2)$ times as many context tokens to achieve the same excess risk. We formalize this through a Permutation Invariance Impossibility result (Proposition 1): the Kalman filter is inherently order-dependent, so no permutation-invariant predictor can implement it.

(3) Reverse separation (Transformer > SSM). We prove a complementary statefulness curse (Theorem 3): for static parameter estimation with i.i.d. observations, any fixed-gain SSM produces exponentially weighted averages whose variance converges to a positive constant, while the uniform sample mean (achievable by Transformers) achieves optimal $O(1/k)$ decay. The efficiency gap grows without bound with context length.

(4) Non-commutativity as root cause. We prove that in the scalar case ($d=1$), the distinction between the SSM and Transformer innovations forms vanishes due to commutativity, and no separation arises (Theorem 4). The separation requires $d\ge 2$, where matrix non-commutativity is the algebraic mechanism enabling both the statistical separation and the structural chasm.

(5) Phase transition. The SSM’s advantage is non-monotone in the noise correlation $\rho$: it collapses near the observability boundary $\rho \to a$ due to condition number blow-up (Corollary 1).

Together, these results give a complete, mechanistic account of when, why, and how fundamentally each architecture is the better estimator. A detailed discussion of related work is provided in Appendix A.

2. Problem Formulation and Preliminaries

2.1. Meta-Learning over Latent Dynamical Systems

An agent encounters tasks drawn from a distribution $\pi \left(\tau \right)$. Each task $\tau$ is a Linear Gaussian State Space Model (LG-SSM) [5,6]:

$$\begin{array}{cc}\hfill {\mathbf{z}}_t& =A_{\tau }{\mathbf{z}}_{t-1}+{\mathbf{w}}_t,{\mathbf{w}}_t\sim \mathcal{N}(0,Q_{\tau }),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\mathbf{x}}_t& =C_{\tau }{\mathbf{z}}_t+{\mathbf{v}}_t,{\mathbf{v}}_t\sim \mathcal{N}(0,R_{\tau }),\hfill \end{array}$$

(2)

with latent state ${\mathbf{z}}_t\in {\mathbb{R}}^d$ and observation ${\mathbf{x}}_t\in {\mathbb{R}}^m$. Task parameters ${\theta }_{\tau }=(A_{\tau },C_{\tau },Q_{\tau },R_{\tau })$ are unknown to the agent. Given a context ${\mathcal{C}}_k=({\mathbf{x}}_1,\dots ,{\mathbf{x}}_k)$, the agent predicts ${\mathbf{x}}_{k+1}$. The Bayes risk of a predictor f is:

$$R_k\left(f\right)={\mathbb{E}}_{\tau ,{\mathcal{C}}_k,{\mathbf{x}}_{k+1}}{\parallel f\left({\mathcal{C}}_k\right)-{\mathbf{x}}_{k+1}\parallel }^2.$$

(3)

The Bayes-optimal predictor is the posterior predictive mean:

$$f_k^{*}\left({\mathcal{C}}_k\right)=\mathbb{E}[{\mathbf{x}}_{k+1}\mid {\mathcal{C}}_k]=\int \mathbb{E}[{\mathbf{x}}_{k+1}\mid {\mathcal{C}}_k,\theta ]p(\theta \mid {\mathcal{C}}_k)d\theta .$$

(4)

2.2. Architecture Definitions

2.2.0.1. Selective SSM.

A selective SSM maintains a hidden state $h_t\in {\mathbb{R}}^n$ updated as:

$$h_t={\overline{A}}_t{h}_{t-1}+{\overline{B}}_t{x}_t,$$

(5)

where ${\overline{A}}_t={\mathcal{N}}_A(\sigma (U{x}_t+b);\varphi )$ and ${\overline{B}}_t={\mathcal{N}}_B(\sigma (U{x}_t+b);\varphi )$ are input-dependent matrices generated by selective networks.

Definition 1 

(Permutation-Invariant Predictor Class). A predictor f ispermutation-invariantif $f(x_{\sigma \left(1\right)},\dots ,x_{\sigma \left(k\right)})=f(x_1,\dots ,x_k)$ for all permutations $\sigma \in S_k$. The class ${\mathcal{P}}_{\mathrm{inv}}$ includes: (i) the sample mean and regularized variants; (ii) attention-based predictors with symmetric softmax weights and no positional encoding; (iii) any predictor whose sufficient statistic is an order-independent function of the context.

Definition 2 

(Asymptotic Relative Efficiency). For two estimators $f,g$ with excess risks both $\Theta (1/k)$, define:

$$\mathrm{ARE}(f,g)=\underset{k\to \infty }{lim}{\displaystyle \frac{R_k\left(g\right)-R_k\left(f^{*}\right)}{R_k\left(f\right)-R_k\left(f^{*}\right)}}.$$

(6)

When $\mathrm{ARE}(f,g)>1$, estimator g requires $\mathrm{ARE}$ times more context than f to achieve the same excess risk.

2.3. Computational Paradigms

The selective SSM and the Transformer embody two fundamentally different computational paradigms. We formalize this distinction:

Definition 3 

(Pairwise Comparison Paradigm). A model belongs to thepairwise comparisonparadigm if each layer decomposes into:

1.

Apairwise interaction step: $z_t={\sum }_{i=1}^{t}w(t,i)·V\left(x_i\right)$, where weights $w(t,i)$ are determined by the similarity of $x_t$ and $x_i$ (possibly modulated by positional encoding).

2.

Apointwise transformation step: $y_t=\mathrm{MLP}(z_t,x_t)$ with residual connections.

The key property is that the pairwise interaction step hasno hidden-state bottleneck: it can attend to all positions simultaneously with effective state dimension $O(L·d_{\mathrm{model}})$ for a sequence of length L.

Definition 4 

(Stateful Recursion Paradigm). A model belongs to thestateful recursionparadigm if its computation follows:

$$h_t=f(h_{t-1},x_t),y_t=g\left(h_t\right),$$

(7)

where $h_t\in {\mathbb{R}}^d$ is a fixed-dimension compressed state. The key property is thestate bottleneck: the effective state dimension is $O\left(d_{\mathrm{state}}\right)$, independent of context length L. Information about past inputs is mediated entirely through $h_t$.

Multi-layer Transformers with residual connections and MLPs fall under Definition 3. While a D-layer Transformer can approximate D sequential computation steps through its depth, this requires the parameter count to grow with the required depth—placing it outside the constant-parameter regime.

2.4. Assumptions

Assumption A1 

(Prior Support and Stability). $\pi \left(\theta \right)$ has full support on a compact Θ, with $\rho \left(A_{\theta }\right)<1$ for all $\theta \in \Theta$.

Assumption A2 

(Uniform Observability and Controllability). The observability matrix ${\mathcal{O}}_d=[C^{\top },{\left(CA\right)}^{\top },\dots ,$${\left(C{A}^{d-1}\right)}^{\top }{]}^{\top }$ and the controllability Gramian ${\sum }_{j=0}^{d-1}{A}^{j}Q{\left(A^{\top }\right)}^j$ are full rank, uniformly over Θ.

Assumption A3 

(Sufficient Expressivity). ${\mathcal{F}}_{\Phi }$ is a universal approximator for continuous functions ${\mathbb{R}}^{km}\to {\mathbb{R}}^m$ on compact domains.

Assumption A4 

(Meta-Training Convergence). Meta-training converges to a global minimizer of the population meta-risk as $N\to \infty$.

2.5. Meta-Training Objective

Both architectures are meta-trained on next-token prediction: minimize ${\widehat{R}}_N\left(\varphi \right)={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\mathbb{E}}_{({\mathcal{C}}_k,x_{k+1})\sim {\tau }_i}$

$[\parallel f_{\varphi }\left({\mathcal{C}}_k\right)-x_{k+1}{\parallel }^2]$ over tasks ${\left\{{\tau }_i\right\}}_{i=1}^N\sim \pi$.

3. The Structural Chasm

Section 2 defines two computational paradigms—stateful recursion and pairwise comparison—that differ at the level of computational topology. A natural question arises: are these paradigms merely different parameterizations of the same capability, or is the gap between them irreducible? We prove the latter: in the constant-parameter regime, no positional encoding scheme can transform pairwise comparison into stateful recursion.

3.1. Non-Commutative Product Tracking

Definition 5 

(Non-Commutative Product Tracking). Given a sequence of $2\times 2$ invertible matrices $(M_1,\dots ,M_k)$ with $M_t\in \mathrm{GL}\left(2\right)$, the task is to predict the running cumulative product at each step:

$$P_t=M_t·M_{t-1}\dots M_1,t=1,\dots ,k.$$

(8)

This task is inherently non-commutative ($\mathrm{GL}\left(2\right)$ is a non-abelian group: $AB\ne BA$ in general) and requires maintaining a running state. A selective SSM can solve it by setting the transition matrix to $A_t=I\otimes M_t$ (the $4\times 4$ Kronecker product), giving $h_t=A_t{h}_{t-1}=\mathrm{vec}\left(M_t{P}_{t-1}\right)$.

Lemma 1 

(SSM Solvability). A selective SSM with state dimension $n=4$ and input-dependent transition $A_t=I\otimes M_t$ computes the exact cumulative product in $O\left(k\right)$ time. No input-dependent term $B_t$ is needed; the state update is purely multiplicative.

Remark 1 

(On Tautological Advantage). The product tracking task is, in a precise sense, the canonical task for which SSMs were designed: maintaining a running product of sequentially observed matrices. The Structural Chasm Theorem should be interpreted as precisely characterizingwhich class of tasks—those requiring sequential composition of non-commuting operations—is the natural domain of SSMs, and showing that this class is structurally distinct from what Transformers can handle in the constant-parameter regime.

3.2. The Structural Chasm Theorem

Theorem 1 

(Structural Chasm). Let ${\mathcal{T}}_{L,P}$ denote any Transformer with fixed depth L and fixed parameter budget P, equipped with an arbitrary positional encoding scheme (including RoPE, ALiBi, learned, or any group-theoretic variant). Then there exists a family of tasks—non-commutative matrix product tracking (Definition 5)—such that:

1.

Computational topology:${\mathcal{T}}_{L,P}$ has an all-pairs (or chain-without-compression) information flow graph, while a selective SSM has a strictly chain-with-compression graph.

2.

Information bottleneck:The effective state of ${\mathcal{T}}_{L,P}$ grows as $O(L·d_{\mathrm{model}})$, while the SSM maintains $O\left(d_{\mathrm{state}}\right)$.

3.

Constant-parameter incompatibility:The excess risk of ${\mathcal{T}}_{L,P}$ on this task has a positive lower bound $c(d,L,P)>0$ that does not vanish with context length, while a selective SSM with state dimension $n=O\left(d\right)$ achieves excess risk approaching zero as $k\to \infty$.

The proof (Appendix E) relies on a circuit-depth argument: a Transformer with L layers can compose at most $2^L$ matrices by tree reduction, so for sequence length $k>2^L$, it cannot exactly track the cumulative product of k non-commuting matrices. In contrast, the SSM composes sequentially in $O\left(k\right)$ steps with $O\left(d\right)$ state.

3.3. Why Position Encoding Cannot Help

Recent group-theoretic analysis [2,7] shows that all “reasonable” positional encodings take the form $e^{tG}$ for some generator matrix G. Our key observation is:

Positional encodings are static position markers; they inject information aboutwhereeach token is, but do not implement the dynamic state update required for sequential composition.

Architecture	Encoding Type	Role	Coupling Depth
RoPE Transformer	Rotational ($e^{i\theta t}$)	Inject into Q/K	Shallow
ALiBi Transformer	Polynomial (t)	Attention bias	Medium
Selective SSM	Exponential ($e^{-\lambda t}$)	Is the recurrence$A_t$	Deep (inseparable)

Mamba’s “exponential decay” is not a position encoding applied to attention weights—it is the recurrent transition matrix $A_t$ itself. This is the fundamental reason the two paradigms cannot be made equivalent: modifying how much to attend (PE) is categorically different from modifying what state to maintain (recurrence).

4. Forward Separation: SSM Advantage

The Structural Chasm (Section 3) establishes an irreducible paradigm gap on the non-commutative product tracking task. We now show that the same paradigm gap manifests in a classical statistical setting—Bayesian sequential estimation—where the SSM’s advantage can be precisely quantified.

4.1. Selective SSMs as Adaptive Filters

Lemma 2 

(Filter Representation). Consider a selective SSM layer with update (5). For any LG-SSM with known parameters $\theta =(A,C,Q,R)$, there exists a parameterization such that $h_t={\widehat{\mathbf{z}}}_{t|t}\left(\theta \right)=\mathbb{E}[{\mathbf{z}}_t\mid {\mathcal{C}}_t,\theta ]$. This is achieved by setting:

$${\overline{A}}_t=(I_d-K_t\left(\theta \right)C)A,{\overline{B}}_t=K_t\left(\theta \right),$$

(9)

where $K_t\left(\theta \right)=P_{t|t-1}{C}^{\top }{(C{P}_{t|t-1}{C}^{\top }+R)}^{-1}$ is the Kalman gain.

The key insight is that the Kalman predict-then-update recursion rearranges into the innovations form ${\widehat{\mathbf{z}}}_{t|t}=(I_d-K_{t}C)A{\widehat{\mathbf{z}}}_{t-1|t-1}+K_t{\mathbf{x}}_t$, matching (5) exactly. The gain $K_t$ converges exponentially to a steady-state $K_{\infty }$ via the Riccati equation. See Appendix B.1 for the full proof.

4.2. Forward Statistical Separation

Theorem 2 

(Forward Statistical Separation). Under Assumptions A1–A4, consider the task family with AR(1) correlated observation noise $v_t=\alpha v_{t-1}+\sqrt{1-{\alpha }^2}{ϵ}_t$, $\alpha \in (0,1)\setminus \left\{a\right\}$, state dynamics $z_t=a{z}_{t-1}+w_t$, and observation $x_t=z_t+v_t$. Let $f_{{\widehat{\varphi }}_N}$ be the meta-trained selective SSM.

(i) Bayes-Optimality.  As $N,k\to \infty$, for a.e. task $\tau \sim \pi$:

$$f_{{\widehat{\varphi }}_N}\left({\mathcal{C}}_k\right)\stackrel{P}{\to }{f}_k^{*}\left({\mathcal{C}}_k\right),R_k\left(f_{{\widehat{\varphi }}_N}\right)-R_k\left(f^{*}\right)=O\left(k^{-1}\right).$$

The implied constant achieves the Bayesian Cramér-Rao Lower Bound.

(ii) Separation from ${\mathcal{P}}_{\mathrm{inv}}$.  For any $g\in {\mathcal{P}}_{\mathrm{inv}}$:

$$\mathrm{ARE}(f_{{\widehat{\varphi }}_N},g)\ge {\displaystyle \frac{1}{1-{\alpha }^2}}.$$

(10)

Equivalently, g requires $\ge 1/(1-{\alpha }^2)$ times more context tokens to match the SSM’s excess risk. At $\alpha =0.90$, the gap is $\approx 5.3\times$; at $\alpha =0.95$, $\approx 10.3\times$; at $\alpha =0.99$, $>50\times$. As $\alpha \to 1$, the gap diverges.

The proof of Part (i) combines meta-training convergence with Local Asymptotic Normality [8]. Part (ii) uses state augmentation—defining ${\xi }_t={[z_t;v_t]}^{\top }$ yields a 2D LG-SSM—and applies misspecified model theory [9] to show that permutation-invariant predictors retain at most a $(1-{\alpha }^2)$ fraction of the available Fisher information. See Appendix B.2.

4.3. Permutation Invariance Impossibility

Proposition 1 

(Order-Dependence of Optimal Filtering). For any LG-SSM with time-varying Kalman gains $(K_1\ne K_2)$ and nontrivial dynamics ($A\ne I$, $C\ne 0$), the Kalman filter estimate ${\widehat{\mathbf{z}}}_{k|k}$ isnotpermutation-invariant. Consequently, no $f\in {\mathcal{P}}_{\mathrm{inv}}$ can implement the Kalman filter for all input sequences.

A concrete counterexample ($A=2,C=1,K_1=1/3,K_2=2/3$) yields ${\widehat{z}}_{2|2}(0,6)=4\ne 4/3={\widehat{z}}_{2|2}(6,0)$. The general argument follows from non-commutativity of the matrix products ${\prod }_t\left[(I-K_{t}C)A\right]$; see Appendix B.3. Note that softmax attention with positional encodings breaks strict permutation invariance, but its global pairwise comparison does not naturally implement the sequential recursive structure of the Kalman filter.

5. Reverse Separation: Transformer Advantage

The structural chasm establishes an SSM advantage on sequential tasks. For tasks without temporal structure, the picture reverses: the SSM’s recurrent bias becomes a liability.

5.1. Static Estimation Tasks

Definition 6 

(Task Family B: Static Estimation). The latent “state” θ is constant: $z_t=\theta$ for all t. Observations are $x_i=\theta +{ϵ}_i$ with ${ϵ}_i\stackrel{\mathrm{iid}}{\sim }\mathcal{N}(0,{\sigma }^2)$. The task distribution π places a prior on θ.

This is the simplest estimation task: no dynamics, no temporal structure. The optimal predictor is the posterior mean, which for diffuse priors approaches the sample mean ${\overline{x}}_k={\displaystyle \frac{1}{k}}{\sum }_{i=1}^k{x}_i$.

5.2. Theorem 2: The Statefulness Curse

Theorem 3 

(Reverse Separation—Statefulness Curse). Consider Task Family B (Definition 6). Let $g\in {\mathcal{P}}_{\mathrm{inv}}$ be the uniform sample mean ${\overline{x}}_k={\displaystyle \frac{1}{k}}{\sum }_{i=1}^k{x}_i$. Let $f_{\mathrm{SSM}}$ be a selective SSM with any fixed recurrent gain $\gamma \in (0,1)$:

$$h_t=(1-\gamma )h_{t-1}+\gamma x_t.$$

(11)

(i)The uniform mean achieves optimal estimation: $\mathrm{Var}\left({\overline{x}}_k\right)={\sigma }^2/k$.

(ii)The fixed-gain SSM produces an exponentially weighted average:

$$h_k=\gamma {\displaystyle \sum _{i=1}^k}{(1-\gamma )}^{k-i}{x}_i.$$

Its asymptotic variance is ${lim}_{k\to \infty }\mathrm{Var}\left(h_k\right)={\displaystyle \frac{{\sigma }^2\gamma }{2-\gamma }}>0$. The variance doesnotdecay with k.

(iii)The MSE ratio grows without bound:

$${\displaystyle \frac{\mathrm{MSE}\left(h_k\right)}{\mathrm{MSE}\left({\overline{x}}_k\right)}}={\displaystyle \frac{k\gamma }{2-\gamma }}\to \infty \mathrm{as}k\to \infty .$$

(12)

The SSM’s performance gap grows without bound with context length.

The proof is a direct variance calculation: the fixed-gain SSM’s variance converges to ${\sigma }^2\gamma /(2-\gamma )>0$, while the uniform mean achieves ${\sigma }^2/k\to 0$. Consequently, the SSM requires asymptotically $\sim k\gamma /(2-\gamma )$ times more observations than the uniform mean to achieve the same MSE. See Appendix B.4. An adaptive SSM (e.g., Mamba) can in principle escape this curse by learning gains ${\gamma }_t=O(1/t)$, but this means abandoning its recurrent bias to emulate a permutation-invariant predictor—the separation is about inductive bias, not representational capacity.

6. Non-Commutativity as the Root Cause

The structural chasm and the bidirectional separation share a common algebraic root. We trace it to a single property: the non-commutativity of matrix multiplication.

6.1. Theorem 3: Scalar Degeneracy and Matrix Separation

Theorem 4 

(Non-Commutativity as Root Cause). Consider the Kalman filter innovations form ${\widehat{\mathbf{z}}}_{t|t}=(I-K_{t}C)A{\widehat{\mathbf{z}}}_{t-1|t-1}+K_t{\mathbf{x}}_t$.

(i) Scalar case ($d=m=1$).The matrix $(I-K_{t}C)A$ reduces to $(1-k_{t}c)a=a-k_{t}ca$. This identity:

$$(1-kc)a=a-kca\forall a,k,c\in \mathbb{R}$$

(13)

holds universally. The distinction between different orderings of prediction and correction vanishes. No separation arises from the innovations form in the scalar case.

(ii) Matrix case ($d\ge 2$).In general, $(I_d-KC)A\ne A-KC$. The discrepancy is:

$$(I_d-KC)A-(A-KC)=KC(I_d-A),$$

(14)

which is nonzero whenever $A\ne I_d$, $K\ne 0$, and $C\ne 0$. The sequential ordering of operations—predict via A, then correct via $(I-KC)$—is essential and cannot be reproduced by any order-independent computation.

Part (i) is immediate from scalar commutativity. Part (ii) follows from expanding $(I_d-KC)A=A-KCA\ne A-KC$; the discrepancy $KC(I_d-A)$ is nonzero whenever $A\ne I_d$. A concrete counterexample ($a=2,k=3,c=5$) yields $-28\ne -13$. See Appendix B.5.

The correlated-noise task in Theorem 2 uses a scalar latent state $z_t$, but the augmented state ${\xi }_t={[z_t;v_t]}^{\top }$ is 2-dimensional. This augmentation creates the non-commutativity enabling the separation.

6.2. Phase Transition in the Advantage Gap

Corollary 1 

(Phase Transition at Observability Boundary). For the correlated-noise task family of Theorem 2 with $a=0.9$ and $\rho \in (a,1)$, theeffectiveadvantage of the SSM is modulated by the observability condition number:

$${\mathrm{ARE}}_{\mathrm{eff}}\left(\rho \right)\approx {\displaystyle \frac{{(\rho -a)}^2}{1-{\rho }^2}},$$

(15)

which vanishes near the observability boundary (${\mathrm{ARE}}_{\mathrm{eff}}\to 0$ as $\rho \to a^{+}$) and grows as the noise becomes more correlated (${\mathrm{ARE}}_{\mathrm{eff}}\to \infty$ as $\rho \to 1^{-}$).

The raw ARE $1/(1-{\rho }^2)$ is degraded near the observability boundary $\rho \approx a$ by $cond{\left(\mathcal{O}\right)}^2\sim 1/{(\rho -a)}^2$; see Appendix B.6.

7. Unified Picture and Architectural Implications

The structural chasm and the bidirectional separation together yield four implications. First, task-dependent architecture selection: the SSM’s advantage is tied to non-commutativity of the underlying dynamics, providing a quantitative criterion—tasks with latent temporal structure and $d_{\mathrm{eff}}\ge 2$ favor SSMs, while exchangeable observations favor Transformers. Second, the structural chasm is irreducible: no positional encoding scheme can transform pairwise comparison into stateful recursion; the paradigms differ at the level of computational topology, not just parameterization. Third, hybrid architectures interleaving SSM and attention layers are formally justified by the complementary nature of the two separations—each paradigm solves a different class of sub-problem (validated empirically in Appendix D). Fourth, non-commutativity as a design principle: tasks whose optimal estimator involves products of non-commuting matrices are precisely those where SSMs have a structural advantage that Transformers cannot close.

8. Experiments

We validate each theoretical result with a dedicated experiment. Experiments I–IV use the synthetic LG-SSM framework matching the theory; Experiment V ablates the permutation-invariance assumption; Experiment VI tests generalization beyond the Gaussian setting. Experiment VII directly tests the Structural Chasm Theorem on a non-commutative product tracking task. Experiment VIII sweeps positional encoding types to confirm that no PE scheme can bridge the structural chasm. A hybrid-architecture experiment is reported in Appendix D. Full hyperparameters and training details are in Appendix C.

8.1. Experiment I: Forward Separation

We meta-train a selective SSM (370 params), a linear Transformer without PE (3.2K params, permutation-invariant), and a softmax Transformer with learned PE (19.6K params) on AR(1) correlated-noise tasks with $\alpha \sim \mathcal{U}(0.5,0.99)$ and evaluate at $\alpha \in \{0.5,0.7,0.9,0.95,0.99\}$, $k\in \{8,\dots ,512\}$. An oracle Kalman filter serves as baseline. See Appendix C for details.

Results.

Figure 2 and Table 1 confirm Theorem 2. The SSM achieves near-oracle excess risk (excess $\le 0.01$ at $k=512$). The permutation-invariant LinTF’s empirical ARE exceeds the bound $1/(1-{\alpha }^2)$ for $\alpha \ge 0.90$ (the value at $\alpha =0.70$ is numerically unstable as both models approach oracle-level performance). The SoftTF with PE partially closes the gap (ARE $=4.5$ at $\alpha =0.95$, below the PI bound of $10.3$), showing that positional encoding breaks the symmetry constraint but cannot fully replicate the sequential state update that AR(1) dynamics require.

8.2. Experiment II: Reverse Separation

We evaluate on Task Family B (Definition 6): $\theta \sim \mathcal{N}(0,1)$, $x_i=\theta +{ϵ}_i$ i.i.d., with a fixed-gain SSM ($\gamma =0.1$), a selective SSM, a linear Transformer, and the oracle sample mean. Theorem 3 predicts the fixed-gain MSE plateaus at $\gamma /(2-\gamma )\approx 0.053$.

Results.

Figure 3 confirms Theorem 3: the fixed-gain MSE plateaus at $0.053$, matching $\gamma /(2-\gamma )$. The LinTF tracks the oracle $1/k$ rate. The selective SSM partially escapes the curse but plateaus at $\sim 0.015$ for $k\ge 128$, confirming inherent recurrent overhead on exchangeable observations—the mirror image of Experiment I.

8.3. Experiment III: Phase Transition

We fix $a=0.9$ and sweep $\rho$ over 50 values in $(0.85,0.999)$ at $k=256$, measuring the empirical ${\mathrm{ARE}}_{\mathrm{eff}}\left(\rho \right)$ between the selective SSM and linear Transformer.

Results.

Figure 4 shows ARE values dropping near the observability boundary $\rho \approx a$ and rising as $\rho \to 1$, consistent with the qualitative prediction of Corollary 1. The empirical ARE ranges from ∼30 to ∼360 after removing numerically unstable points (ARE $>{10}^6$). The advantage vanishes when both architectures can solve the task ($\rho \approx a$) and grows when sequential state accumulation dominates ($\rho \to 1$), consistent with the structural chasm of Section 3.

8.4. Experiment IV: Non-Commutativity Ablation

We compare Case (a) scalar ($d_{\mathrm{eff}}=1$, i.i.d. noise) versus Case (b) matrix ($d_{\mathrm{eff}}=2$, AR(1) noise $\rho =0.95$), testing selective SSM, LinTF (no PE), and SoftTF (+PE) at $k=256$.

Results.

Table 2 shows that the LinTF (no PE) has high ARE in both cases, reflecting inability to exploit temporal order. The SoftTF (+PE) reveals the key contrast: in Case (b), its ARE of $13.4$ exceeds the theoretical bound of $10.3$, while in Case (a) ($d_{\mathrm{eff}}=1$) the SSM achieves oracle-level risk. The jump between cases isolates non-commutativity as the mechanism (Theorem 4): when the order of composition matters, pairwise comparison cannot replicate the sequential update.

8.5. Experiment V: Positional Encoding Ablation

We test whether positional encodings close the forward-separation gap by comparing four Transformer variants at $\alpha =0.95$: (a) linear attention, no PE; (b) softmax, no PE; (c) softmax + sinusoidal PE; (d) softmax + learned PE. All ∼4K params.

Results.

Figure 5(left) confirms the predicted hierarchy. PI variants (a, b) suffer high excess risk (0.38 and 0.52 at $k=512$). Sinusoidal PE (c) reduces it to 0.022, and learned PE (d) to $<0.001$, nearly matching the SSM. PE partially closes the gap, but the residual gap for (c) confirms that order awareness alone is insufficient; the recursive predict-then-correct structure matters. On this scalar task, learned PE can compensate because the required sequential update is low-dimensional; Experiment VIII shows this compensation fails on inherently non-commutative tasks.

8.6. Experiment VI: Beyond LG-SSM — Character-Level HMM

To test generalization beyond the LG-SSM setting, we evaluate on next-character prediction in HMMs with 20 hidden states and 50 characters. We meta-train a small Mamba (∼5K params) and a small Transformer (∼22K params) across three transition sparsity levels (${\alpha }_{\mathrm{trans}}\in \{0.01,0.1,1.0\}$). See Appendix C for details.

Results.

Figure 5(right) provides preliminary evidence that the forward separation generalizes beyond LG-SSMs. For sparse HMMs (${\alpha }_{\mathrm{trans}}=0.01$), Mamba outperforms the Transformer by up to $+0.024$ accuracy; the gap shrinks for less structured HMMs, consistent with the prediction that the SSM advantage depends on temporal correlation strength. Sparse transitions create strong sequential dependencies that favor stateful recursion; dense transitions approach exchangeability.

8.7. Experiment VII: Non-Commutative Product Tracking

We test the Structural Chasm Theorem (Theorem 1) directly. The task is non-commutative matrix product tracking (Definition 5): given a sequence of $2\times 2$ matrices $M_t\in \mathrm{GL}\left(2\right)$, predict the running product $P_t=M_t\dots M_1$ at each step. We compare a selective SSM with Kronecker-structured transition ($A_t=I\otimes M_t$, 4.7K params), a general SSM with unstructured transition (6.1K), a linear Transformer (12.9K), a softmax Transformer + RoPE (12.9K), 2-layer and 4-layer Transformers + RoPE (25.5K and 50.7K), and a hybrid model (20.5K). All models are trained at $k=64$ and evaluated at $k\in \{8,16,32,64,128,256\}$; extrapolation is tested at $k=128,256,512$.

Results.

Table 3 and Figure 6 confirm the structural chasm. At the training length $k=64$, TF-4L achieves the lowest MSE (0.195) using 10× more parameters. However, at longer sequences the picture reverses: the SSM (4.7K) achieves MSE 0.594 at $k=256$, outperforming TF-4L (1.44, 50.7K). In extrapolation ($k=512$), the SSM maintains MSE 0.808 while TF-4L degrades to 1.60. The linear Transformer collapses entirely at $k=256$ (MSE $>10$), and the hybrid model also degrades, suggesting the attention component does not aid—and may hinder—sequential state tracking. These results confirm that the SSM’s architectural advantage is structural: its input-dependent transition matrix directly implements the sequential composition that Transformers can only approximate through depth.

8.8. Experiment VIII: Positional Encoding Cannot Bridge the Chasm

Experiment VII shows that the SSM’s advantage grows with sequence length, but leaves open the question: does the choice of positional encoding matter? The Structural Chasm Theorem predicts that no PE scheme can bridge the gap in the constant-parameter regime. We test this by sweeping PE types (none, sinusoidal, RoPE) on the same non-commutative product tracking task, evaluating at $k=64$ and $k=256$ to verify that the SSM advantage grows with k regardless of PE. All 1-layer Transformers have $\sim 3.4$K parameters (comparable to the SSM’s 4.7K); we also test 2-layer Transformers with $\sim 25$K params ($5\times$ the SSM budget) to probe the boundary of the constant-parameter regime.

Results.

Table 4 confirms the structural chasm. At comparable parameter budgets, the SSM outperforms all 1-layer Transformers regardless of PE type, with ratios ranging from 1.38 to 1.77 at $k=64$. Crucially, the ratio increases with sequence length: TF-1L (no PE) goes from 1.77 at $k=64$ to 2.78 at $k=256$, and TF-1L + RoPE from 1.64 to 2.64. This is consistent with Theorem 1, which predicts that the Transformer’s excess risk has a positive lower bound as $k\to \infty$. The PE type has minimal effect: the gap remains large whether the Transformer has no PE, sinusoidal PE, or RoPE, confirming that the barrier is not about lacking position information but about lacking the stateful recursion paradigm.

At $5\times$ the SSM parameter budget, 2-layer Transformers become competitive at the training length $k=64$ (TF-2L + RoPE achieves ratio 0.89), but this advantage erodes at $k=256$ (ratio 1.15). This is again consistent with the theorem: more parameters can reduce the constant $c(d,L,P)$ at fixed k, but cannot make it vanish as $k\to \infty$. Experiment VII confirms this more strongly: even a 4-layer Transformer with $10\times$ more parameters (50.7K) loses to the SSM at $k\ge 256$.

9. Discussion

Limitations.

Our theoretical results are for the LG-SSM setting; Experiment VI provides preliminary evidence of generalization to HMMs, but formal extension to nonlinear dynamics remains open. We prove separation in the population limit $N,k\to \infty$; finite-sample rates need characterization. The Structural Chasm Theorem is stated for the constant-parameter regime; in practice, increasing Transformer depth can partially close the gap on fixed-length tasks, at the cost of parameter count scaling with sequence length.

Conclusion.

We established a bidirectional statistical separation—SSMs excel when temporal structure creates non-commutative dynamics, Transformers excel when observations are exchangeable—and proved that this separation reflects a deeper structural chasm: the two architectures inhabit fundamentally different computational paradigms (stateful recursion vs. pairwise comparison) that no positional encoding scheme can bridge in the constant-parameter regime. Matrix non-commutativity is the algebraic root cause and the advantage is graded by the observability condition number. Empirically, a 4.7K-parameter SSM outperforms a 50.7K-parameter Transformer at long sequences, and the excess risk ratio increases with sequence length regardless of PE type—from 1.77 at $k=64$ to 2.78 at $k=256$ for a parameter-matched Transformer. These results shift the question from “what can each model compute?” to “which paradigm is the right inductive bias for this task?”—and suggest that hybrid architectures, combining both paradigms, are not a compromise but a necessity.

In-Context Learning (ICL).

The phenomenon of ICL—where pretrained models learn new tasks from examples in the prompt—was first systematically studied for simple function classes by Garg et al. [10]. Von Oswald et al. [11] showed that Transformers implement gradient descent in their forward pass, and Zhang et al. [12] proved that trained linear Transformers learn linear models in-context. Subsequent work has characterized ICL for nonlinear regression [13,14,15], low-dimensional target functions [16], modular arithmetic [17], and linear classification [18]. Linear Transformers were shown to be versatile in-context learners [19], and Giannou et al. [20] studied how Transformers emulate Newton’s method. A Bayesian perspective on ICL has been developed by several groups: Wies et al. [21] analyzed learnability, Jeon et al. [22] provided an information-theoretic analysis, Wakayama and Suzuki [23] gave generalization guarantees via hierarchical Bayes, and Deora et al. [24] studied how Transformers prefer simpler hypotheses. Chain-of-continuous-thought models have also been analyzed [25]. Our work differs from all of the above by establishing a bidirectional separation between architectures rather than analyzing a single architecture’s ICL capability.

SSMs and In-Context Learning.

Sushma et al. [26] demonstrated that SSMs can learn in-context by gradient descent, paralleling the Transformer result of Von Oswald et al. [11]. Oh et al. [27] showed that Mamba can learn low-dimensional targets via test-time feature learning. Bondaschi et al. [28] studied how Mamba learns Markov chains in-context. Cole et al. [29] analyzed in-context learning of linear dynamical systems with Transformers. Bick et al. [30] proposed Kalman Linear Attention as a connection between Bayesian filtering and efficient attention. Our Lemma 2 provides a complementary structural result: selective SSMs can exactly represent the Kalman filter, not just approximate it.

Expressiveness and Separation.

The expressive capacity of SSMs on regular languages was studied by Sarrof et al. [31], who showed limitations from a formal language perspective. Terzic et al. [3] proved that diagonal SSMs can only simulate commutative automata, and Shakerinava et al. [2] established stronger limits for state-tracking tasks. Merrill et al. [32] identified an “illusion of state” in SSMs. On the separation side, Wen et al. [33] identified in-context retrieval as a key bottleneck for RNNs, Jelassi et al. [34] showed Transformers are better at copying, and Bick et al. [35] analyzed the skill gap in recurrent LMs. Dao and Gu [36] unified the two paradigms via structured state space duality. Cooper et al. [4] studied efficiency-expressivity tradeoffs in hybrid models. Ebrahimi et al. [37] analyzed induction biases in sequence models. From a statistical perspective, Mousavi-Hosseini et al. [38] studied when Transformers outperform feedforward and recurrent networks, and Haas and Bruna [39] analyzed the statistical advantage of softmax attention. Our work contributes a Bayesian statistical separation grounded in estimation theory, complementing these expressiveness-based results.

Meta-Learning.

The meta-learning framework underlying our setup—where models learn to learn from task distributions—dates to Thrun and Pratt [40]. Grant et al. [41] recast gradient-based meta-learning as hierarchical Bayes. The nonparametric estimation theory we build on follows Györfi et al. [42].

Appendix B.1. Proof of Lemma 2 (Filter Representation)

Appendix B.1.1. Setup

Consider an LG-SSM with known parameters $\theta =(A,C,Q,R)$:

$${\mathbf{z}}_t=A{\mathbf{z}}_{t-1}+{\mathbf{w}}_t,{\mathbf{x}}_t=C{\mathbf{z}}_t+{\mathbf{v}}_t,$$

with ${\mathbf{w}}_t\sim \mathcal{N}(0,Q)$, ${\mathbf{v}}_t\sim \mathcal{N}(0,R)$. The Kalman filter computes ${\widehat{\mathbf{z}}}_{t|t}=\mathbb{E}[{\mathbf{z}}_t\mid {\mathcal{C}}_t]$ via predict-then-update:

$${\widehat{\mathbf{z}}}_{t|t}=A{\widehat{\mathbf{z}}}_{t-1|t-1}+K_t({\mathbf{x}}_t-CA{\widehat{\mathbf{z}}}_{t-1|t-1}),$$

where $K_t=P_{t|t-1}{C}^{\top }{(C{P}_{t|t-1}{C}^{\top }+R)}^{-1}$ is the Kalman gain. Expanding:

$${\widehat{\mathbf{z}}}_{t|t}=(I_d-K_{t}C)A{\widehat{\mathbf{z}}}_{t-1|t-1}+K_t{\mathbf{x}}_t.$$

(A1)

Appendix B.1.2. Selective SSM Parameterization

The selective SSM updates $h_t={\overline{A}}_t{h}_{t-1}+{\overline{B}}_t{x}_t$ with input-dependent matrices:

$${\overline{A}}_t={\mathcal{N}}_A(s_t;\varphi ),{\overline{B}}_t={\mathcal{N}}_B(s_t;\varphi ),s_t=\sigma (U{x}_t+b).$$

We need parameters $\varphi$ such that ${\overline{A}}_t=(I_d-K_t\left(\theta \right)C)A$ and ${\overline{B}}_t=K_t\left(\theta \right)$ for all t.

Appendix B.1.3. Constructive Argument

The gain $K_t$ depends on $\theta$ and the error covariance $P_{t|t-1}$, which evolves via the discrete Riccati equation:

$$\begin{array}{cc}\hfill P_{t|t-1}& =A{P}_{t-1|t-1}{A}^{\top }+Q,\hfill \end{array}$$

(A2)

$$\begin{array}{cc}\hfill P_{t|t}& =(I_d-K_{t}C)P_{t|t-1},\hfill \end{array}$$

(A3)

where the Kalman gain is $K_t=P_{t|t-1}{C}^{\top }{(C{P}_{t|t-1}{C}^{\top }+R)}^{-1}$. Under Assumptions A1–A2 (stability $\rho \left(A\right)<1$ and uniform observability), the sequence $\left\{P_{t|t-1}\right\}$ converges exponentially to the unique positive-definite fixed point $P_{\infty }$ of the Discrete Algebraic Riccati Equation (DARE):

$$P_{\infty }=A(P_{\infty }-P_{\infty }{C}^{\top }{(C{P}_{\infty }{C}^{\top }+R)}^{-1}C{P}_{\infty })A^{\top }+Q.$$

Correspondingly, $K_t\to K_{\infty }=P_{\infty }{C}^{\top }{(C{P}_{\infty }{C}^{\top }+R)}^{-1}$ at rate $\parallel K_t-K_{\infty }\parallel =O\left({\lambda }^t\right)$ for some $\lambda \in (0,1)$.

For a time-invariant system, the transient sequence $\{K_1,K_2,\dots \}$ is deterministic given $\theta$ and the initial covariance $P_{0|0}$.

Define the gain-generating function ${\mathcal{G}}_{\theta }:\mathbb{N}\to {\mathbb{R}}^{d\times m}$ mapping step t to $K_t\left(\theta \right)$. We construct the SSM parameterization in two phases.

Phase 1 (Transient, $t\le T_{\mathrm{transient}}$): The state $h_{t-1}$ encodes both the estimate ${\widehat{\mathbf{z}}}_{t-1|t-1}$ and the step index $t-1$ (the latter can be maintained by a counter in an additional state dimension). The selective networks use the step index to look up the pre-computed gain:

$${\overline{A}}_t=(I_d-K_{t}C)A,{\overline{B}}_t=K_t.$$

Phase 2 (Steady state, $t>T_{\mathrm{transient}}$): Once $\parallel K_t-K_{\infty }\parallel <ϵ$ for desired precision $ϵ$, the gain is fixed:

$${\overline{A}}_t=(I_d-K_{\infty }C)A,{\overline{B}}_t=K_{\infty }.$$

Induction. We verify $h_t={\widehat{\mathbf{z}}}_{t|t}\left(\theta \right)$ for all t by induction on t.

Base case ($t=0$): Both are initialized to the prior mean, $h_0={\widehat{\mathbf{z}}}_{0|0}=\mathbb{E}\left[{\mathbf{z}}_0\right]$.

Inductive step: Assume $h_{t-1}={\widehat{\mathbf{z}}}_{t-1|t-1}$. Then:

$$\begin{array}{cc}\hfill h_t& ={\overline{A}}_t{h}_{t-1}+{\overline{B}}_t{\mathbf{x}}_t\hfill \\ \hfill & =(I_d-K_{t}C)A{\widehat{\mathbf{z}}}_{t-1|t-1}+K_t{\mathbf{x}}_t\hfill \\ \hfill & ={\widehat{\mathbf{z}}}_{t|t}\left(\theta \right),\hfill \end{array}$$

where the last equality follows from (A1). The induction is complete.

Since the gain sequence is bounded ($\parallel K_t\parallel \le \parallel P_{\infty }\parallel \parallel C\parallel \parallel R^{-1}\parallel +O\left({\lambda }^t\right)$) and converges exponentially, Assumption A3 (universal approximation) guarantees that a selective network of sufficient width can approximate ${\mathcal{G}}_{\theta }\left(t\right)$ to arbitrary precision $ϵ>0$. The approximation error at step t satisfies $\parallel h_t-{\widehat{\mathbf{z}}}_{t|t}\parallel \le {\sum }_{s=1}^t{\parallel (I_d-K_{\infty }C)A\parallel }^{t-s}ϵ=O\left(ϵt\right)$ for $\parallel (I_d-K_{\infty }C)A\parallel <1$ (guaranteed by stability), or more precisely $O(ϵ/(1-\parallel (I_d-K_{\infty }C)A\parallel \left)\right)$ in the steady-state regime. □

Appendix B.2. Proof of Theorem 2 (Forward Statistical Separation)

The proof combines three stages for Part (i) and a misspecification argument for Part (ii).

Appendix B.2.1. Part (i): Bayes-Optimality

Appendix B.2.2.1. Stage 1: Meta-training learns the conditional expectation.

The empirical meta-risk is

$${\widehat{R}}_N\left(\varphi \right)={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\mathbb{E}}_{({\mathcal{C}}_k,x_{k+1})\sim {\tau }_i}\left[\parallel f_{\varphi }\left({\mathcal{C}}_k\right)-x_{k+1}{\parallel }^2\right].$$

Denote the population meta-risk $R\left(\varphi \right)={\mathbb{E}}_{\theta \sim \pi }\left[{\mathbb{E}}_{({\mathcal{C}}_k,x_{k+1})\mid \theta }{\parallel f_{\varphi }\left({\mathcal{C}}_k\right)-x_{k+1}\parallel }^2\right]$. A standard bias–variance decomposition gives, for any $\varphi$,

$$\begin{array}{cc}\hfill R\left(\varphi \right)& ={\mathbb{E}}_{\theta }\left[\mathbb{E}\left[\parallel f_{\varphi }\left({\mathcal{C}}_k\right)-\mathbb{E}[x_{k+1}\mid {\mathcal{C}}_k]{\parallel }^2\mid \theta \right]\right]+{\mathbb{E}}_{\theta }\left[\mathrm{Var}(x_{k+1}\mid {\mathcal{C}}_k)\right].\hfill \end{array}$$

The second term is independent of $\varphi$, so the population minimizer is

$$f^{*}\left({\mathcal{C}}_k\right)=\mathbb{E}[x_{k+1}\mid {\mathcal{C}}_k]={\mathbb{E}}_{\theta \mid {\mathcal{C}}_k}\left[CA{\widehat{\mathbf{z}}}_{k|k}\left(\theta \right)\right],$$

the Bayes-optimal one-step predictor. Next we establish ${\widehat{R}}_N\to R$ uniformly over the hypothesis class $\mathcal{F}$. Since $f_{\varphi }$ is bounded on bounded contexts (Assumption A3), the loss $\ell (\varphi ;\tau )=\parallel f_{\varphi }\left({\mathcal{C}}_k\right)-x_{k+1}{\parallel }^2$ is uniformly bounded. By a standard symmetrization argument,

$$\mathbb{E}\left[\underset{\varphi \in \mathcal{F}}{sup}|R\left(\varphi \right)-{\widehat{R}}_N\left(\varphi \right)|\right]\le 2{\mathfrak{R}}_N(\ell \circ \mathcal{F}),$$

where ${\mathfrak{R}}_N$ denotes the Rademacher complexity. For parametric networks with P parameters and Lipschitz activations, ${\mathfrak{R}}_N(\ell \circ \mathcal{F})=O(P^{1/2}/N^{1/2})$ [43]. Hence ${\widehat{R}}_N\left({\widehat{\varphi }}_N\right)\to R\left(f^{*}\right)={inf}_{\varphi }R\left(\varphi \right)$ as $N\to \infty$, and by Assumption A3 (universal approximation) the class $\mathcal{F}$ contains a member $ϵ$-close to $f^{*}$ for any $ϵ>0$. Combining, $f_{{\widehat{\varphi }}_N}\to f^{*}$ in $L^2\left(\pi \right)$.

Appendix B.2.2.2. Stage 2: SSM dynamics as approximate EM.

From Lemma 2, for any fixed $\theta$ the SSM state can represent the Kalman filter estimate ${\widehat{\mathbf{z}}}_{t|t}\left(\theta \right)$ exactly by setting ${\overline{A}}_t=(I_d-K_{t}C)A$ and ${\overline{B}}_t=K_t$. During meta-training, $\theta$ is unknown; the selective networks must simultaneously infer $\theta$ from the growing context ${\mathcal{C}}_k$ and track the latent state. The optimal predictor marginalizes over $\theta$:

$$\mathbb{E}[x_{k+1}\mid {\mathcal{C}}_k]=\int CA{\widehat{\mathbf{z}}}_{k|k}\left(\theta \right)p(\theta \mid {\mathcal{C}}_k)d\theta .$$

We show that the gradient dynamics of the prediction loss naturally decompose into two coupled operations that mirror the Expectation–Maximization (EM) algorithm:

• E-step (state inference): Given the current parameter estimate ${\widehat{\theta }}^{\left(n\right)}$, the SSM recurrence computes $h_t={\overline{A}}_t\left({\widehat{\theta }}^{\left(n\right)}\right)h_{t-1}+{\overline{B}}_t\left({\widehat{\theta }}^{\left(n\right)}\right)x_t$, which by Lemma 2 equals ${\widehat{\mathbf{z}}}_{t|t}\left({\widehat{\theta }}^{\left(n\right)}\right)$. This is the posterior mean of ${\mathbf{z}}_t$ conditioned on ${\mathcal{C}}_t$ under ${\widehat{\theta }}^{\left(n\right)}$, exactly the E-step sufficient statistic.
• M-step (parameter update): The prediction error at step $t+1$ is $e_{t+1}=x_{t+1}-CA{h}_t$. The gradient of $\parallel e_{t+1}{\parallel }^2$ with respect to the selective network parameters $\varphi$ propagates through ${\overline{A}}_t\left(\varphi \right)$ and ${\overline{B}}_t\left(\varphi \right)$:

  $${\displaystyle \frac{\partial \parallel e_{t+1}{\parallel }^2}{\partial \varphi }}=-2e_{t+1}^{\top }CA{\displaystyle \frac{\partial h_t}{\partial \varphi }},$$

  where $\partial h_t/\partial \varphi$ satisfies the sensitivity recursion $\partial h_t/\partial \varphi ={\overline{A}}_t\partial h_{t-1}/\partial \varphi +(\partial {\overline{A}}_t/\partial \varphi )h_{t-1}+(\partial {\overline{B}}_t/\partial \varphi )x_t$. This gradient update adjusts $({\overline{A}}_t,{\overline{B}}_t)$ to reduce prediction error, analogous to the M-step of maximum likelihood estimation for state-space model parameters.

The meta-training objective aggregates these per-step gradients across N tasks, so the learned selective mapping $\varphi \mapsto ({\overline{A}}_t\left(\varphi \right),{\overline{B}}_t\left(\varphi \right))$ converges to the Bayesian posterior-weighted Kalman parameterization as $N\to \infty$.

Stage 3: Local Asymptotic Normality and efficiency.

We establish that the meta-trained SSM achieves asymptotic efficiency via the theory of Local Asymptotic Normality (LAN) [8]. Consider a local perturbation $\theta ={\theta }_0+\delta /\sqrt{k}$ with $\parallel \delta \parallel =O\left(1\right)$. In the LG-SSM setting, the joint log-likelihood of context ${\mathcal{C}}_k=(x_1,\dots ,x_k)$ given $\theta$ is

$${\ell }_k\left(\theta \right)=logp({\mathcal{C}}_k\mid \theta )=-{\displaystyle \frac{1}{2}}{\displaystyle \sum _{t=1}^k}\left[logdet\left(S_t\left(\theta \right)\right)+e_t{\left(\theta \right)}^{\top }{S}_t{\left(\theta \right)}^{-1}{e}_t\left(\theta \right)\right]+\mathrm{const},$$

where $e_t\left(\theta \right)=x_t-CA{\widehat{\mathbf{z}}}_{t-1|t-1}\left(\theta \right)$ is the innovation and $S_t\left(\theta \right)=C{P}_{t|t-1}\left(\theta \right)C^{\top }+R$ is the innovation covariance. The score function at ${\theta }_0$ is

$${\Delta }_k={\displaystyle \frac{1}{\sqrt{k}}}{\displaystyle \frac{\partial {\ell }_k}{\partial \theta }}{|}_{\theta ={\theta }_0}={\displaystyle \frac{1}{\sqrt{k}}}{\displaystyle \sum _{t=1}^k}{\displaystyle \frac{\partial e_t}{\partial \theta }}{|}_{{\theta }_0}^{\top }{S}_t^{-1}{e}_t\left({\theta }_0\right).$$

Under regularity conditions (stability of A, full observability of $(A,C)$, positive definite $Q,R$), the LAN expansion holds:

$$log{\displaystyle \frac{p({\mathcal{C}}_k\mid {\theta }_0+\delta /\sqrt{k})}{p({\mathcal{C}}_k\mid {\theta }_0)}}={\delta }^{\top }{\Delta }_k-{\displaystyle \frac{1}{2}}{\delta }^{\top }I\left({\theta }_0\right)\delta +o_P\left(1\right),$$

where ${\Delta }_k\stackrel{d}{\to }\mathcal{N}(0,I\left({\theta }_0\right))$ and the Fisher information matrix is

$$I\left({\theta }_0\right)=\underset{k\to \infty }{lim}{\displaystyle \frac{1}{k}}{\displaystyle \sum _{t=1}^k}{\displaystyle \frac{\partial e_t}{\partial \theta }}{|}_{{\theta }_0}^{\top }{S}_t^{-1}{\displaystyle \frac{\partial e_t}{\partial \theta }}{|}_{{\theta }_0}.$$

By the Hájek–Le Cam convolution theorem, no regular estimator can achieve MSE below $I{\left({\theta }_0\right)}^{-1}$ (the Bayesian Cramér–Rao bound, BCRB). We now show the SSM attains this bound.

The SSM state $h_k$ is an asymptotically sufficient statistic for $\delta$ in the following sense: by Stages 1–2, the meta-trained SSM learns to compute $h_k\approx {\widehat{\mathbf{z}}}_{k|k}\left(\widehat{\theta }\right)$ where $\widehat{\theta }$ is implicitly adapted from the context. The Kalman filter state ${\widehat{\mathbf{z}}}_{k|k}\left(\theta \right)$ together with $P_{k|k}\left(\theta \right)$ determines the innovations ${\left\{e_t\left(\theta \right)\right\}}_{t=1}^k$, which in turn determine the score ${\Delta }_k$. Therefore $h_k$ retains all information about ${\Delta }_k$, making it asymptotically sufficient. Consequently, the SSM predictor achieves the BCRB: $\mathrm{MSE}\left(f_{{\widehat{\varphi }}_N}\right)\to I{\left({\theta }_0\right)}^{-1}$ as $N,k\to \infty$.

Appendix B.2.2. Part (ii): Separation from P inv

State augmentation.

Define ${\xi }_t={[z_t;v_t]}^{\top }$. The augmented system is:

$${\xi }_t=\underset{A_{\mathrm{aug}}}{\underbrace{\left[\begin{array}{cc}a& 0\\ 0& \alpha \end{array}\right]}}{\xi }_{t-1}+\underset{{\mathbf{w}}_{\mathrm{aug},t}}{\underbrace{\left[\begin{array}{c}{w}_t\\ \sqrt{1-{\alpha }^2}{ϵ}_t\end{array}\right]}},x_t=\underset{C_{\mathrm{aug}}}{\underbrace{\left[11\right]}}{\xi }_t.$$

This is an LG-SSM with $R_{\mathrm{aug}}=0$. The observability matrix $\mathcal{O}=\left[\begin{array}{cc}1& 1\\ a& \alpha \end{array}\right]$ has $det\left(\mathcal{O}\right)=\alpha -a\ne 0$ when $\alpha \ne a$.

ERM lower bound under misspecification.

Any $g\in {\mathcal{P}}_{\mathrm{inv}}$ treats observations as exchangeable. This implies that g models the observation noise $v_t$ as i.i.d., misspecifying the true AR(1) correlation structure $v_t=\alpha v_{t-1}+\sqrt{1-{\alpha }^2}{ϵ}_t$.

Under the true model, the observations have autocovariance $\mathrm{Cov}(x_s,x_t)={\sigma }_z^2{a}^{|s-t|}+{\sigma }_v^2{\alpha }^{|s-t|}$ (for $s\ne t$). A permutation-invariant estimator ignores the off-diagonal structure and uses only the marginal variance $\mathrm{Var}\left(x_t\right)={\sigma }_z^2/(1-a^2)+{\sigma }_v^2/(1-{\alpha }^2)$.

The true Fisher information for estimating $z_{k+1}$ from ${\mathcal{C}}_k$ with full temporal structure is $I_{\mathrm{true}}\left(\theta \right)=C_{\mathrm{aug}}^{\top }{S}_{\infty }^{-1}{C}_{\mathrm{aug}}$, where $S_{\infty }$ is the steady-state innovation covariance. The misspecified Fisher information (under the i.i.d. assumption) is, by White [9]’s sandwich formula,

$$I_{\mathrm{mis}}\left(\theta \right)=B{\left(\theta \right)}^{\top }A{\left(\theta \right)}^{-1}B\left(\theta \right),$$

where $A\left(\theta \right)$ is the expected outer product of the misspecified score and $B\left(\theta \right)$ is the expected Hessian of the misspecified likelihood. Since the misspecified model ignores temporal correlations of order $\alpha$, the effective information satisfies

$$I_{\mathrm{mis}}\le (1-{\alpha }^2)I_{\mathrm{true}}.$$

By the Cramér–Rao inequality under misspecification, the variance of any permutation-invariant estimator is bounded below:

$$R_k^{\mathrm{ERM}}\left(\alpha \right)\ge {\displaystyle \frac{1}{k{I}_{\mathrm{mis}}}}\ge {\displaystyle \frac{1}{k(1-{\alpha }^2)I_{\mathrm{true}}}}={\displaystyle \frac{c_2\left(\alpha \right)}{k}},c_2\left(\alpha \right)=\Omega \left({\displaystyle \frac{1}{1-{\alpha }^2}}\right).$$

SSM upper bound.

By Part (i), the meta-trained SSM approximates the Kalman filter on the augmented system $({\xi }_t,x_t)$. The Kalman filter with true parameters achieves the BCRB:

$${\mathrm{MSE}}_{\mathrm{KF}}\left(k\right)=\mathrm{tr}\left(P_{k|k}\right)\to \mathrm{tr}\left(P_{\infty }\right),$$

where $P_{\infty }$ solves the DARE for the augmented system. For prediction of $x_{k+1}$, the Kalman filter achieves

$$R_k^{\mathrm{KF}}\left(\alpha \right)=C_{\mathrm{aug}}{P}_{\infty }{C}_{\mathrm{aug}}^{\top }+\underset{=0}{\underbrace{R_{\mathrm{aug}}}}=\mathrm{const}\left(\alpha \right).$$

After averaging over the prior on $\theta$ and accounting for the $O(1/k)$ learning rate for $\theta$ (from Stage 3), the SSM’s risk satisfies

$$R_k^{\mathrm{SSM}}\left(\alpha \right)={\displaystyle \frac{c_1\left(\alpha \right)}{k}}+o(1/k),c_1\left(\alpha \right)=\mathrm{tr}\left(I{\left({\theta }_0\right)}^{-1}\right)=O\left(1\right)\mathrm{uniformly}\mathrm{in}\alpha ,$$

since $I\left({\theta }_0\right)$ remains bounded away from zero for $\alpha \in (0,1)$ with $\alpha \ne a$ (by the observability condition $det\left(\mathcal{O}\right)=\alpha -a\ne 0$).

The asymptotic relative efficiency is

$$\mathrm{ARE}\left(\alpha \right)={\displaystyle \frac{c_2\left(\alpha \right)}{c_1\left(\alpha \right)}}\ge {\displaystyle \frac{1}{1-{\alpha }^2}}\to \infty \mathrm{as}\alpha \to 1.\square$$

Appendix B.3. Proof of Proposition 1 (Permutation Invariance Impossibility)

Appendix B.3.1. Concrete Counterexample

Let $A=2$, $C=1$, with time-varying gains $K_1=1/3$, $K_2=2/3$. The Kalman recursion is ${\widehat{z}}_{t|t}=(1-K_t)·2·{\widehat{z}}_{t-1|t-1}+K_t{x}_t$, starting from ${\widehat{z}}_{0|0}=0$.

Order $(x_1,x_2)=(0,6)$:

$$\begin{array}{cc}\hfill {\widehat{z}}_{1|1}& =(1-1/3)·2·0+(1/3)·0=0,\hfill \\ \hfill {\widehat{z}}_{2|2}& =(1-2/3)·2·0+(2/3)·6=4.\hfill \end{array}$$

Order $(x_1,x_2)=(6,0)$:

$$\begin{array}{cc}\hfill {\widehat{z}}_{1|1}& =(1-1/3)·2·0+(1/3)·6=2,\hfill \\ \hfill {\widehat{z}}_{2|2}& =(1-2/3)·2·2+(2/3)·0=4/3.\hfill \end{array}$$

Since $4\ne 4/3$, the Kalman filter output depends on observation order.

Appendix B.3.2. General Argument

We derive the condition under which the Kalman filter is permutation-invariant and show it fails generically when $d\ge 2$.

Step 1: Unrolling the recursion. The Kalman filter innovations form (Lemma 2) gives the recursion

$${\widehat{\mathbf{z}}}_{t|t}=(I_d-K_{t}C)A{\widehat{\mathbf{z}}}_{t-1|t-1}+K_t{x}_t.$$

Define the propagation matrix ${\Phi }_t(I_d-K_{t}C)A$. Unrolling for k steps with ${\widehat{\mathbf{z}}}_{0|0}=\mathbf{0}$:

$$\begin{array}{cc}\hfill {\widehat{\mathbf{z}}}_{k|k}& ={\displaystyle \sum _{t=1}^k}\left({\displaystyle \prod _{s=t+1}^k}{\Phi }_s\right)K_t{x}_t={\displaystyle \sum _{t=1}^k}{\Phi }_{k:t+1}{K}_t{x}_t,\hfill \end{array}$$

where ${\Phi }_{k:t+1}={\Phi }_k{\Phi }_{k-1}\dots {\Phi }_{t+1}$ (empty product equals $I_d$ when $t=k$).

Step 2: Permutation-invariance condition. The estimator ${\widehat{\mathbf{z}}}_{k|k}$ is permutation-invariant in $(x_1,\dots ,x_k)$ if and only if the coefficient of $x_t$ is the same for all $t\in \{1,\dots ,k\}$:

$${\Phi }_{k:t+1}{K}_t={\Phi }_{k:s+1}{K}_s\forall t,s\in \{1,\dots ,k\}.$$

In particular, for $k=2$, this requires

$${\Phi }_2{K}_1=K_2⟺(I_d-K_{2}C)A{K}_1=K_2.$$

(A4)

Step 3: Generic failure for $d\ge 2$. The gains $K_1,K_2$ are determined by the Riccati recursion (equations (17)–()) and satisfy $K_t=P_{t|t-1}{C}^{\top }{(C{P}_{t|t-1}{C}^{\top }+R)}^{-1}$. For generic $(A,C,Q,R)$ with $d\ge 2$, we have $K_1\ne K_2$ (the Riccati recursion has not converged in one step), and $A\ne I_d$ (nontrivial dynamics). Substituting into (A2):

$$(I_d-K_{2}C)A{K}_1-K_2=A{K}_1-K_{2}CA{K}_1-K_2.$$

This is a $d\times m$ matrix equation. For $d\ge 2$, the left-hand side depends on products $A{K}_1$ and $K_{2}CA{K}_1$ involving non-commuting matrices. The set of parameters $(A,C,Q,R)$ satisfying (A2) has measure zero in the parameter space ${\mathbb{R}}^{d^2}\times {\mathbb{R}}^{m\times d}\times S_{+}^d\times S_{+}^m$ (where $S_{+}^d$ denotes the positive definite cone), since (A2) imposes $d·m$ algebraic constraints on $d^2+md+d(d+1)/2+m(m+1)/2$ free parameters. Therefore, for generic system parameters, the Kalman filter is not permutation-invariant. □

Appendix B.4. Proof of Theorem 3 (Reverse Separation—Statefulness Curse)

Appendix B.4.1. Setup

Task Family B: $x_i=\theta +{ϵ}_i$, ${ϵ}_i\stackrel{\mathrm{iid}}{\sim }\mathcal{N}(0,{\sigma }^2)$, $\theta$ random.

Appendix B.4.2. Uniform Mean

${\overline{x}}_k={\displaystyle \frac{1}{k}}{\sum }_{i=1}^k{x}_i=\theta +{\displaystyle \frac{1}{k}}{\sum }_{i=1}^k{ϵ}_i$. Since ${ϵ}_i$ are i.i.d.:

$$\mathrm{Var}({\overline{x}}_k-\theta )={\displaystyle \frac{{\sigma }^2}{k}}.$$

Appendix B.4.3. Fixed-Gain SSM

The recursion $h_t=(1-\gamma )h_{t-1}+\gamma x_t$ with $h_0=0$ unrolls to:

$$h_k=\gamma {\displaystyle \sum _{i=1}^k}{(1-\gamma )}^{k-i}{x}_i=\gamma {\displaystyle \sum _{i=1}^k}{(1-\gamma )}^{k-i}(\theta +{ϵ}_i).$$

The bias term: $\mathbb{E}\left[h_k\right]=\theta ·\gamma {\sum }_{i=1}^k{(1-\gamma )}^{k-i}=\theta [1-{(1-\gamma )}^k]\to \theta$.

The variance:

$$\begin{array}{cc}\hfill \mathrm{Var}(h_k-\theta )& =\mathrm{Var}\left(\gamma {\displaystyle \sum _{i=1}^k}{(1-\gamma )}^{k-i}{ϵ}_i-\theta {(1-\gamma )}^k\right)\hfill \\ \hfill & ={\sigma }^2{\gamma }^2{\displaystyle \sum _{i=1}^k}{(1-\gamma )}^{2(k-i)}+\mathrm{Var}\left(\theta \right){(1-\gamma )}^{2k}.\hfill \end{array}$$

The geometric sum: ${\sum }_{i=1}^k{(1-\gamma )}^{2(k-i)}={\displaystyle \frac{1-{(1-\gamma )}^{2k}}{1-{(1-\gamma )}^2}}={\displaystyle \frac{1-{(1-\gamma )}^{2k}}{\gamma (2-\gamma )}}$.

As $k\to \infty$:

$$\mathrm{Var}(h_k-\theta )\to {\displaystyle \frac{{\sigma }^2\gamma }{2-\gamma }}>0.$$

The MSE of $h_k$ as an estimator of $\theta$ converges to a positive constant, while $\mathrm{Var}({\overline{x}}_k-\theta )={\sigma }^2/k\to 0$. The ratio $\mathrm{MSE}\left(h_k\right)/\mathrm{MSE}\left({\overline{x}}_k\right)\to k\gamma /(2-\gamma )\to \infty$. □

Appendix B.5. Proof of Theorem 4 (Non-Commutativity Root Cause)

Appendix B.5.1. Part (i): Scalar Case

For $d=m=1$, all quantities are scalars. The Kalman innovations form gives:

$${\widehat{z}}_{t|t}=(1-k_{t}c)·a·{\widehat{z}}_{t-1|t-1}+k_t{x}_t.$$

Expanding: $(1-k_{t}c)a=a-k_{t}ca$. This equals $a-k_{t}ca$ trivially. The “alternative form” $a-k_{t}c$ differs from $(1-k_{t}c)a$ by $k_{t}c(a-1)$, which is nonzero when $a\ne 1$. However, the correct innovations form in the scalar case always gives $(1-k_{t}c)a$, and since scalar multiplication commutes, the ordering of A and $(I-KC)$ is irrelevant:

$$(1-kc)·a=a·(1-kc)\forall a,k,c\in \mathbb{R}.$$

Appendix B.5.2. Part (ii): Matrix Case

For $d\ge 2$, consider $(I_d-KC)A$ vs. $(A-KC)$:

$$\begin{array}{cc}\hfill (I_d-KC)A& =A-KCA,\hfill \\ \hfill (A-KC)& =A-KC.\hfill \end{array}$$

The difference is $KC-KCA=KC(I_d-A)$, which is nonzero when $A\ne I_d$, $K\ne 0$, $C\ne 0$. A concrete counterexample ($a=2,k=3,c=5$) yields $-28\ne -13$. □

Appendix B.6. Proof of Corollary 1 (Phase Transition)

Appendix B.6.1. Observability Analysis

The augmented system has $C_{\mathrm{aug}}=[1,1]$ and $A_{\mathrm{aug}}=diag(a,\rho )$. The observability matrix is:

$$\mathcal{O}=\left[\begin{array}{c}{C}_{\mathrm{aug}}\\ C_{\mathrm{aug}}{A}_{\mathrm{aug}}\end{array}\right]=\left[\begin{array}{cc}1& 1\\ a& \rho \end{array}\right].$$

$det\left(\mathcal{O}\right)=\rho -a$. As $\rho \to a$, $det\left(\mathcal{O}\right)\to 0$ and $cond\left(\mathcal{O}\right)\sim 1/|\rho -a|$.

Appendix B.6.2. Impact on Kalman Filter Convergence

The convergence rate of the Kalman filter’s error covariance $P_{t|t}$ to the steady-state $P_{\infty }$ is governed by the observability Gramian ${\mathcal{O}}^{\top }\mathcal{O}$. We make this precise.

The Riccati recursion for the augmented system is

$$P_{t|t-1}=A_{\mathrm{aug}}{P}_{t-1|t-1}{A}_{\mathrm{aug}}^{\top }+Q_{\mathrm{aug}},P_{t|t}=P_{t|t-1}-P_{t|t-1}{C}_{\mathrm{aug}}^{\top }{\left(C_{\mathrm{aug}}{P}_{t|t-1}{C}_{\mathrm{aug}}^{\top }\right)}^{-1}{C}_{\mathrm{aug}}{P}_{t|t-1}.$$

Define the error ${\Delta }_t=P_{t|t}-P_{\infty }$. By linearizing the Riccati map around $P_{\infty }$, the error satisfies

$${\Delta }_{t+1}\approx F{\Delta }_t{F}^{\top },F=(I_2-K_{\infty }{C}_{\mathrm{aug}})A_{\mathrm{aug}},$$

where $K_{\infty }=P_{\infty }{C}_{\mathrm{aug}}^{\top }{\left(C_{\mathrm{aug}}{P}_{\infty }{C}_{\mathrm{aug}}^{\top }\right)}^{-1}$ is the steady-state gain. The spectral radius of F determines the convergence rate: $\parallel {\Delta }_t{\parallel \le \parallel F\parallel }^{2t}\parallel {\Delta }_0\parallel$. For the system to be detectable (a necessary condition for Kalman filter stability), we need $\rho \left(F\right)<1$, which requires the observability matrix $\mathcal{O}$ to have full column rank.

When $\rho \to a$, the observability Gramian satisfies

$${\mathcal{O}}^{\top }\mathcal{O}=\left[\begin{array}{cc}1+a^2& 1+a\rho \\ 1+a\rho & 1+{\rho }^2\end{array}\right],{\lambda }_{min}\left({\mathcal{O}}^{\top }\mathcal{O}\right)=O\left({(\rho -a)}^2\right).$$

The steady-state gain $K_{\infty }$ scales as $K_{\infty }\propto {\left({\mathcal{O}}^{\top }\mathcal{O}\right)}^{-1}{C}_{\mathrm{aug}}^{\top }$, so $\parallel K_{\infty }\parallel =O(1/|\rho -a\left|\right)$. The spectral radius of F approaches 1:

$$\rho \left(F\right)=1-O\left({(\rho -a)}^2\right),$$

and the number of observations required to reach $ϵ$-close to steady state scales as

$$t_{ϵ}\sim {\displaystyle \frac{log(1/ϵ)}{|log\rho (F\left)\right|}}=O\left({\displaystyle \frac{1}{{(\rho -a)}^2}}\right)=O\left(cond{\left(\mathcal{O}\right)}^2\right).$$

Thus, the transient regime—during which the Kalman filter has not yet converged to its optimal performance—is inflated by a factor proportional to $cond{\left(\mathcal{O}\right)}^2$.

Appendix B.6.3. Effective ARE

The raw ARE from Theorem 2 is $1/(1-{\rho }^2)$, measuring the statistical penalty of misspecification. The effective ARE accounts for the filter’s convergence penalty:

$${\mathrm{ARE}}_{\mathrm{eff}}\left(\rho \right)\approx {\displaystyle \frac{{\mathrm{ARE}}_{\mathrm{raw}}\left(\rho \right)}{cond{\left(\mathcal{O}\right)}^2}}={\displaystyle \frac{1/(1-{\rho }^2)}{1/{(\rho -a)}^2}}={\displaystyle \frac{{(\rho -a)}^2}{1-{\rho }^2}}.$$

Appendix B.6.4. Monotonicity and Finite-Sample Effects

Setting $f\left(\rho \right)={(\rho -a)}^2/(1-{\rho }^2)$ and differentiating:

$$f^{\prime }\left(\rho \right)={\displaystyle \frac{2(\rho -a)(1-{\rho }^2)+2\rho {(\rho -a)}^2}{{(1-{\rho }^2)}^2}}={\displaystyle \frac{2(\rho -a)[1-{\rho }^2+\rho (\rho -a)]}{{(1-{\rho }^2)}^2}}.$$

The numerator factor $1-{\rho }^2+{\rho }^2-\rho a=1-\rho a>0$ for $\rho ,a\in (0,1)$. Since $\rho >a$ in our range, $f^{\prime }\left(\rho \right)>0$: the effective ARE is strictly increasing in $\rho$ for $\rho >a$, vanishing at the observability boundary $\rho =a$ and diverging as $\rho \to 1$. However, the convergence to the asymptotic regime is slower for $\rho \approx a$ (requiring larger k to see the benefit), creating a practical degradation at finite k that mimics non-monotonicity. □

Appendix C.1. Experiment I & III: LG-SSM Task Distribution

Appendix C.7.1. Prior Distribution π(θ)

$A_{\tau }\in {\mathbb{R}}^{4\times 4}$: random orthogonal with eigenvalues in $[0.7,0.95]$. $C_{\tau }\in {\mathbb{R}}^{2\times 4}$: i.i.d. $\mathcal{N}(0,1)$ entries. $Q_{\tau }=V{\Sigma }_Q{V}^{\top }$ with ${\Sigma }_Q=diag(exp\left(u_i\right))$, $u_i\sim \mathcal{U}(log0.1,log1.0)$. $R_{\tau }$ similar with $v_i\sim \mathcal{U}(log0.05,log0.5)$.

Appendix C.7.2. Colored-Noise Task (Experiment I and III)

Signal: $z_t=0.9z_{t-1}+w_t$, $w_t\sim \mathcal{N}(0,1)$. Noise: $v_t=\rho v_{t-1}+\sqrt{1-{\rho }^2}{ϵ}_t$. Observation: $x_t=z_t+v_t$. Meta-training: $\rho \sim \mathcal{U}(0.5,0.99)$, 10,000 steps per epoch.

Appendix C.7.3. Model Architectures

Selective SSM: Hidden dimension $n=16$, selective projection $s_t=\mathrm{SiLU}(U{x}_t+b)$, diagonal ${\overline{A}}_t=tanh(W_A{s}_t+b_A)$, ${\overline{B}}_t=W_B{s}_t+b_B$. 370 parameters.

Linear Transformer: Single layer, embedding $d=32$, linear attention with ELU+1 kernel: $\varphi \left(x\right)=\mathrm{ELU}\left(x\right)+1$, $\mathrm{Attn}(Q,K,V)=\varphi \left(Q\right)\varphi {\left(K\right)}^{\top }V/\left(\varphi \left(Q\right)\varphi {\left(K\right)}^{\top }\mathbf{1}\right)$, causal masking, no positional encoding. 3.2K parameters.

Softmax Transformer (+PE): Single layer, embedding $d=32$, softmax attention with learned positional embedding table of size $512\times d$. 19.6K parameters (primarily from PE table).

Appendix C.7.4. Training

AdamW (${\beta }_1=0.9$, ${\beta }_2=0.999$), LR $3\times {10}^{-4}$ with cosine decay, batch size 128, MSE loss, 10,000 steps. SSM trained on sequence length 128 (recurrent generalisation to longer sequences); Transformers trained on sequence length 512.

Appendix C.7.5. Oracle Computation

For each test $\alpha$, the oracle is the steady-state Kalman filter with true parameters $(a,\alpha )$ on the augmented 2D state space ${\xi }_t=[z_t;v_t]$. The steady-state gain $K_{\infty }$ is computed via the Discrete Algebraic Riccati Equation (DARE). 1,000 test sequences per $(\alpha ,k)$ configuration.

Appendix C.2. Experiment II: Static Estimation Task

$\theta \sim \mathcal{N}(0,1)$, $x_i=\theta +{ϵ}_i$, ${ϵ}_i\sim \mathcal{N}(0,1)$. Context lengths $k\in \{8,16,32,64,128,256,512\}$. Metric: MSE.

Fixed-gain SSM: Uses the same recurrent architecture as the selective SSM but with a fixed, non-adaptive gain $\gamma =0.1$: $h_t=0.9h_{t-1}+0.1x_t$. This directly instantiates the setting of Theorem 3. No meta-training is applied; the gain is set by construction.

Selective SSM and linear Transformer: Same architectures and training procedure as Experiment I, but meta-trained on static estimation tasks.

Appendix C.3. Experiment IV: Non-Commutativity Ablation

Case (a): Scalar LG-SSM ($d=1$) with i.i.d. noise: $z_t=0.9z_{t-1}+w_t$, $x_t=z_t+v_t$, $v_t\sim \mathcal{N}(0,1)$ i.i.d. No augmentation needed.

Case (b): Augmented 2D system with AR(1) noise (as in Experiment I with $\rho =0.95$).

Metric: empirical ARE at $k=256$.

Appendix C.4. Experiment V: Positional Encoding Ablation

All four Transformer variants share the same core architecture (single layer, embedding $d=16$, ∼4K parameters) and differ only in the attention mechanism and positional encoding:

• Variant (a): Linear attention, no PE. Attention weights: $\mathrm{Attn}(Q,K,V)=Q{K}^{\top }V/d$ with causal masking. This is the permutation-invariant baseline (identical to Experiment I Transformer).
• Variant (b): Softmax attention, no PE. $\mathrm{Attn}(Q,K,V)=\mathrm{softmax}(Q{K}^{\top }/\sqrt{d})V$ with causal masking. Softmax introduces nonlinearity but preserves permutation invariance (within the causal mask).
• Variant (c): Softmax attention + sinusoidal PE. Same as (b) but with fixed sinusoidal positional encoding $\mathrm{PE}(t,2i)=sin(t/{10000}^{2i/d})$, $\mathrm{PE}(t,2i+1)=cos(t/{10000}^{2i/d})$, added to input embeddings.
• Variant (d): Softmax attention + learned PE. Same as (b) but with a learnable positional embedding table of size $k_{max}\times d$.

All variants are meta-trained on the colored-noise task with $\alpha =0.95$, using the same training protocol as Experiment I.

Appendix C.5. Experiment VI: Character-Level HMM

HMM with 20 hidden states, 50 observable characters. Transition matrix: rows from $\mathrm{Dirichlet}\left({\alpha }_{\mathrm{trans}}\right)$. Emission matrix: rows from $\mathrm{Dirichlet}(\alpha =0.1)$.

Small Mamba: Embedding $d=32$, one Mamba block (state $n=16$), LayerNorm residual, linear output + softmax. ∼5K params.

Small Transformer: Embedding $d=32$, one attention layer (4 heads, $d_{\mathrm{head}}=8$), FFN $d=64$, learned positional embeddings, LayerNorm. ∼22K params.

Training: 5,000 steps per ${\alpha }_{\mathrm{trans}}$, 128-character sequences, batch 32, AdamW LR ${10}^{-3}$, cosine decay. Evaluation: 500 test sequences across 50 HMM tasks per k, top-1 accuracy.

To vary the degree of temporal structure, we generate three HMM families with different Dirichlet concentration parameters for the transition matrix: ${\alpha }_{\mathrm{trans}}\in \{0.01,0.1,1.0\}$. Lower ${\alpha }_{\mathrm{trans}}$ produces sparser, more predictable transitions (analogous to higher $\alpha$ in the LG-SSM setting).

Appendix C.6. Experiment VII: Non-Commutative Product Tracking

Task. Generate sequences of $2\times 2$ invertible matrices $M_t\in \mathrm{GL}\left(2\right)$ with bounded Frobenius norm ($\le 1.5$). Each matrix is parameterized as a rotation-scaling-shear combination: $M=R\left(\theta \right)·\mathrm{diag}(s_1,s_2)+J$, where $\theta \sim \mathrm{Uniform}(-0.5,0.5)$, $s_i\sim \mathrm{Uniform}(0.8,1.2)$, and J is a shear perturbation. The target is the running cumulative product $P_t=M_t\dots M_1$. All matrices are non-commuting (100% of random pairs satisfy $AB\ne BA$).

Models.

• SSM (Kronecker): 4,760 params. Input-dependent transition $A_t=I\otimes f\left(x_t\right)$, where f maps the 4D input to a $2\times 2$ matrix. The Kronecker product enforces the correct block structure for matrix multiplication. State dimension $n=4$; hidden dimension 64.
• General SSM: 6,104 params. Full $4\times 4$ transition matrix parameterized from input (no Kronecker constraint). Same architecture as Gu and Dao [1] but with non-diagonal transition.
• LinTF (no PE): 12,868 params. ELU+1 kernel, causal masking.
• SoftTF + RoPE: 12,868 params. 4-head softmax attention with RoPE.
• TF-2L/4L + RoPE: 25,508 / 50,660 params. 2/4-layer Transformer with residual connections, MLP, LayerNorm, and RoPE.
• Hybrid: 20,472 params. One SSM block (Kronecker-structured) + one attention block with RoPE.

Training. 8,000 steps, batch size 64, sequence length 64, AdamW (LR $3\times {10}^{-4}$, cosine decay), MSE loss. Evaluation: MSE at $k\in \{8,16,32,64,128,256\}$ (20 test batches of 32 sequences each). Extrapolation: train at $k=64$, test at $k\in \{128,256,512\}$.

Appendix C.7. Experiment VIII: PE Equivalence Sweep

Task. Non-commutative product tracking, same as Experiment VII (Definition 5). This task directly tests the Structural Chasm Theorem, which predicts that the SSM advantage should grow with sequence length k.

Models. We compare a KroneckerSSM baseline (4,760 params) against 1-layer and 2-layer softmax Transformers with PE variants (none, sinusoidal, RoPE). 1-layer models have $\sim 3.4$K params (comparable to the SSM); 2-layer models have $\sim 25$K params ($5\times$ the SSM budget).

Training. 8,000 steps, batch size 64, sequence length 64, AdamW (LR $3\times {10}^{-4}$, cosine decay), MSE loss. Evaluation: MSE and excess risk ratio ${\mathrm{MSE}}_{\mathrm{TF}}/{\mathrm{MSE}}_{\mathrm{SSM}}$ at $k\in \{64,128,256\}$, demonstrating how the ratio changes with sequence length.

Appendix D.1. Mixed-Structure Task

Each sequence of length $T=128$ consists of alternating blocks:

• Filtering blocks (length 32): observations from an LG-SSM with AR(1) noise ($\alpha =0.95$), requiring sequential belief updating.
• Retrieval blocks (length 32): the model must recall a “key” observation from the filtering block (the observation with maximum absolute value) and output it. This requires associative retrieval from unordered memory.

Two filtering blocks and two retrieval blocks alternate within each sequence. The loss is the total MSE across both block types.

Appendix D.2. Models

• Pure SSM: Two stacked Mamba blocks (embedding $d=32$, state $n=16$). ∼4K params.
• Pure Transformer: Two stacked attention layers (4 heads, $d=32$, FFN $d=64$, learned PE). ∼25K params.
• Hybrid: One Mamba block followed by one attention layer (same dimensions). ∼19K params.

All models are meta-trained for 5,000 steps, AdamW LR $3\times {10}^{-4}$, batch size 64.

Appendix D.3. Predicted Outcome

• The pure SSM should excel on filtering blocks but struggle on retrieval blocks (the compressed state discards the specific observation needed for retrieval).
• The pure Transformer should excel on retrieval blocks but underperform on filtering blocks (permutation invariance penalizes sequential estimation).
• The hybrid should achieve the best overall MSE by using the SSM layer for sequential state tracking and the attention layer for associative recall.

Table A1. Hybrid architecture experiment. MSE on filtering vs. retrieval sub-tasks.

	Filtering MSE	Retrieval MSE	Total MSE
Pure SSM	0.578	2.934	1.756
Pure Transformer	0.886	1.280	1.083
Hybrid (SSM + Attn)	0.355	1.953	1.154

Table A1. Hybrid architecture experiment. MSE on filtering vs. retrieval sub-tasks.

	Filtering MSE	Retrieval MSE	Total MSE
Pure SSM	0.578	2.934	1.756
Pure Transformer	0.886	1.280	1.083
Hybrid (SSM + Attn)	0.355	1.953	1.154

Appendix D.4. Results

Table A1 confirms the predicted complementarity. The pure SSM achieves a moderate filtering MSE (0.578) but the worst retrieval MSE (2.934), reflecting the information bottleneck of the compressed recurrent state. The pure Transformer achieves the best retrieval MSE (1.280) and best total MSE (1.083), leveraging its global attention to recall the key value. The hybrid achieves the best filtering MSE (0.355), substantially outperforming both pure architectures on the sequential estimation sub-task. Its retrieval MSE (1.953) is intermediate, as expected for a single attention layer following the SSM. While the pure Transformer achieves the lowest total MSE in this experiment—reflecting the relatively simple retrieval task—the hybrid’s dominance on filtering validates the theoretical prediction that SSM layers provide a unique advantage for sequential belief updating.

Appendix E.1. Claims 1–2: Computational Topology and Information Bottleneck

These follow directly from Definitions 3 and 4.

Claim 1 (Topology). In the pairwise comparison paradigm, the information flow graph at each layer is all-pairs: position t can access information from all positions $i\le t$ in a single step. In the stateful recursion paradigm, the information flow graph is strictly chain-structured: position t accesses information from position $t-1$ only, mediated through the compressed state $h_{t-1}$.

Claim 2 (Bottleneck). A Transformer with $d_{\mathrm{model}}$-dimensional representations and sequence length L maintains an effective state of dimension $O(L·d_{\mathrm{model}})$ (all key-value pairs are accessible). A selective SSM with state dimension $d_{\mathrm{state}}$ maintains an effective state of dimension $O\left(d_{\mathrm{state}}\right)$, independent of L.

Appendix E.2. Claim 3: Constant-Parameter Incompatibility

We prove that for the non-commutative product tracking task (Definition 5), no Transformer with fixed depth L and fixed parameter budget P can achieve vanishing excess risk as $k\to \infty$.

Proof 

(Proof of Claim 3). We use a circuit-depth argument. Consider a Transformer with L layers, each consisting of one attention block and one MLP.

Step 1: Composition capacity of a single layer. Each attention head in layer ℓ can, at best, produce a weighted average of the representations from the previous layer. For the matrix product tracking task, the relevant computation is composing two matrices $M_i$ and $M_j$. A single attention operation can access both $M_i$ (as the query position) and $M_j$ (as a key position), and the MLP can then compute $M_i{M}_j$. Thus, one layer can compose at most one pair of matrices.

Step 2: Composition capacity of L layers. With L layers, the Transformer can perform at most L sequential composition steps. By tree reduction (composing pairs at each level), L layers can compose at most $2^L$ matrices. For a sequence of $k>2^L$ matrices, the Transformer cannot compute the full product $M_k\dots M_1$ in a single forward pass.

Step 3: Error lower bound. For the product tracking task, the excess risk at position k is:

$${\mathcal{E}}_k=\mathbb{E}\left[\parallel {\widehat{P}}_k-P_k{\parallel }_F^2\right],$$

(A5)

where ${\widehat{P}}_k$ is the model’s prediction. If the Transformer cannot compute the full product (Step 2), then ${\widehat{P}}_k$ is a function of at most $2^L$ of the k input matrices. Since the matrices are drawn independently and are non-commutative, the expected error from missing $k-2^L$ matrices is bounded below by a positive constant $c>0$ that depends on the matrix distribution but not on k. Formally, for $k>2^L$:

$${\mathcal{E}}_k\ge c(d,L,P)>0,$$

(A6)

where c does not vanish as $k\to \infty$.

Step 4: SSM achieves vanishing error. A selective SSM with state dimension $n=d^2=4$ and transition $A_t=I\otimes M_t$ computes the exact cumulative product sequentially:

$$h_t=A_t{h}_{t-1}=(I\otimes M_t)·\mathrm{vec}\left(P_{t-1}\right)=\mathrm{vec}\left(M_t{P}_{t-1}\right)=\mathrm{vec}\left(P_t\right).$$

(A7)

The excess risk approaches zero (up to learning the input-to-transition mapping) as the number of training sequences grows.

This completes the proof of Theorem 1. □

Appendix F.19.19.1. The classification describes static markers, not dynamic updates.

The group-theoretic classification describes how position information is injected into the attention computation:

$$w(t,i)=\mathrm{softmax}\left({\displaystyle \frac{{(Q_t+{\mathrm{PE}}_t)}^{\top }(K_i+{\mathrm{PE}}_i)}{\sqrt{d_k}}}\right).$$

(A8)

The PE modulates the attention weights $w(t,i)$, determining how much position t attends to position i. This is fundamentally different from the SSM’s recurrence:

$$h_t=A_t{h}_{t-1}+B_t{x}_t,$$

(A9)

where $A_t$ determines what the new state is, not how much to attend.

Appendix F.19.19.2. The two mechanisms are mathematically distinct.

• PE (Transformer): Modifies $w(t,i)\in \mathbb{R}$, a scalar weight in $[0,1]$. The PE changes how much information flows from position i to position t.
• Transition matrix (SSM):$A_t\in {\mathbb{R}}^{n\times n}$, a full matrix. The transition changes what the state becomes, implementing a linear (or nonlinear, via input-dependence) transformation of the previous state.

Appendix F.19.19.3. Consequence: PE cannot implement state transitions.

No choice of PE scheme—however clever—can make the attention mechanism maintain and update a compressed state. The attention output is always a weighted sum of value vectors; it cannot apply a matrix transformation to a persistent hidden state. This is the mathematical content of the structural chasm.

Appendix F.19.19.4. Reconciliation.

The group-theoretic classification and the structural chasm are complementary, not contradictory:

• The classification describes the variety of position encodings (limited to $e^{tG}$ forms).
• The chasm describes the fundamental limitation of all position encodings (they modify attention weights, not state transitions).
• Together, they give the complete picture: not only are PE forms limited, but even the full space of allowed PE forms cannot bridge the paradigm gap.