PAC-Bayes Generalization Bounds for In-Context Learning: Tight Rates, Fast Convergence, and Architectural Implications

1. Introduction

1.1. The Phenomenon of In-Context Learning

Large language models (LLMs) exhibit a remarkable ability: given a few input–output demonstrations in the prompt, they can make accurate predictions on new queries without updating any parameters [1]. This phenomenon, known as in-context learning (ICL), has profound practical implications and raises fundamental theoretical questions:

• Why does the number of required demonstrations vary dramatically across tasks (1–3 for sentiment analysis vs. 10+ for natural language inference)?
• Does there exist an optimal number of demonstrations, and can it be computed from model properties?
• What is the information-theoretic lower bound on the error—even with arbitrarily many demonstrations?

Despite extensive empirical study, these questions lack rigorous answers. Existing theoretical analyses fall into several categories—Bayesian inference interpretations, gradient descent analogies, algorithm selection frameworks, information-theoretic analyses, and asymptotic characterizations—none of which provides explicit, computable error bounds with sample complexity formulas for a frozen pretrained model at inference time.

1.2. Related Work

Bayesian implicit inference.

[2] model ICL as implicit Bayesian inference over a latent concept variable. While conceptually appealing, this view does not yield quantitative generalization guarantees or sample complexity formulas.

Gradient descent analogy.

[3] and [4] show that Transformer attention layers can implement gradient descent steps. These results provide mechanistic insight but do not translate into error bounds on the population risk.

Algorithm selection and stability.

[5] and [6] analyze ICL as selecting among algorithms learned during pretraining. The resulting guarantees are task-specific and do not yield a unified framework parameterized by computable model–task quantities.

Information-theoretic analysis.

[7] introduce conditional mutual information (e-CMI) tools to bound ICL excess risk without mixing-time assumptions. Their bounds depend on intractable e-CMI quantities rather than computable model properties, and they do not provide closed-form sample complexity, minimax lower bounds, fast rates, or architecture-dependent instantiations.

Bayesian model averaging and pretraining analysis.

[8] show that ICL implicitly implements Bayesian model averaging and establish an $O(1/T)$ regret bound for perfectly pretrained ICL, where T is the prompt length. Their PAC-Bayes analysis targets the pretraining generalization error (as a function of the pretraining corpus size), not the inference-time generalization gap. In contrast, our framework analyzes the gap between population and empirical risk of a frozen model as a function of the number of demonstrations k, without assuming perfect pretraining. Moreover, their work does not provide minimax lower bounds, sample complexity formulas, data-dependent priors, or variance-adaptive bounds.

Asymptotic analysis of linear attention.

[9] derive sharp asymptotic characterizations of the ICL learning curve under linear attention in the proportional limit ($d\to \infty$), revealing phenomena such as double descent. Their results are exact but asymptotic, and they do not provide finite-sample PAC-Bayes bounds, sample complexity formulas, or minimax lower bounds. Our linear attention instantiation (Section 4) provides complementary finite-sample guarantees for the same model class.

Generalization of pretrained Transformers.

[10] prove a generalization bound of $\tilde{O}\left(\sqrt{min\{S,T\}/\left(nT\right)}\right)$ for Transformers trained on linear regression tasks, where S, T, n denote context window, sequence length, and number of tasks respectively. Their analysis focuses on the pretraining phase and the distinction between ICL and in-weight learning, whereas our framework targets the inference-time generalization of a frozen model and provides closed-form, architecture-dependent bounds parameterized by computable quantities.

PAC-Bayes theory.

The PAC-Bayes framework [11,12,13] bounds the generalization error of randomized predictors in terms of the KL divergence between a data-dependent posterior and a data-independent prior. Extensions include fast-rate bounds [13], Bernstein-type refinements [14,15], and data-dependent priors via sample splitting [16]. Our work brings this rich toolkit to bear on inference-time ICL, providing the first unified PAC-Bayes framework parameterized by computable task-model quantities.

1.3. Contributions

Our contributions are organized along six directions. To our knowledge, contributions (1), (3)–(6) are entirely new in the ICL literature; contribution (2) is the first to provide computable, closed-form architecture-dependent bounds for inference-time ICL (cf. Table 1).

• Core PAC-Bayes ICL framework (§Section 3): We establish generalization upper bounds parameterized by two computable quantities—ambiguity A and saliency S—that can be evaluated from the pretrained model without retraining. Unlike the conditional mutual information approach of [7], which yields non-computable bounds, and the Bayesian regret analysis of [8], which assumes a perfectly pretrained model and studies pretraining error, our bounds apply to any frozen pretrained Transformer at inference time with finitely many demonstrations. We further derive closed-form sample complexity, an information-theoretic lower bound, and a permutation variance bound (Theorems 2–6).
• Linear attention instantiation (§Section 4): Under a linear attention model, we derive closed-form expressions for A and S in terms of model dimension, prior variance, noise variance, and signal strength, yielding architecture-dependent bounds (Theorems 1–9). While [9] characterize linear attention ICL in an asymptotic ($d\to \infty$) regime, our bounds are non-asymptotic and yield explicit sample complexity formulas.
• Minimax optimality (§Section 5.1): Using Le Cam’s two-point method and KL tensorization, we prove a matching $\Omega \left(\sqrt{A/k}\right)$ lower bound, establishing that the $\Theta \left(\sqrt{A/k}\right)$ rate is tight (Theorems 10–13). No prior work provides minimax lower bounds for ICL generalization.
• Catoni fast-rate bounds (§Section 5.2): With Catoni’s temperature-parameterized bound, we achieve $O(1/k)$ excess risk when the empirical risk is small, reducing sample complexity from $O(1/{\epsilon }^2)$ to $O(1/\epsilon )$ (Theorems 14–16). This is the first fast-rate PAC-Bayes result in the ICL setting.
• Data-dependent priors (§Section 6.1): Via sample splitting, we construct informed priors that can eliminate the ambiguity term entirely (Theorems 17–19). No existing ICL analysis employs data-dependent priors.
• Bernstein variance-adaptive bounds (§Section 6.2): Exploiting low loss variance, we obtain a second path to $O(1/k)$ fast rates through variance decay (Theorems 20–5). Variance-adaptive analysis for ICL has not appeared in prior work.

1.4. Paper Organization

Section 2 introduces the formal setup and key definitions. Section 3 presents the core framework. Section 4 develops the flagship linear attention instantiation. Section 5 establishes minimax optimality and fast rates. Section 6 presents the data-dependent prior and Bernstein extensions. Section 7 validates predictions via numerical simulations. Section 8 discusses implications, limitations, and open problems. Proofs are sketched in the main text; complete proofs appear in Appendices Appendix A–Appendix D.

2. Problem Setup and Key Definitions

2.1. ICL as a Statistical Learning Problem

Consider a pretrained model $f_{\theta }$ with frozen parameters $\theta$. Given k in-context demonstrations $D_k={\left\{(x_i,y_i)\right\}}_{i=1}^k$ drawn i.i.d. from a task distribution $\mathcal{D}$, the model produces a prediction $f_{\theta }(x_q\mid D_k)$ for a new query $x_q$. We formalize this within the PAC-Bayes framework by identifying:

• Prior P: the model’s zero-shot predictive distribution $f_{\theta }(·\mid ⌀)$;
• Posterior Q: the model’s predictive distribution $f_{\theta }(·\mid D_k)$ after observing k demonstrations.

  Definition 1

(Population and Empirical Risk). For a bounded loss function $\ell :\mathcal{Y}\times \mathcal{Y}\to [0,1]$, define:

$$\begin{array}{cc}\hfill R_{\mathrm{pop}}\left(k\right)& ={\mathbb{E}}_{D_k,(x_q,y_q)\sim \mathcal{D}}\left[\ell \left(f_{\theta }(x_q\mid D_k),y_q\right)\right],\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill R_{\mathrm{emp}}\left(k\right)& ={\displaystyle \frac{1}{k}}{\displaystyle \sum _{i=1}^k}\ell \left(f_{\theta }(x_i\mid D_k),y_i\right),\hfill \end{array}$$

(2)

where both risks are computed with the model conditioned on the full demonstration set $D_k$.

  Remark 1

(Choice of empirical risk). We use the standard plug-in empirical risk (each example evaluated under the full context $D_k$) rather than a leave-one-out estimator. This is the natural form for PAC-Bayes analysis, where the bound requires the empirical risk and the posterior to be computed on the same sample. In ICL the model is frozen, so conditioning on $D_k$ introduces a mild optimistic bias (the prediction for $x_i$ benefits from having seen $(x_i,y_i)$). This bias is negligible for large k and can be corrected via a leave-one-out adjustment at the cost of a $(k-1)/k$ factor; we retain the standard form for compatibility with the PAC-Bayes theorem.

2.2. PAC-Bayes Preliminaries

  Definition 2

(KL Divergence). For probability measures $\mu ,\nu$ on a measurable space $(\Omega ,\mathcal{F})$ with $\mu \ll \nu$,

$$KL(\mu \parallel \nu )=\int log{\displaystyle \frac{d\mu }{d\nu }}d\mu .$$

The fundamental tool throughout this paper is the McAllester PAC-Bayes bound:

  Theorem 1

(McAllester [12]). Let P be a prior distribution over hypotheses fixed before observing data, Q any data-dependent posterior, $\ell \in [0,1]$ a bounded loss, and k i.i.d. samples. Then for any $\delta \in (0,1]$, with probability at least $1-\delta$:

$$R_{\mathrm{pop}}\left(Q\right)\le R_{\mathrm{emp}}\left(Q\right)+\sqrt{{\displaystyle \frac{KL(Q\parallel P)+log(1/\delta )}{2k}}}.$$

This bound is derived from the Donsker–Varadhan variational inequality [17] combined with Hoeffding’s lemma [18]; see Appendix E for precise statements.

2.3. Ambiguity and Saliency

We parameterize the PAC-Bayes framework with two task-model quantities that are, in principle, computable from the pretrained model.

  Definition 3

(Ambiguity). Theambiguityof task $\mathcal{T}$ given model θ is the expected zero-shot predictive entropy:

$$A(\mathcal{T}\mid \theta )={\mathbb{E}}_x\left[H\left(p_{\theta }(y\mid x,⌀)\right)\right].$$

Ambiguity serves as a natural upper bound on the KL divergence: $KL(Q\parallel P)\le A$.

  Definition 4

(Saliency). Thesaliencyof model θ on task $\mathcal{T}$ is the expected per-demonstration KL reduction:

$$S(\theta ,\mathcal{T})=-{\displaystyle \frac{\partial }{\partial k}}{\mathbb{E}}_{D_k}\left[KL\left(f_{\theta }(·\mid D_k)\parallel f_{\theta }(·\mid ⌀)\right)\right]{|}_{k=0^{+}}.$$

Equivalently, S is the initial rate at which each demonstration reduces the posterior–prior KL divergence, measured in nats per demonstration. Under the linear attention model (Section 4), $S=d·s/\left(2{\sigma }_{\mathrm{prior}}^2\right)$, which is the per-example Fisher information of the linear predictor along the task direction.

  Definition 5

(Prior Quality Ratio). Theprior quality ratiois $Q_{\mathrm{ratio}}=S/A$. A high $Q_{\mathrm{ratio}}$ indicates a model that is both uncertain (A large but finite) and responsive to demonstrations (S large), leading to efficient ICL.

2.4. KL Reduction: Rigorous Bound and Modeling Assumption

Our framework connects the posterior–prior KL divergence to the demonstration count k. We present a two-level treatment: (i) a rigorous KL bound for the Gaussian linear model, and (ii) a first-order modeling assumption for general architectures.

  Proposition 1

(Exact KL Bound under Gaussian Model). Under the Bayesian linear regression model of Section 4 (prior $w\sim \mathcal{N}(0,{\sigma }_p^2{I}_d)$, noise $ϵ\sim \mathcal{N}(0,{\sigma }_n^2)$, isotropic inputs $x\sim \mathcal{N}(0,I_d)$, signal strength s), the expected posterior–prior KL divergence after k demonstrations satisfies:

$${\mathbb{E}}_{D_k}\left[KL(Q_k\parallel P)\right]\le {\displaystyle \frac{d}{2}}log\left(1+{\displaystyle \frac{{\sigma }_p^2}{{\sigma }_n^2+ks}}\right)=:\overline{A}\left(k\right).$$

(3)

The bound $\overline{A}\left(k\right)$ is: (a) monotonically decreasing in k, (b) $\overline{A}\left(0\right)=A_{\mathrm{eff}}$, and (c) $\overline{A}\left(k\right)\to 0$ as $k\to \infty$.

Proof 

(Proof sketch). The posterior covariance satisfies ${\Sigma }_k⪯{({\sigma }_p^{-2}I+ks{\sigma }_n^{-2}I)}^{-1}$ (since $\mathbb{E}\left[x_i{x}_i^{\top }\right]=I$ and $s\le {\sigma }_p^2$). The Gaussian KL in closed form then yields (3). See Appendix B.2 for the full derivation.    □

  Corollary 1

(Linearization with Explicit Error). The exact bound $\overline{A}\left(k\right)$ and the linear approximation satisfy, for all $k\ge 0$:

$$\overline{A}\left(k\right)\le A_{\mathrm{eff}},\left|\overline{A}\left(k\right)-(A_{\mathrm{eff}}-k{S}_{\mathrm{eff}})\right|\le {\displaystyle \frac{d{k}^2{s}^2}{4{\sigma }_n^2({\sigma }_n^2+ks)}}.$$

The error is $O(k^2{s}^2/{\sigma }_n^4)$ and is negligible when $ks\ll {\sigma }_n^2$.

For general (non-Gaussian) architectures, we introduce a modeling assumption:

  Assumption A1

(Approximate KL Reduction). For a general pretrained model $f_{\theta }$, the posterior–prior KL divergence satisfies:

$$KL\left(Q(·\mid D_k)\parallel P\right)\le max(A-k·S,0).$$

  Remark 2.

For the Gaussian model, Assumption A1 follows from Proposition 1 up to the linearization error in Corollary 1. For general Transformer architectures, it remains a modeling hypothesis whose validity for softmax attention is an open question (Section 8).All results in Section 3, Section 4, Section 5 and Section 6 hold verbatim if $max(A-kS,0)$ is replaced by any monotone upper bound on $KL(Q_k\parallel P)$, including the exact bound $\overline{A}\left(k\right)$.

3. Core Generalization Framework

3.1. Upper Bounds

  Theorem 2

(Basic ICL Generalization Bound). Let $\ell \in [0,1]$ be a bounded loss, $k>0$, and suppose $KL(Q\parallel P)\le A$ with $A\ge 0$. Then for any $\delta \in (0,1]$, with probability at least $1-\delta$:

$$R_{\mathrm{pop}}\le R_{\mathrm{emp}}+\sqrt{{\displaystyle \frac{A+log(1/\delta )}{2k}}}.$$

(4)

Proof 

(Proof sketch). Apply Theorem 1 directly. Since $KL(Q\parallel P)\le A$, monotonicity of the square root gives (4). The full proof is in Appendix A.1.    □

  Theorem 3

(Saliency-Enhanced ICL Bound). Under Assumption A1, with probability at least $1-\delta$:

$$R_{\mathrm{pop}}\le R_{\mathrm{emp}}+\sqrt{{\displaystyle \frac{max(A-kS,0)+log(1/\delta )}{2k}}}.$$

(5)

Proof 

(Proof sketch). Replace the KL bound A in Theorem 2 with the tighter bound $max(A-kS,0)$ from Assumption A1. See Appendix A.2.    □

The saliency-enhanced bound (5) converges faster than the basic bound (4) in the pre-saturation regime ($kS<A$), and reduces to $R_{\mathrm{emp}}+\sqrt{log(1/\delta )/\left(2k\right)}$ in the post-saturation regime ($kS\ge A$), where only the confidence term survives.

3.2. Sample Complexity

  Theorem 4

(Sample Complexity). Given a target accuracy $\epsilon >{\epsilon }_{\mathrm{prior}}$ and confidence $\delta \in (0,1]$, if the number of demonstrations satisfies

$$k\ge ⌈{\displaystyle \frac{A+log(1/\delta )}{2{(\epsilon -{\epsilon }_{\mathrm{prior}})}^2}}⌉,$$

(6)

then $R_{\mathrm{pop}}\le \epsilon$.

Proof 

(Proof sketch). From Theorem 2, condition (6) ensures $\sqrt{(A+log(1/\delta \left)\right)/\left(2k\right)}\le \epsilon -{\epsilon }_{\mathrm{prior}}$. Combining with $R_{\mathrm{emp}}\le {\epsilon }_{\mathrm{prior}}$ yields $R_{\mathrm{pop}}\le \epsilon$. The algebraic details are in Appendix A.3.    □

  Remark 3.

The formula exhibits the familiar $O(1/{\epsilon }^2)$ scaling of PAC-Bayes sample complexity. The numerator $A+log(1/\delta )$ captures task difficulty (ambiguity) and confidence requirements; the denominator ${(\epsilon -{\epsilon }_{\mathrm{prior}})}^2$ captures the “improvability gap” between the target accuracy and the zero-shot baseline. High $Q_{\mathrm{ratio}}$ tasks (e.g., sentiment analysis) have small A, requiring few demonstrations; low $Q_{\mathrm{ratio}}$ tasks (e.g., natural language inference) have large A, requiring many more.

Table 2. Predicted sample complexities $k^{*}$ for representative ICL tasks ($\delta =0.05$). Values of A, S, ${\epsilon }_{\mathrm{prior}}$ are illustrative parameters chosen to span different difficulty regimes; Table 7 reports empirically measured values from GPT-2.

Task analogue	A	S	${\mathit{\epsilon }}_{\mathbf{prior}}$	$\mathit{\epsilon }$	${\mathit{k}}^{*}$
Easy (SST-2–like)	0.5	0.3	0.15	0.10	270
Medium (AG News–like)	2.0	0.4	0.30	0.15	237
Hard (SNLI–like)	5.0	0.2	0.45	0.20	567

Table 2. Predicted sample complexities $k^{*}$ for representative ICL tasks ($\delta =0.05$). Values of A, S, ${\epsilon }_{\mathrm{prior}}$ are illustrative parameters chosen to span different difficulty regimes; Table 7 reports empirically measured values from GPT-2.

Task analogue	A	S	${\mathit{\epsilon }}_{\mathbf{prior}}$	$\mathit{\epsilon }$	${\mathit{k}}^{*}$
Easy (SST-2–like)	0.5	0.3	0.15	0.10	270
Medium (AG News–like)	2.0	0.4	0.30	0.15	237
Hard (SNLI–like)	5.0	0.2	0.45	0.20	567

3.3. Information-Theoretic Lower Bound

  Theorem 5

(ICL Classification Lower Bound via Fano). Consider ICL for aclassificationtask with $m>1$ classes. Let $H\left(Y\right)\ge H_{min}$ denote the label entropy and suppose each demonstration provides at most S nats of mutual information about the label, so $I(X;Y\mid D_k)\le kS$. Then the misclassification probability of any ICL estimator satisfies:

$$P_e\ge {\displaystyle \frac{H_{min}-kS-log2}{logm}}.$$

(7)

Proof 

(Proof sketch). Apply Fano’s inequality [19]: $P_e\ge (H\left(Y\right)-I(X;Y)-log2)/logm$. Substitute the bounds $H\left(Y\right)\ge H_{min}$ and $I(X;Y)\le kS$. See Appendix A.4.    □

  Remark 4

(Scope of Theorem 5). Fano’s inequality applies to discrete label spaces. For the regression setting of Section 4, the minimax lower bound (Theorem 10) provides the analogous result via Le Cam’s method without requiring a discrete label space. The two bounds are complementary: Theorem 5 gives a lower bound on classification error; Theorem 10 gives a lower bound on excess risk for regression.

Theorem 5 shows that each demonstration contributes at most S nats of task-relevant information. Complete disambiguation (reducing $P_e$ to zero) requires $k>(H_{min}-log2)/S$ demonstrations—an information-theoretic floor that no model, however powerful, can surpass.

3.4. Permutation Sensitivity

The ordering of in-context demonstrations can affect performance. We quantify this sensitivity using the Efron–Stein framework.

  Theorem 6

(Permutation Variance Bound). Let σ denote a random permutation of k demonstrations. Under the bounded-differences condition $c^2\le 4S/k^2$ (i.e., swapping any two demonstrations changes the loss by at most c), the variance of the ICL error over permutations satisfies:

$${Var}_{\sigma }\left[\epsilon (k,\sigma )\right]\le {\displaystyle \frac{S}{k}}.$$

(8)

Proof 

(Proof sketch). The Efron–Stein inequality [20] gives $Var\le k{c}^2/4$. Substituting $c^2\le 4S/k^2$ and simplifying yields $S/k$. See Appendix A.5.    □

The $1/k$ decay rate implies that ordering sensitivity vanishes for large k. High-saliency tasks are more order-sensitive—a double-edged sword of saliency: responsiveness accelerates learning but increases sensitivity to demonstration arrangement.

4. Linear Attention: Architecture-Dependent Bounds

The bounds in Section 3 are expressed in terms of the abstract quantities A and S. In this section, we instantiate these for a concrete architecture—linear attention—and obtain bounds that depend directly on model parameters.

4.1. Linear Attention as Bayesian Linear Regression

Following [4] and [21], we model a single-layer linear attention head performing ICL on d-dimensional inputs as implicit Bayesian linear regression. The generative model is:

• Task parameter: $w\sim \mathcal{N}(0,{\sigma }_{\mathrm{prior}}^2{I}_d)$;
• Observations: $y_i=w^{\top }{x}_i+{ϵ}_i$, with ${ϵ}_i\sim \mathcal{N}(0,{\sigma }_{\mathrm{noise}}^2)$;
• Signal strength: a parameter $s\in (0,{\sigma }_{\mathrm{prior}}^2]$ measuring the effective task signal captured by the attention head.

Under this model, the posterior after k demonstrations is Gaussian with a closed-form KL divergence to the prior, enabling exact computation of A and S.

4.2. Closed-Form Ambiguity and Saliency

  Definition 6

(Effective Ambiguity). Under the linear attention model with dimension d, prior variance ${\sigma }_{\mathrm{prior}}^2$, and noise variance ${\sigma }_{\mathrm{noise}}^2$:

$$A_{\mathrm{eff}}={\displaystyle \frac{d}{2}}log\left(1+{\displaystyle \frac{{\sigma }_{\mathrm{prior}}^2}{{\sigma }_{\mathrm{noise}}^2}}\right).$$

  Definition 7

(Effective Saliency). Under the same model with signal strength s:

$$S_{\mathrm{eff}}={\displaystyle \frac{d·s}{2{\sigma }_{\mathrm{prior}}^2}}.$$

Both $A_{\mathrm{eff}}$ and $S_{\mathrm{eff}}$ are strictly positive for all valid model parameters, since $d\ge 1$, ${\sigma }_{\mathrm{prior}}^2>0$, ${\sigma }_{\mathrm{noise}}^2>0$, $s>0$, and $log(1+x)>0$ for $x>0$.

4.3. Simplified Bounds via Classical Inequalities

  Lemma 1

(Ambiguity Upper Bound). For all valid parameters:

$$A_{\mathrm{eff}}\le {\displaystyle \frac{d·{\sigma }_{\mathrm{prior}}^2}{2{\sigma }_{\mathrm{noise}}^2}}.$$

(9)

Proof. 

Apply $log(1+x)\le x$ for $x\ge 0$ with $x={\sigma }_{\mathrm{prior}}^2/{\sigma }_{\mathrm{noise}}^2$, then multiply by $d/2$.    □

  Lemma 2

(Saliency Upper Bound). When $s\le {\sigma }_{\mathrm{prior}}^2$: $S_{\mathrm{eff}}\le d/2$.

Proof 

(Proof sketch). Since $s/{\sigma }_{\mathrm{prior}}^2\le 1$, we have $S_{\mathrm{eff}}=(d/2)(s/{\sigma }_{\mathrm{prior}}^2)\le d/2$. See Appendix B.1.    □

The bound (9) replaces the logarithmic expression with a simpler rational one, making sample complexity formulas more transparent.

4.4. The Saturation Phenomenon

  Lemma 3

(KL Saturation). When $k·S_{\mathrm{eff}}\ge A_{\mathrm{eff}}$, $max(A_{\mathrm{eff}}-k·S_{\mathrm{eff}},0)=0$ and the KL complexity term vanishes entirely.

  Theorem 7

(Post-Saturation Bound). For $k\ge \lceil A_{\mathrm{eff}}/S_{\mathrm{eff}}\rceil$, with probability at least $1-\delta$:

$$R_{\mathrm{pop}}\le R_{\mathrm{emp}}+\sqrt{{\displaystyle \frac{log(1/\delta )}{2k}}}.$$

(10)

Proof 

(Proof sketch). By Theorem 3, the KL term in Assumption A1 equals zero. Apply Theorem 1 with $KL(Q\parallel P)=0$ to obtain (10), the tightest possible PAC-Bayes bound where only the sub-Gaussian concentration term remains. See Appendix B.3.    □

The saturation point $k^{*}=\lceil A_{\mathrm{eff}}/S_{\mathrm{eff}}\rceil$ represents a phase transition: below it, each demonstration contributes $S_{\mathrm{eff}}$ nats of information, tightening the bound; above it, the KL term has been fully absorbed, and only the confidence penalty persists. Under the linear attention model:

$$k^{*}=⌈{\displaystyle \frac{{\sigma }_{\mathrm{prior}}^2}{s}}·log\left(1+{\displaystyle \frac{{\sigma }_{\mathrm{prior}}^2}{{\sigma }_{\mathrm{noise}}^2}}\right)⌉,$$

which is computable from the model architecture.

4.5. End-to-End Architecture-Dependent Bound

  Theorem 8

(Linear Attention ICL Bound). Under the linear attention model with parameters $(d,{\sigma }_{\mathrm{prior}}^2,{\sigma }_{\mathrm{noise}}^2,s)$, with probability at least $1-\delta$:

$$R_{\mathrm{pop}}\le R_{\mathrm{emp}}+\sqrt{{\displaystyle \frac{max(A_{\mathrm{eff}}-k·S_{\mathrm{eff}},0)+log(1/\delta )}{2k}}},$$

(11)

where $A_{\mathrm{eff}}={\displaystyle \frac{d}{2}}log(1+{\sigma }_{\mathrm{prior}}^2/{\sigma }_{\mathrm{noise}}^2)$ and $S_{\mathrm{eff}}=ds/\left(2{\sigma }_{\mathrm{prior}}^2\right)$.

Proof 

(Proof sketch). The Bayesian conjugacy structure of the Gaussian model implies that the posterior KL after k demonstrations satisfies Assumption A1 with $A=A_{\mathrm{eff}}$ and $S=S_{\mathrm{eff}}$. Apply Theorem 3 with these values. See Appendix B.4.    □

4.6. Simplified Sample Complexity

  Theorem 9

(Architecture-Dependent Sample Complexity). Under the linear attention model, if

$$k\ge ⌈{\displaystyle \frac{d·{\sigma }_{\mathrm{prior}}^2/\left(2{\sigma }_{\mathrm{noise}}^2\right)+log(1/\delta )}{2{(\epsilon -{\epsilon }_{\mathrm{prior}})}^2}}⌉,$$

(12)

then $R_{\mathrm{pop}}\le \epsilon$.

Proof 

(Proof sketch). Apply Theorem 1 to weaken $A_{\mathrm{eff}}$ in Theorem 4, replacing the logarithmic term with the simpler rational upper bound. See Appendix B.5.    □

  Corollary 2.

The required number of demonstrations scales as $k^{*}=O\left(d·SNR/{\epsilon }^2\right)$, where $SNR={\sigma }_{\mathrm{prior}}^2/{\sigma }_{\mathrm{noise}}^2$ is the signal-to-noise ratio. Reducing d (via projection or compression) or improving SNR (via better pretraining) directly reduces the demonstration requirement.

Table 3. Architecture-dependent sample complexities for $\epsilon -{\epsilon }_{\mathrm{prior}}=0.1$, $\delta =0.05$.

d	${\mathit{\sigma }}_{\mathbf{prior}}^2$	${\mathit{\sigma }}_{\mathbf{noise}}^2$	${\mathit{A}}_{\mathbf{eff}}$	SNR	${\mathit{k}}^{*}$
5	1.0	1.0	1.73	1.0	238
10	1.0	1.0	3.47	1.0	325
10	1.0	0.5	5.49	2.0	426
20	0.5	1.0	2.23	0.5	263
50	1.0	1.0	17.33	1.0	1018

Table 3. Architecture-dependent sample complexities for $\epsilon -{\epsilon }_{\mathrm{prior}}=0.1$, $\delta =0.05$.

d	${\mathit{\sigma }}_{\mathbf{prior}}^2$	${\mathit{\sigma }}_{\mathbf{noise}}^2$	${\mathit{A}}_{\mathbf{eff}}$	SNR	${\mathit{k}}^{*}$
5	1.0	1.0	1.73	1.0	238
10	1.0	1.0	3.47	1.0	325
10	1.0	0.5	5.49	2.0	426
20	0.5	1.0	2.23	0.5	263
50	1.0	1.0	17.33	1.0	1018

5. Minimax Optimality and Fast Rates

5.1. Minimax Lower Bounds

Having established an $O\left(\sqrt{A/k}\right)$ upper bound, a natural question is whether this rate is optimal. We answer affirmatively using Le Cam’s two-point method.

  Theorem 10

(Minimax Lower Bound). Consider the class of ICL tasks with bounded loss $\ell \in [0,1]$ and ambiguity at most A. For any ICL estimator and any $k\ge 1$, there exist two tasks ${\mathcal{T}}_0,{\mathcal{T}}_1$ with $A\left({\mathcal{T}}_j\right)\le A$ such that the maximum risk over the pair satisfies:

$$R_{max}\ge {\displaystyle \frac{\sqrt{A/k}}{4}}\left(1-\sqrt{{\displaystyle \frac{A}{4}}}\right).$$

(13)

Proof 

(Proof sketch) Concrete construction.. Let $e_1$ denote the first standard basis vector in ${\mathbb{R}}^d$. Define two Gaussian regression tasks:

• ${\mathcal{T}}_0$: $y=0·x^{\top }{e}_1+ϵ$,    $ϵ\sim \mathcal{N}(0,{\sigma }_n^2)$;
• ${\mathcal{T}}_1$: $y=\Delta ·x^{\top }{e}_1+ϵ$,    $ϵ\sim \mathcal{N}(0,{\sigma }_n^2)$;

with risk gap $\Delta =\sqrt{A/k}/2$ and per-sample KL divergence $\kappa ={\Delta }^2/\left(2{\sigma }_n^2\right)$. Both tasks satisfy $A\left({\mathcal{T}}_j\right)\le A$ since the zero-shot prediction is the same (zero mean) with entropy determined by ${\sigma }_p,{\sigma }_n$. By KL tensorization, $KL(P_0^k\parallel P_1^k)=k\kappa =A/2$. Le Cam’s two-point method [22] gives $R_{max}\ge (\Delta /2)(1-\sqrt{k\kappa /2})$. Substituting and simplifying yields (13). See Appendix C.1 for the complete proof.    □

  Theorem 11

(Simplified Minimax Lower Bound). When $A\le 1$, the lower bound simplifies to:

$$R_{max}\ge {\displaystyle \frac{\sqrt{A/k}}{8}}.$$

(14)

Proof 

(Proof sketch). When $A\le 1$, we have $\sqrt{A/4}\le 1/2$, so $1-\sqrt{A/4}\ge 1/2$. Multiply the factor of $1/4$ by $1/2$ to get $1/8$. See Appendix C.2.    □

  Theorem 12

(Rate Optimality). The ICL generalization rate is $\Theta \left(\sqrt{A/k}\right)$:

• Upper bound: $R_{\mathrm{pop}}-R_{\mathrm{emp}}=O\left(\sqrt{A/k}\right)$ (Theorem 2);
• Lower bound: $R_{max}=\Omega \left(\sqrt{A/k}\right)$ (Theorem 11).

Both bounds match up to universal constants, establishing rate optimality.

  Theorem 13

(Saturation-Regime Lower Bound). After saturation ($kS\ge A$), an information-theoretic cost persists:

$$R_{max}\ge {\displaystyle \frac{\sqrt{log2/k}}{8}}.$$

(15)

This shows that even after the KL complexity term vanishes, a $\Theta (1/\sqrt{k})$ confidence-based irreducible term remains.

5.2. Catoni Fast-Rate Bounds

The McAllester bound (Theorem 1) yields an $O(1/\sqrt{k})$ excess risk. Using Catoni’s temperature-parameterized PAC-Bayes bound [13], we can achieve $O(1/k)$ when the empirical risk is small.

  Theorem 14

(Catoni ICL Bound). Setting the temperature parameter $\lambda =1$ in Catoni’s bound and using $KL(Q\parallel P)\le A$:

$$R_{\mathrm{pop}}\le 2\left(R_{\mathrm{emp}}+{\displaystyle \frac{A+log(2\sqrt{k}/\delta )}{k}}\right).$$

(16)

The excess risk is $O(1/k)$ when $R_{\mathrm{emp}}$ is small.

Proof 

(Proof sketch). Catoni’s bound [13] with parameter $\lambda \in (0,2)$ gives

$$R_{\mathrm{pop}}\le {\displaystyle \frac{1}{1-\lambda /2}}\left(R_{\mathrm{emp}}+{\displaystyle \frac{KL(Q\parallel P)+log(2\sqrt{k}/\delta )}{\lambda k}}\right).$$

Setting $\lambda =1$: the coefficient becomes $1/(1-1/2)=2$ and $\lambda k=k$. Weakening $KL(Q\parallel P)\le A$ yields (16). See Appendix C.3.    □

  Theorem 15

(Fast-Rate Sample Complexity). If $R_{\mathrm{emp}}\le \epsilon /4$ and $k\ge 8(A+log(2\sqrt{k}/\delta ))/\epsilon$, then $R_{\mathrm{pop}}\le \epsilon$. Since $log(2\sqrt{k}/\delta )\le log(2k/\delta )$, an explicit sufficient condition is $k\ge 16A/\epsilon +8log(4A/(\epsilon \delta \left)\right)/\epsilon$ (obtained by solving the implicit inequality via $k\ge 2k_0$ where $k_0=8A/\epsilon$). The sample complexity is $O\left(\right(A+log(1/\delta ))/\epsilon )$—a quadratic improvement over the McAllester-based $O(1/{\epsilon }^2)$.

Proof 

(Proof sketch). Each of the two terms in (16) contributes at most $\epsilon /4$, and $2(\epsilon /4+\epsilon /4)=\epsilon$. See Appendix C.4.    □

  Lemma 4

(Catoni Dominates McAllester for Large k). Let $C_0=A+log(2/\delta )$. An explicit sufficient condition for the Catoni excess risk to be strictly smaller than the McAllester excess risk is $k\ge 16C_0$ (since $C=C_0+{\displaystyle \frac{1}{2}}logk\le 2C_0$ for $k\le e^{2C_0}$, giving $8C\le 16C_0\le k$).

Proof. 

Under $k\ge 8C$, squaring both sides of $2C/k\le \sqrt{C/\left(2k\right)}$ gives $4C^2/k^2\le C/\left(2k\right)$, i.e., $8C\le k$, which holds by hypothesis.    □

  Theorem 16

(Fast Rate with Zero Empirical Risk). Under perfect empirical fit ($R_{\mathrm{emp}}=0$):

$$R_{\mathrm{pop}}\le {\displaystyle \frac{2\left(A+log(2\sqrt{k}/\delta )\right)}{k}}.$$

(17)

This is the purest form of the fast rate and is particularly relevant for ICL, where well-pretrained models typically achieve near-zero empirical risk on in-context demonstrations.

6. Extensions

6.1. Data-Dependent Priors via Sample Splitting

Standard PAC-Bayes requires data-independent priors, but in ICL the zero-shot distribution implicitly depends on pretraining data. The split-sample technique [16] resolves this by partitioning k demonstrations into $k_1$ (for constructing an informed prior) and $k_2=k-k_1$ (for evaluation).

  Theorem 17

(Information Gain Reduces KL). Using $k_1$ demonstrations to build an informed prior, the effective KL satisfies:

$$KL(Q\parallel P_{\mathrm{informed}})\le max(A-k_1·\mathrm{ig},0),$$

(18)

where $\mathrm{ig}>0$ is the per-demonstration information gain.

Proof 

(Proof sketch). Each of the $k_1$ demonstrations contributes $\mathrm{ig}$ nats toward resolving the task ambiguity, reducing the remaining KL from A to $A-k_1·\mathrm{ig}$. The $max(·,0)$ ensures non-negativity. See Appendix D.1.    □

  Theorem 18

(Data-Dependent ICL Bound). With split-sample PAC-Bayes, the population risk satisfies:

$$R_{\mathrm{pop}}\le R_{\mathrm{emp}}\left(k_2\right)+\sqrt{{\displaystyle \frac{max(A-k_1·\mathrm{ig},0)+log(1/\delta )}{2k_2}}}.$$

(19)

Proof 

(Proof sketch). Apply the split-sample PAC-Bayes bound [16] with the reduced KL from Theorem 17, then weaken the KL term via $\sqrt{·}$ monotonicity. See Appendix D.1.    □

  Theorem 19

(Fully Informed Prior). When $k_1·\mathrm{ig}\ge A$ (the prior is “fully informed”):

$$R_{\mathrm{pop}}\le R_{\mathrm{emp}}\left(k_2\right)+\sqrt{{\displaystyle \frac{log(1/\delta )}{2k_2}}}.$$

(20)

The ambiguity term vanishes entirely.

Proof 

(Proof sketch). When $k_1·\mathrm{ig}\ge A$, we have $max(A-k_1·\mathrm{ig},0)=0$. Substitute into (19) and simplify. See Appendix D.1.    □

The split bound’s numerator is always at most the standard bound’s, since $max(A-k_1·\mathrm{ig},0)\le A$.

The optimal split ratio $k_1/k$ depends on the information gain rate. When $\mathrm{ig}$ is large, a small $k_1$ suffices to eliminate ambiguity, leaving most demonstrations for evaluation.

6.2. Bernstein Variance-Adaptive Bounds

When the model is highly confident (low loss variance), the Hoeffding-based bounds in Section 3 are loose. Bernstein-type PAC-Bayes bounds [14,15] exploit low variance.

  Theorem 20

(Bernstein ICL Bound). Let V denote the loss variance and $C_{\delta }=log(2\sqrt{k}/\delta )$. Under $KL(Q\parallel P)\le A$:

$$R_{\mathrm{pop}}\le R_{\mathrm{emp}}+\sqrt{{\displaystyle \frac{2V(A+C_{\delta })}{k}}}+{\displaystyle \frac{7(A+C_{\delta })}{3k}}.$$

(21)

Proof 

(Proof sketch). Apply the Bernstein PAC-Bayes bound [15] and weaken $KL(Q\parallel P)\le A$ in both the $\sqrt{·}$ (variance) term and the linear (remainder) term via monotonicity. See Appendix D.2.    □

The bound (21) has two terms: a “variance term” $\sqrt{2V(A+C_{\delta })/k}$ and a “remainder term” $7(A+C_{\delta })/\left(3k\right)$. When V is small, the variance term becomes negligible, and the remainder term dominates at rate $O(1/k)$.

  Theorem 21

(Variance Decay ⇒ Fast Rate). Under variance decay $V\left(k\right)\le V_0/k$, the Bernstein bound becomes:

$$R_{\mathrm{pop}}\le R_{\mathrm{emp}}+\sqrt{{\displaystyle \frac{2V_0(A+C_{\delta })}{k^2}}}+{\displaystyle \frac{7(A+C_{\delta })}{3k}}.$$

(22)

Since $\sqrt{2V_{0}C/k^2}=\sqrt{2V_{0}C}/k$,both terms are $O(1/k)$, yielding a fast rate.

Proof 

(Proof sketch). Substitute $V\le V_0/k$ into the variance term of (21). The numerator factor $2V·C$ becomes $2(V_0/k)·C$, and dividing by k gives $2V_{0}C/k^2$. See Appendix D.2.    □

  Lemma 5

(Bernstein Sqrt Term Absorption). When $V\le C/\left(2k\right)$ (where $C=A+C_{\delta }$): $\sqrt{2VC/k}\le C/k$. The variance term is absorbed into the $O(1/k)$ scale.

Proof 

(Proof sketch). We show $2VC/k\le {(C/k)}^2$ by substituting $V\le C/\left(2k\right)$: the left side is at most $2·C/\left(2k\right)·C/k=C^2/k^2$, which equals the right side. Apply $\sqrt{·}$ monotonicity and $\sqrt{{(C/k)}^2}=C/k$. See Appendix D.2.    □

Table 4. Comparison of the three PAC-Bayes bound families for ICL.

Bound Family	Excess Risk Rate	Key Assumption	Best Regime
McAllester (Thm 2)	$O(1/\sqrt{k})$	$KL\le A$	General
Catoni (Thm 14)	$O(1/k)$	$R_{\mathrm{emp}}$ small	Large k, low emp. risk
Bernstein (Thm 20)	$O(1/k)$	Variance V small	Confident model

Table 4. Comparison of the three PAC-Bayes bound families for ICL.

Bound Family	Excess Risk Rate	Key Assumption	Best Regime
McAllester (Thm 2)	$O(1/\sqrt{k})$	$KL\le A$	General
Catoni (Thm 14)	$O(1/k)$	$R_{\mathrm{emp}}$ small	Large k, low emp. risk
Bernstein (Thm 20)	$O(1/k)$	Variance V small	Confident model

Theorem 21 provides a second path to fast rates, complementing Catoni (Section 5.2). The Catoni path requires low empirical risk; the Bernstein path requires low variance. Both conditions are natural for well-pretrained models: on tasks where the model is well-matched, empirical risk is near zero and the model is confident (low variance).

7. Experimental Validation

We validate the key theoretical predictions in two complementary settings: (i) synthetic Bayesian linear regression tasks that exactly match the linear attention model of Section 4 (Experiments 1–4), and (ii) real in-context learning with GPT-2 on NLP classification benchmarks where model assumptions may not hold (Experiments 5–7). Synthetic experiments use $N=2,000$ independent trials and confidence level $\delta =0.05$.

7.1. Setup

Data generation.

For each trial, we sample a task parameter $w\sim \mathcal{N}(0,{\sigma }_{\mathrm{prior}}^2{I}_d)$ and k input–output pairs $(x_i,y_i)$ with $x_i\sim \mathcal{N}(0,I_d)$ and $y_i=w^{\top }{x}_i+{ϵ}_i$, ${ϵ}_i\sim \mathcal{N}(0,{\sigma }_{\mathrm{noise}}^2)$. The ICL prediction is the Bayesian posterior mean ${\widehat{w}}_k$ after k demonstrations. The loss is the squared error clipped to $[0,1]$: $\ell (\widehat{y},y)=min({(\widehat{y}-y)}^2,1)$.

Configurations.

We consider three configurations that span different effective ambiguity and saliency regimes, with signal strength parameter s controlling the effective saliency:

Config	d	${\sigma }_p^2$	${\sigma }_n^2$	s	$A_{\mathrm{eff}}$	$S_{\mathrm{eff}}$	$k^{*}$
A	5	1.0	1.0	0.15	1.73	0.375	5
B	10	1.0	1.0	0.10	3.47	0.500	7
C	10	1.0	0.5	0.10	5.49	0.500	11

7.2. Experiment 1: Generalization Bound Tightness

For each configuration and each k, we compute the average population risk $\widehat{R_{\mathrm{pop}}}\left(k\right)$ and training risk $\widehat{R_{\mathrm{emp}}}\left(k\right)$ over N trials, and compare the generalization gap $\widehat{R_{\mathrm{pop}}}-\widehat{R_{\mathrm{emp}}}$ against the saliency-enhanced bound (Theorem 3).

Table 5 reports results for Config C ($d=10$, ${\sigma }_n^2=0.5$, $k^{*}=11$), which has the largest effective ambiguity.

Observations.

The bound holds as a high-probability statement over individual trials. When examining the average gap, we observe that at $k=10-15$ the mean gap exceeds the bound. This is not a bound violation: the PAC-Bayes bound guarantees $R_{\mathrm{pop}}-R_{\mathrm{emp}}\le \mathrm{Bound}$ with probability $\ge 1-\delta$, and the empirical violation rate over trials is $<\delta =0.05$, consistent with the theoretical guarantee. Before saturation ($k<k^{*}$), the bound decreases rapidly due to the $max(A_{\mathrm{eff}}-k{S}_{\mathrm{eff}},0)$ term; after saturation, only the $O(1/\sqrt{k})$ confidence term remains. At $k=1-2$, the bound exceeds 1, making it vacuous (for $\ell \in [0,1]$); this is typical of PAC-Bayes bounds at very small sample sizes [23]. Results for Configs A and B exhibit the same qualitative pattern.

7.3. Experiment 2: Risk Scaling

We track population risk as a function of k for Config B ($d=10$, ${\sigma }_n^2=1.0$) over the range $k\in \{1,\dots ,200\}$. Figure 1 shows the convergence behavior.

Observations.

The population risk decreases monotonically with k, from $0.815$ at $k=1$ to $0.518$ at $k=200$. The increasing ratio $\widehat{R_{\mathrm{pop}}}/\sqrt{A/k}$ reflects the fact that the total risk includes an irreducible noise component ${\sigma }_n^2$-dependent Bayes risk that does not vanish. The theoretical bound remains valid throughout, converging toward the empirical risk as k grows. This confirms that the $O\left(\sqrt{A/k}\right)$ rate correctly captures the learnable component of the risk.

7.4. Experiment 3: Saturation Point Verification

Theorems 3–7 predict a phase transition at $k^{*}=\lceil A_{\mathrm{eff}}/S_{\mathrm{eff}}\rceil$. We verify this using Config C ($k^{*}=11$), which has the largest saturation point. Figure 2 shows the KL complexity term and bound gap before and after $k^{*}$.

Observations.

The KL complexity term $max(A_{\mathrm{eff}}-k{S}_{\mathrm{eff}},0)$ decreases linearly from $4.99$ at $k=1$ to 0 at $k=11$ and remains zero thereafter, exactly as predicted. The bound gap drops rapidly in the pre-saturation regime (from $2.00$ to $0.37$) due to the combined effect of KL reduction and $1/\sqrt{k}$ decay, then transitions to the slower $O(1/\sqrt{k})$-only rate post-saturation. The empirical risk mirrors this transition: the risk decreases by $0.21$ units over $k=1$ to 11 (pre-saturation), but only by $0.20$ units over $k=11$ to 50 (post-saturation), despite the latter interval being $4\times$ wider. This confirms the phase transition at $k^{*}=11$.

7.5. Experiment 4: McAllester vs. Catoni Crossover

We compare the McAllester bound (Theorem 2) and the Catoni bound (Theorem 14) as functions of k for Config B with $R_{\mathrm{emp}}=0$ (the perfect empirical fit regime most relevant for well-pretrained models). Figure 3 shows the two bound curves.

Observations.

For small k, the Catoni bound is substantially looser due to its multiplicative constant of 2 (cf. Theorem 14). As k increases, the $O(1/k)$ scaling of the Catoni excess term dominates the $O(1/\sqrt{k})$ McAllester term. The crossover occurs at $k=113$, after which Catoni provides strictly tighter guarantees. Theorem 4 predicts the crossover condition $k\ge 8C$ where $C=A+log(2\sqrt{k}/\delta )$; at $k=113$, $C\approx 9.5$, giving $8C\approx 76$, so the condition $k=113>76$ is indeed satisfied. The gap widens dramatically for large k: at $k=500$, the Catoni bound ($0.041$) is approximately half the McAllester bound ($0.080$), confirming the practical relevance of fast rates.

With $R_{\mathrm{emp}}=0.01$, the crossover shifts to $k=130$; with $R_{\mathrm{emp}}=0.05$, the crossover exceeds $k=500$, illustrating the sensitivity of the fast-rate advantage to the empirical risk level.

The preceding experiments validate the framework in a controlled setting where all theoretical quantities are known in closed form. We now turn to experiments with a real pretrained Transformer—GPT-2 (124M parameters) [24]—on genuine NLP classification tasks, where Assumption A1 may not hold exactly and quantities A and S must be estimated empirically.

7.6. Real Transformer Setup

Model and protocol.

We use GPT-2 [24] (124M parameters, 12 layers, 768 hidden dimensions) with softmax attention and positional encoding—the full architecture, not the linear attention approximation of Section 4. For each task, we format ICL as: k labeled demonstrations followed by an unlabeled query, all concatenated in the prompt. The model’s prediction is obtained via restricted softmax: we extract the logits at the last token position for the valid label tokens only (each verified to be a single BPE token) and apply softmax to obtain a probability distribution over labels. The loss is the 0–1 classification error: $\ell (\widehat{y},y)=\mathbf{1}[\widehat{y}\ne y]\in [0,1]$.

Tasks.

We evaluate on three NLP classification benchmarks spanning different numbers of classes: SST-2 [25] (binary sentiment, label tokens: Positive/Negative), AG News [26] (4-class topic classification: World/Sports/Business/Technology), and SNLI [27] (3-class natural language inference: Yes/Maybe/No).

Estimation protocol.

For each task, we use 100 test examples and 500 demonstration candidates. Ambiguity $\widehat{A}$ is estimated as the mean zero-shot entropy: $\widehat{A}=(1/N){\sum }_{i=1}^{N}H\left(p_{\theta }(y\mid x_i,⌀)\right)$. KL divergence $\widehat{\mathrm{KL}}\left(k\right)$ is estimated as the mean $\mathrm{KL}(p_{\theta }(y\mid x,D_k)\parallel p_{\theta }(y\mid x,⌀))$ averaged over $N=100$ test examples and $T=20$ random demonstration draws. Saliency $\widehat{S}$ is estimated from the empirical risk curve: $\widehat{S}=\widehat{A}/k_{\mathrm{sat}}$, where $k_{\mathrm{sat}}$ is the smallest k achieving $90\%$ of the maximum risk improvement.

7.7. Experiment 5: Bound Verification with GPT-2

Table 6 reports the PAC-Bayes bound evaluation on SST-2 with GPT-2. The estimated ambiguity is $\widehat{A}=0.663$ nats (near the maximum $ln2\approx 0.693$ for binary classification), reflecting that GPT-2’s zero-shot label distribution is nearly uniform for sentiment under our prompt format.

Observations.

The PAC-Bayes bound holds for all k values: $\widehat{R_{\mathrm{pop}}}<\widehat{R_{\mathrm{emp}}}+\mathrm{Bd}$ throughout. The bound gaps decrease from $1.24$ ($k=1$) to $0.25$ ($k=24$) at the $O(1/\sqrt{k})$ rate predicted by Theorem 2, and the measured-KL bound is consistently tighter than the A-only bound (6–$9\%$ improvement), confirming that using the empirical KL rather than the ambiguity upper bound yields tighter guarantees. The absolute bounds remain vacuous ($\widehat{R_{\mathrm{emp}}}+\mathrm{Bd}>1$) because GPT-2’s empirical risk on its own demonstrations is high ($\widehat{R_{\mathrm{emp}}}\approx 0.73$–$1.0$), reflecting that this small model does not reliably “copy” label patterns from context. The high $\widehat{R_{\mathrm{emp}}}$ is itself informative: the bound correctly diagnoses poor ICL by producing large values. As the model quality improves (larger models, lower $\widehat{R_{\mathrm{emp}}}$), the bounds become progressively tighter.

7.8. Experiment 6: KL Behavior Under Softmax Attention

Assumption A1 posits that $KL(Q_k\parallel P)$ decreases linearly with k as $max(A-kS,0)$. Figure 4 tests this prediction by comparing the measured $\widehat{\mathrm{KL}}\left(k\right)$ against the linear model.

Observations.

The measured KL divergence $\widehat{\mathrm{KL}}\left(k\right)$ is approximately constant across k ($0.08$–$0.14$ nats), in contrast to the linear decrease predicted by Assumption A1. This confirms that the linear KL reduction is a modeling hypothesis specific to the Gaussian model, not a property of softmax Transformers. Crucially, the measured-KL bound (Theorem 2) remains valid throughout, as guaranteed by the theory. The saliency-enhanced bound (which assumes linear KL decay) is also valid here because the saturation point $k_{\mathrm{pred}}^{*}=1$ means $max(\widehat{A}-k\widehat{S},0)=0$ for all tested k, reducing the saliency bound to the same $log(1/\delta )/\left(2k\right)$ form. As noted in Remark 2, all bounds hold verbatim when $max(A-kS,0)$ is replaced by any upper bound on $KL(Q_k\parallel P)$; the constant KL profile observed here satisfies $\widehat{\mathrm{KL}}\left(k\right)<\widehat{A}$ for all k, so the A-only bound (which requires only $KL\le A$) is always valid.

7.9. Experiment 7: Multi-Task Complexity Estimation

Table 7 compares the estimated $\widehat{A}$, $\widehat{S}$, and predicted $k^{*}$ across three tasks of varying difficulty.

Observations.

The results reveal a clear saliency ordering: SST-2 has the highest estimated saliency ($\widehat{S}=0.041$) and shows the largest ICL improvement ($\Delta R=0.17$), while SNLI has the lowest saliency ($\widehat{S}=0.023$) and negligible improvement ($\Delta R=0.03$). This ordering—easy sentiment task responds most to demonstrations, hard inference task responds least—matches the qualitative predictions of the framework.

The AG News result is instructive: despite having the highest ambiguity ($\widehat{A}=0.785$), the model’s performance degrades with demonstrations ($\Delta R<0$). This occurs because GPT-2’s zero-shot topic prediction is already reasonable (70% accuracy), but ICL with the prompt format used here introduces conflicting signals that confuse the model. The framework correctly captures this failure: the low $\widehat{S}$ and high $\widehat{A}$ predict that many demonstrations are needed ($k^{*}=24$), and indeed performance has not converged within the $k\le 24$ window. This demonstrates that the $(A,S)$ parameterization provides meaningful diagnostic information even when ICL fails.

Compared to the hypothetical values in Table 2, the measured ambiguities ($0.56$–$0.79$ nats) are of the same order as the assumed values ($0.5$–$5.0$ nats), though the measured range is compressed because GPT-2’s label-space entropy is limited by the number of classes. The saliency estimates ($0.02$–$0.04$) are substantially smaller than the assumed values ($0.2$–$0.4$), reflecting GPT-2’s limited ICL capacity on these tasks. Larger models with stronger ICL abilities would exhibit higher $\widehat{S}$, yielding tighter bounds and smaller $k^{*}$.

8. Discussion

8.1. Implications for Model Design

The sample complexity formula $k^{*}=O(d·SNR/{\epsilon }^2)$ from Theorem 9 and Corollary 2 has direct design implications:

• Dimensionality reduction: Reducing the effective dimension d (via projection heads or attention bottlenecks) linearly reduces the demonstration requirement.
• Pretraining quality: Improving the SNR through better pretraining data directly improves ICL sample efficiency.
• Diminishing returns: Beyond the saturation point $k^{*}=\lceil A_{\mathrm{eff}}/S_{\mathrm{eff}}\rceil$, additional demonstrations provide only $O\left(\sqrt{log(1/\delta )/\left(2k\right)}\right)$ improvement—a much slower rate.

8.2. ICL vs. Fine-Tuning

Fine-tuning a model on k examples yields excess risk scaling as $O\left(\sqrt{d_{\mathrm{eff}}/k}\right)$, where $d_{\mathrm{eff}}$ is the effective parameter count. ICL achieves $O\left(\sqrt{A/k}\right)$ where A is typically much smaller than $d_{\mathrm{eff}}$, since ICL operates in the model’s “natural” predictive space rather than the full parameter space. This explains why ICL can outperform fine-tuning with very few demonstrations when the model’s prior is well-matched to the task. The crossover occurs when k is large enough that fine-tuning’s greater flexibility outweighs ICL’s lower effective complexity.

8.3. ICL vs. Standard Machine Learning

Our framework reveals that ICL and standard supervised learning differ not merely in constants but in the objects the PAC-Bayes machinery operates on. In standard PAC-Bayes theory, the posterior Q and prior P live over model parameters; $KL(Q\parallel P)$ measures parameter deviation, and sample complexity scales as $O(d_{\mathrm{eff}}/{\epsilon }^2)$. In ICL, no parameters are updated: Q is the conditional predictive distribution $f_{\theta }(·\mid D_k)$ of a frozen model, P is its zero-shot prediction encoding pretraining knowledge, and $KL(Q\parallel P)$ equals the ambiguity A—a measure of prediction shift, not parameter deviation. Consequently, sample complexity $O(A/{\epsilon }^2)$ (Theorem 4) is decoupled from model size: a 175B-parameter model may have $A\ll 1$ for a well-matched task. Moreover, ICL exhibits hard saturation at $k^{*}=\lceil A/S\rceil$ (Theorem 3), with no analog in standard learning where more data always helps. Table 8 summarizes these structural contrasts.

8.4. Limitations

Several limitations warrant discussion:

1.

KL reduction assumption: Proposition 1 provides a rigorous KL bound for the Gaussian model with explicit linearization error (Corollary 1). For general Transformer architectures, Assumption A1 remains a modeling hypothesis. Experiment 6 confirms that softmax attention exhibits approximately constant (not linearly decreasing) KL divergence across k, but the core PAC-Bayes bound (Theorem 2) remains valid with the measured KL values.

2.

Bounded loss: The $\ell \in [0,1]$ requirement excludes unbounded losses such as cross-entropy on raw logits. Extending to sub-Gaussian losses is a natural direction.

3.

Vacuous bounds at small k: As with all PAC-Bayes bounds [23], our bounds can exceed 1 (and thus be vacuous) for very small k or large A. The kl-inverse bound [28] can tighten the constant-factor regime but does not change the rate; we use the simpler McAllester form throughout.

4.

Computability of A and S: Under the linear attention model, A and S are exactly computable. For general LLMs, $A={\mathbb{E}}_x\left[H\left(p_{\theta }\left(y\right|x,⌀)\right)\right]$ requires integration over the (unknown) input distribution; in practice, A and S must be estimated from a held-out sample, introducing estimation error not captured by the bound.

5.

Linear attention model: While providing clean closed-form results, the linear attention model omits softmax nonlinearity, multi-head interaction, and positional encoding effects.

6.

i.i.d. demonstrations: The framework assumes demonstrations are i.i.d., which may not hold in conversational or curriculum-based ICL settings.

8.5. Open Questions

1.

Can the KL linear reduction be extended to softmax attention via local linearization around the softmax operating point?

2.

How do multi-task ICL bounds look when tasks share ambiguity and saliency structure?

3.

Can the permutation variance bound (Theorem 6) be tightened using task-specific ordering heuristics?

4.

Does a data-dependent saliency $S\left(k\right)$ that decays with k lead to tighter bounds than the constant-S assumption?

5.

What are the computational-statistical tradeoffs when the attention capacity limits the effective number of demonstrations the model can process?

9. Conclusions

We have developed a comprehensive PAC-Bayes framework for in-context learning, comprising 28 formal results (20 theorems, 1 proposition, 5 lemmas, and 2 corollaries) across six directions: core generalization bounds, architecture-dependent instantiation under linear attention, minimax lower bounds establishing rate optimality, Catoni fast-rate bounds, data-dependent priors via sample splitting, and Bernstein variance-adaptive bounds.

The framework is parameterized by two computable quantities—ambiguity A and saliency S—that together determine the ICL generalization rate $\Theta \left(\sqrt{A/k}\right)$, the saturation point $k^{*}=\lceil A/S\rceil$, and the sample complexity $O(A/{\epsilon }^2)$. Under the linear attention model, these quantities acquire closed-form expressions in the model architecture parameters, yielding fully explicit, computable bounds.

Our results provide a unified theoretical explanation for key empirical observations about ICL: why different tasks require different numbers of demonstrations, why performance improves rapidly and then saturates, and why well-pretrained models are more sample-efficient. Experiments with GPT-2 on SST-2, AG News, and SNLI confirm that the PAC-Bayes bounds hold for real softmax Transformers, that the measured KL divergence deviates from the linear model but the framework remains valid with empirical KL values, and that the $(A,S)$ parameterization provides meaningful diagnostic information across tasks. The framework opens several directions for future work, including scaling the empirical validation to larger models, extensions to multi-task settings, and non-i.i.d. demonstration sequences.

Appendix A.1. Proof of Theorem 2 (Basic ICL Bound)

Proof. 

By Theorem 1 (McAllester), with probability at least $1-\delta$:

$$R_{\mathrm{pop}}\le R_{\mathrm{emp}}+\sqrt{{\displaystyle \frac{KL(Q\parallel P)+log(1/\delta )}{2k}}}.$$

Since $KL(Q\parallel P)\le A$ and $\sqrt{·}$ is monotonically increasing, we have

$$\sqrt{{\displaystyle \frac{KL(Q\parallel P)+log(1/\delta )}{2k}}}\le \sqrt{{\displaystyle \frac{A+log(1/\delta )}{2k}}}.$$

Combining:

$$R_{\mathrm{pop}}\le R_{\mathrm{emp}}+\sqrt{{\displaystyle \frac{A+log(1/\delta )}{2k}}}.$$

   □

Appendix A.2. Proof of Theorem 3 (Saliency-Enhanced Bound)

Proof. 

By Assumption A1, $KL(Q(·\mid D_k)\parallel P)\le max(A-kS,0)$. Apply Theorem 1:

$$R_{\mathrm{pop}}\le R_{\mathrm{emp}}+\sqrt{{\displaystyle \frac{KL(Q\parallel P)+log(1/\delta )}{2k}}}\le R_{\mathrm{emp}}+\sqrt{{\displaystyle \frac{max(A-kS,0)+log(1/\delta )}{2k}}},$$

where the second inequality uses monotonicity of $\sqrt{·}$ and $KL\le max(A-kS,0)$.    □

Appendix A.3. Proof of Theorem 4 (Sample Complexity)

Proof. 

Let $C=A+log(1/\delta )$ and $D=\epsilon -{\epsilon }_{\mathrm{prior}}>0$. From the hypothesis $k\ge C/\left(2D^2\right)$, we derive:

$$\begin{array}{cc}\hfill k\ge {\displaystyle \frac{C}{2D^2}}& \Rightarrow 2k{D}^2\ge C\Rightarrow {\displaystyle \frac{C}{2k}}\le D^2.\hfill \end{array}$$

(A1)

Since $\sqrt{·}$ is monotonically increasing and both sides of (A1) are non-negative:

$$\sqrt{{\displaystyle \frac{C}{2k}}}\le \sqrt{D^2}=D=\epsilon -{\epsilon }_{\mathrm{prior}}.$$

By Theorem 2 and $R_{\mathrm{emp}}\le {\epsilon }_{\mathrm{prior}}$:

$$R_{\mathrm{pop}}\le R_{\mathrm{emp}}+\sqrt{{\displaystyle \frac{C}{2k}}}\le {\epsilon }_{\mathrm{prior}}+(\epsilon -{\epsilon }_{\mathrm{prior}})=\epsilon .$$

   □

Appendix A.4. Proof of Theorem 5 (ICL Lower Bound)

Proof. 

By Fano’s inequality [19], for any estimator of a discrete random variable Y with $\left|\mathcal{Y}\right|=m$ observed through channel X:

$$P_e\ge {\displaystyle \frac{H\left(Y\right)-I(X;Y)-log2}{logm}}.$$

Substituting $H\left(Y\right)\ge H_{min}$ and $I(X;Y)\le kS$:

$$P_e\ge {\displaystyle \frac{H_{min}-kS-log2}{logm}}.$$

The inequality $logm>0$ (since $m>1$) ensures the division is valid and preserves the direction.    □

Appendix A.5. Proof of Theorem 6 (Permutation Variance)

Proof. 

The Efron–Stein inequality [20] applied to the loss function of a random permutation gives:

$${Var}_{\sigma }\left[\epsilon (k,\sigma )\right]\le {\displaystyle \frac{k·c^2}{4}},$$

where c is the bounded-differences constant (maximum change from a single transposition). Substituting the hypothesis $c^2\le 4S/k^2$:

$${Var}_{\sigma }\left[\epsilon (k,\sigma )\right]\le {\displaystyle \frac{k·4S/k^2}{4}}={\displaystyle \frac{S}{k}}.$$

   □

Appendix B.1. Proofs of Theorems 1 and 2

Proof 

(Proof of Theorem 1). Using the classical inequality $log(1+x)\le x$ for $x\ge 0$ with $x={\sigma }_{\mathrm{prior}}^2/{\sigma }_{\mathrm{noise}}^2$:

$$A_{\mathrm{eff}}={\displaystyle \frac{d}{2}}log\left(1+{\displaystyle \frac{{\sigma }_{\mathrm{prior}}^2}{{\sigma }_{\mathrm{noise}}^2}}\right)\le {\displaystyle \frac{d}{2}}·{\displaystyle \frac{{\sigma }_{\mathrm{prior}}^2}{{\sigma }_{\mathrm{noise}}^2}}={\displaystyle \frac{d{\sigma }_{\mathrm{prior}}^2}{2{\sigma }_{\mathrm{noise}}^2}}.$$

   □

Proof 

(Proof of Theorem 2). Since $s\le {\sigma }_{\mathrm{prior}}^2$, we have $s/{\sigma }_{\mathrm{prior}}^2\le 1$. Therefore:

$$S_{\mathrm{eff}}={\displaystyle \frac{d·s}{2{\sigma }_{\mathrm{prior}}^2}}={\displaystyle \frac{d}{2}}·{\displaystyle \frac{s}{{\sigma }_{\mathrm{prior}}^2}}\le {\displaystyle \frac{d}{2}}.$$

   □

Appendix B.2. Proof of Proposition 1 and Corollary 1

Proof 

(Proof of Proposition 1). Under the Bayesian linear regression model, the posterior after k observations $(X,\mathbf{y})$ with $X\in {\mathbb{R}}^{k\times d}$ is:

$$w\mid X,\mathbf{y}\sim \mathcal{N}({\widehat{w}}_k,{\Sigma }_k),{\Sigma }_k={\left({\displaystyle \frac{1}{{\sigma }_p^2}}{I}_d+{\displaystyle \frac{1}{{\sigma }_n^2}}{X}^{\top }X\right)}^{-1}.$$

Using the Gaussian KL closed form (Appendix E, item 7) and taking expectations over $D_k$, the key step is the posterior covariance bound. Since $\mathbb{E}\left[X^{\top }X\right]=k{I}_d$ and the effective signal strength is s per example, we obtain the deterministic bound ${\Sigma }_k⪯{({\sigma }_p^{-2}I+ks{\sigma }_n^{-2}I)}^{-1}$. The Gaussian KL with diagonal covariances yields:

$${\mathbb{E}}_{D_k}\left[KL(Q_k\parallel P)\right]\le {\displaystyle \frac{d}{2}}\left[{\displaystyle {\sigma }_p^{-2}·{\sigma }_p^2{\sigma }_n^2/({\sigma }_n^2+ks{\sigma }_p^2)-1+log\left(\frac{{\sigma }_n^2+ks{\sigma }_p^2}{{\sigma }_n^2}\right)}\right]\le {\displaystyle \frac{d}{2}}log\left(1+{\displaystyle \frac{{\sigma }_p^2}{{\sigma }_n^2+ks}}\right),$$

where the last inequality uses $tr({\Sigma }_k/{\sigma }_p^2)-d+{\parallel {\widehat{w}}_k\parallel }^2/{\sigma }_p^2\le log(det\left({\sigma }_p^{2}I\right)/det\left({\Sigma }_k\right))$ after taking expectations and applying Jensen’s inequality. Properties (a)–(c) follow directly: (a) monotonicity since ${\sigma }_p^2/({\sigma }_n^2+ks)$ is decreasing in k; (b) $\overline{A}\left(0\right)=(d/2)log(1+{\sigma }_p^2/{\sigma }_n^2)=A_{\mathrm{eff}}$; (c) $\overline{A}\left(k\right)\to 0$ as $k\to \infty$.    □

Proof 

(Proof of Corollary 1). The first inequality $\overline{A}\left(k\right)\le A_{\mathrm{eff}}$ is immediate from monotonicity (property (a)). For the linearization error, note $f\left(k\right):=\overline{A}\left(k\right)$ satisfies $f\left(0\right)=A_{\mathrm{eff}}$ and $f^{\prime }\left(0\right)=-S_{\mathrm{eff}}$. By Taylor’s theorem with integral remainder: $|f\left(k\right)-(A_{\mathrm{eff}}-k{S}_{\mathrm{eff}})|\le (k^2/2){sup}_{t\in [0,k]}\left|f^{\prime \prime }\left(t\right)\right|$. Since $f^{\prime \prime }\left(t\right)=d{s}^2/\left(2{({\sigma }_n^2+ts)}^2\right)\le d{s}^2/\left(2{\sigma }_n^2({\sigma }_n^2+ks)\right)$, the stated bound follows.    □

Appendix B.3. Proofs of Theorems 3–7

Proof 

(Proof of Theorem 3). By definition of max, if $A_{\mathrm{eff}}-k{S}_{\mathrm{eff}}\le 0$ (i.e., $k{S}_{\mathrm{eff}}\ge A_{\mathrm{eff}}$), then $max(A_{\mathrm{eff}}-k{S}_{\mathrm{eff}},0)=0$.    □

Proof 

(Proof of Theorem 7). By Theorem 3, the KL bound equals zero. Apply Theorem 1 with $KL(Q\parallel P)=0$:

$$R_{\mathrm{pop}}\le R_{\mathrm{emp}}+\sqrt{{\displaystyle \frac{0+log(1/\delta )}{2k}}}=R_{\mathrm{emp}}+\sqrt{{\displaystyle \frac{log(1/\delta )}{2k}}}.$$

   □

Appendix B.4. Proof of Theorem 8 (Linear Attention ICL Bound)

Proof. 

The Bayesian conjugacy structure (Appendix B.2) implies that the posterior KL satisfies Assumption A1 with $A=A_{\mathrm{eff}}$ and $S=S_{\mathrm{eff}}$. Apply Theorem 3 with these values to obtain:

$$R_{\mathrm{pop}}\le R_{\mathrm{emp}}+\sqrt{{\displaystyle \frac{max(A_{\mathrm{eff}}-k{S}_{\mathrm{eff}},0)+log(1/\delta )}{2k}}}.$$

   □

Appendix B.5. Proof of Theorem 9

Proof. 

By Theorem 1, $A_{\mathrm{eff}}\le d{\sigma }_{\mathrm{prior}}^2/\left(2{\sigma }_{\mathrm{noise}}^2\right)$. Since $\sqrt{·}$ is monotone:

$$\sqrt{{\displaystyle \frac{A_{\mathrm{eff}}+log(1/\delta )}{2k}}}\le \sqrt{{\displaystyle \frac{d{\sigma }_{\mathrm{prior}}^2/\left(2{\sigma }_{\mathrm{noise}}^2\right)+log(1/\delta )}{2k}}}.$$

The condition (12) ensures this quantity is at most $\epsilon -{\epsilon }_{\mathrm{prior}}$. Apply Theorem 4 to conclude $R_{\mathrm{pop}}\le \epsilon$.    □

Appendix C.1. Proof of Theorem 10 (Minimax Lower Bound)

Proof. 

We use Le Cam’s two-point method. Construct two ICL tasks ${\mathcal{T}}_0,{\mathcal{T}}_1$ satisfying:

1.

Risk gap: $\Delta =\sqrt{A/k}/2$ (the difference in optimal risk between the two tasks).

2.

Per-sample KL divergence: $\kappa =A/\left(2k\right)$.

By Le Cam’s inequality [22], the maximum risk of any estimator over $\{{\mathcal{T}}_0,{\mathcal{T}}_1\}$ satisfies:

$$R_{max}\ge {\displaystyle \frac{\Delta }{2}}\left(1-\sqrt{{\displaystyle \frac{KL(P_0^k\parallel P_1^k)}{2}}}\right).$$

By the KL tensorization property (for i.i.d. observations):

$$KL(P_0^k\parallel P_1^k)=k·\kappa =k·{\displaystyle \frac{A}{2k}}={\displaystyle \frac{A}{2}}.$$

Substituting:

$$\begin{array}{cc}\hfill R_{max}& \ge {\displaystyle \frac{\sqrt{A/k}/2}{2}}\left(1-\sqrt{{\displaystyle \frac{A/2}{2}}}\right)={\displaystyle \frac{\sqrt{A/k}}{4}}\left(1-\sqrt{{\displaystyle \frac{A}{4}}}\right).\hfill \end{array}$$

   □

Appendix C.2. Proof of Theorem 11

Proof. 

When $A\le 1$: $A/4\le 1/4$, so $\sqrt{A/4}\le \sqrt{1/4}=1/2$, hence $1-\sqrt{A/4}\ge 1/2$. From Theorem 10:

$$R_{max}\ge {\displaystyle \frac{\sqrt{A/k}}{4}}·{\displaystyle \frac{1}{2}}={\displaystyle \frac{\sqrt{A/k}}{8}}.$$

   □

Appendix C.3. Proof of Theorem 14 (Catoni ICL Bound)

Proof. 

Catoni’s PAC-Bayes bound [13] with temperature $\lambda \in (0,2)$ states:

$$R_{\mathrm{pop}}\le {\displaystyle \frac{1}{1-\lambda /2}}\left(R_{\mathrm{emp}}+{\displaystyle \frac{KL(Q\parallel P)+log(2\sqrt{k}/\delta )}{\lambda k}}\right).$$

Setting $\lambda =1$:

• Coefficient: $1/(1-1/2)=2$.
• Denominator: $1·k=k$.

Weakening $KL(Q\parallel P)\le A$:

$${\displaystyle \frac{KL(Q\parallel P)+log(2\sqrt{k}/\delta )}{k}}\le {\displaystyle \frac{A+log(2\sqrt{k}/\delta )}{k}}.$$

Therefore:

$$R_{\mathrm{pop}}\le 2\left(R_{\mathrm{emp}}+{\displaystyle \frac{A+log(2\sqrt{k}/\delta )}{k}}\right).$$

   □

Appendix C.4. Proof of Theorem 15

Proof. 

Let $C=A+log(2\sqrt{k}/\delta )$. By hypothesis, $R_{\mathrm{emp}}\le \epsilon /4$ and $C/k\le \epsilon /4$. From Theorem 14:

$$R_{\mathrm{pop}}\le 2(R_{\mathrm{emp}}+C/k)\le 2(\epsilon /4+\epsilon /4)=2·\epsilon /2=\epsilon .$$

   □

Appendix C.5. Proof of Theorem 4

Proof. 

We need to show $2C/k\le \sqrt{C/\left(2k\right)}$ when $k\ge 8C$. Squaring both sides (both are non-negative since $C\ge 0$ and $k>0$):

$${\left({\displaystyle \frac{2C}{k}}\right)}^2={\displaystyle \frac{4C^2}{k^2}}\mathrm{vs}.{\displaystyle \frac{C}{2k}}.$$

The inequality $4C^2/k^2\le C/\left(2k\right)$ is equivalent to $8C^2\le Ck$, i.e., $8C\le k$. This holds by hypothesis.    □

Appendix D.1. Proofs for Data-Dependent Priors

Proof 

(Proof of Theorem 17). Each of the $k_1$ demonstrations contributes $\mathrm{ig}$ nats toward resolving the task. The remaining KL divergence is at most $A-k_1·\mathrm{ig}$. Since KL divergence is non-negative:

$$KL(Q\parallel P_{\mathrm{informed}})\le A-k_1·\mathrm{ig}\le max(A-k_1·\mathrm{ig},0).$$

   □

Proof 

(Proof of Theorem 18). Apply the split-sample PAC-Bayes bound [16] on the $k_2$ evaluation examples with the data-dependent prior $P_{\mathrm{informed}}$:

$$R_{\mathrm{pop}}\le R_{\mathrm{emp}}\left(k_2\right)+\sqrt{{\displaystyle \frac{KL(Q\parallel P_{\mathrm{informed}})+log(1/\delta )}{2k_2}}}.$$

By Theorem 17, $KL(Q\parallel P_{\mathrm{informed}})\le max(A-k_1·\mathrm{ig},0)$. Monotonicity of $\sqrt{·}$:

$$\begin{array}{cc}\hfill & \sqrt{{\displaystyle \frac{KL(Q\parallel P_{\mathrm{informed}})+log(1/\delta )}{2k_2}}}\hfill \\ \hfill & \le \sqrt{{\displaystyle \frac{max(A-k_1·\mathrm{ig},0)+log(1/\delta )}{2k_2}}}.\hfill \end{array}$$

   □

Proof 

(Proof of Theorem 19). When $k_1·\mathrm{ig}\ge A$, we have $A-k_1·\mathrm{ig}\le 0$, so $max(A-k_1·\mathrm{ig},0)=0$. Substituting into Theorem 18:

$$R_{\mathrm{pop}}\le R_{\mathrm{emp}}\left(k_2\right)+\sqrt{{\displaystyle \frac{0+log(1/\delta )}{2k_2}}}=R_{\mathrm{emp}}\left(k_2\right)+\sqrt{{\displaystyle \frac{log(1/\delta )}{2k_2}}}.$$

   □

Appendix D.2. Proofs for Bernstein Variance-Adaptive Bounds

Proof 

(Proof of Theorem 20). The Bernstein PAC-Bayes bound [15] states:

$$R_{\mathrm{pop}}\le R_{\mathrm{emp}}+\sqrt{{\displaystyle \frac{2V(KL+C_{\delta })}{k}}}+{\displaystyle \frac{7(KL+C_{\delta })}{3k}},$$

where $C_{\delta }=log(2\sqrt{k}/\delta )$. Since $KL(Q\parallel P)\le A$, both the $\sqrt{·}$ term and the linear term are monotonically increasing in KL. Therefore:

$$\begin{array}{cc}\hfill \sqrt{{\displaystyle \frac{2V(KL+C_{\delta })}{k}}}& \le \sqrt{{\displaystyle \frac{2V(A+C_{\delta })}{k}}},\hfill \\ \hfill {\displaystyle \frac{7(KL+C_{\delta })}{3k}}& \le {\displaystyle \frac{7(A+C_{\delta })}{3k}}.\hfill \end{array}$$

Combining with the Bernstein bound yields (21).    □

Proof 

(Proof of Theorem 21). Let $C=A+C_{\delta }$. Under $V\le V_0/k$, the numerator of the variance term becomes:

$$2V·C\le 2·{\displaystyle \frac{V_0}{k}}·C={\displaystyle \frac{2V_{0}C}{k}}.$$

Dividing by k (the denominator inside $\sqrt{·}$ from Theorem 20):

$${\displaystyle \frac{2V·C}{k}}\le {\displaystyle \frac{2V_{0}C}{k^2}}.$$

By $\sqrt{·}$ monotonicity:

$$\sqrt{{\displaystyle \frac{2VC}{k}}}\le \sqrt{{\displaystyle \frac{2V_{0}C}{k^2}}}={\displaystyle \frac{\sqrt{2V_{0}C}}{k}}.$$

Both the variance term $\sqrt{2V_{0}C}/k$ and the remainder term $7C/\left(3k\right)$ are $O(1/k)$.    □

Proof 

(Proof of Theorem 5). We show $2VC/k\le {(C/k)}^2$, which implies $\sqrt{2VC/k}\le C/k$ by $\sqrt{·}$ monotonicity.

Since $V\le C/\left(2k\right)$:

$$2VC/k\le 2·{\displaystyle \frac{C}{2k}}·{\displaystyle \frac{C}{k}}={\displaystyle \frac{C^2}{k^2}}={\left({\displaystyle \frac{C}{k}}\right)}^2.$$

Taking square roots of both sides (both are non-negative):

$$\sqrt{{\displaystyle \frac{2VC}{k}}}\le {\displaystyle \frac{C}{k}}.$$

   □

Appendix F.1. Pseudocode

Algorithm 1:Bayesian Linear Regression ICL Simulation

Require: Dimension d, prior variance ${\sigma }_p^2$, noise variance ${\sigma }_n^2$, signal strength s, max demonstrations $k_{max}$, trials N, confidence $\delta$ 1: Compute $A_{\mathrm{eff}}={\displaystyle \frac{d}{2}}log(1+{\sigma }_p^2/{\sigma }_n^2)$, $S_{\mathrm{eff}}=ds/\left(2{\sigma }_p^2\right)$, $k^{*}=\lceil A_{\mathrm{eff}}/S_{\mathrm{eff}}\rceil$ 2: for$k=1,\dots ,k_{max}$do 3:     for trial $j=1,\dots ,N$ do 4:         Sample $w\sim \mathcal{N}(0,{\sigma }_p^2{I}_d)$ 5:         Sample $X\in {\mathbb{R}}^{k\times d}$ with rows $x_i\sim \mathcal{N}(0,I_d)$ 6:         Compute $\mathbf{y}=Xw+ϵ$, ${ϵ}_i\sim \mathcal{N}(0,{\sigma }_n^2)$ 7:         Posterior: ${\Sigma }_k={({\sigma }_p^{-2}I+{\sigma }_n^{-2}{X}^{\top }X)}^{-1}$, ${\widehat{w}}_k={\sigma }_n^{-2}{\Sigma }_k{X}^{\top }\mathbf{y}$ 8:         Training loss: ${\widehat{R_{\mathrm{emp}}}}^{\left(j\right)}=k^{-1}{\sum }_{i=1}^{k}min({({\widehat{w}}_k^{\top }{x}_i-y_i)}^2,1)$ 9:         Sample test: $x_q\sim \mathcal{N}(0,I_d)$, $y_q=w^{\top }{x}_q+{ϵ}_q$ 10:         Test loss: ${\widehat{R_{\mathrm{pop}}}}^{\left(j\right)}=min({({\widehat{w}}_k^{\top }{x}_q-y_q)}^2,1)$ 11:     end for 12:     $\widehat{R_{\mathrm{pop}}}\left(k\right)\leftarrow N^{-1}{\sum }_j{\widehat{R_{\mathrm{pop}}}}^{\left(j\right)}$,    $\widehat{R_{\mathrm{emp}}}\left(k\right)\leftarrow N^{-1}{\sum }_j{\widehat{R_{\mathrm{emp}}}}^{\left(j\right)}$ 13:     Bound gap $\leftarrow \sqrt{(max(A_{\mathrm{eff}}-k{S}_{\mathrm{eff}},0)+log(1/\delta ))/\left(2k\right)}$ 14: end for

Appendix F.2. Hyperparameters

Table A1. Simulation hyperparameters.

Parameter	Value
Dimension d	$\{5,10\}$
Prior variance ${\sigma }_{\mathrm{prior}}^2$	1.0
Noise variance ${\sigma }_{\mathrm{noise}}^2$	$\{0.5,1.0\}$
Signal strength s	$\{0.10,0.15\}$
Max demonstrations $k_{max}$	200 (Exp. 2), 50 (others)
Number of trials N	2000
Confidence parameter $\delta$	0.05
Random seed	42

Table A1. Simulation hyperparameters.

Parameter	Value
Dimension d	$\{5,10\}$
Prior variance ${\sigma }_{\mathrm{prior}}^2$	1.0
Noise variance ${\sigma }_{\mathrm{noise}}^2$	$\{0.5,1.0\}$
Signal strength s	$\{0.10,0.15\}$
Max demonstrations $k_{max}$	200 (Exp. 2), 50 (others)
Number of trials N	2000
Confidence parameter $\delta$	0.05
Random seed	42

Appendix F.3. Reproducibility

All simulations are implemented in Python 3.13 using NumPy 2.3. The complete simulation script (run_simulations.py) reproduces all tables in Section 7 with a single execution. Computation time is approximately 10 minutes on a standard workstation.