Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization

1. Introduction

Transformer architectures [18] have become foundational in numerous areas of machine learning, including natural language processing (NLP), computer vision, and time-series modeling. By eschewing recurrence and convolution in favor of a self-attention mechanism, transformers achieve superior parallelism and scaling properties. However, the lack of inherent sequential processing necessitates the incorporation of positional encodings (PEs) to inject information about the order of tokens. The original transformer paper introduced sinusoidal positional encodings, and subsequent work explored learned absolute encodings [18], relative encodings [14], and bias-based schemes such as Attention with Linear Biases (ALiBi) [12]. Despite widespread empirical adoption, the theoretical understanding of how PEs influence a transformer’s expressiveness, generalization, and extrapolation capabilities is limited.

This paper proposes a thorough theoretical analysis of positional encodings, addressing the following overarching questions:

• Expressiveness: What classes of sequence-to-sequence functions can a transformer approximate under different PE schemes? Are there inherent limitations imposed by specific encodings?
• Generalization: How do PEs affect a transformer’s ability to generalize from training to unseen data, especially when sequence lengths vary? Can we derive generalization bounds that capture the influence of different PEs?
• Extrapolation: Why do certain encoding methods (e.g., ALiBi) facilitate extrapolation to longer sequences? Can we formalize this phenomenon and propose new encodings that further enhance extrapolation?
• Novel Encodings: Are there theoretically motivated PE schemes—based on wavelet transforms, Legendre polynomials, or information-theoretic principles—that surpass existing methods in expressiveness, generalization, or extrapolation?

By developing a unified mathematical framework to answer these questions, we aim to provide deep insights into why PEs work, how they can be improved, and what limitations current methods entail. We also propose new PE schemes grounded in orthogonal function families (e.g., wavelets, Legendre polynomials) and analyze their theoretical properties. Lightweight experiments on small-scale synthetic tasks corroborate our theoretical findings, demonstrating that wavelet-based encodings can yield superior extrapolation performance compared to standard sinusoidal PEs.

• Contributions. Our main contributions are:

• Expressiveness Characterization: We formally define the expressiveness of transformer models under different PE schemes, showing how absolute, relative, and bias-based encodings impact the set of sequence-to-sequence functions that can be approximated.
• Generalization Bounds: Using tools from statistical learning theory (Rademacher complexity, covering numbers), we derive generalization bounds that explicate the role of PEs in controlling model capacity and overfitting for varying sequence lengths.
• Extrapolation Analysis: We extend the theoretical understanding of ALiBi’s biasing mechanism, provide a unified extrapolation framework for bias-based PEs, and identify the limits of extrapolation for alternative encoding schemes.
• Novel PE Schemes: We propose several novel positional encodings based on orthogonal functions (e.g., wavelets, Legendre polynomials) and information-theoretic criteria (maximizing mutual information between positions). We analyze their expressiveness, generalization, and extrapolation properties.
• Lightweight Validation: We implement the proposed PE schemes in pure NumPy to run small-scale experiments on synthetic tasks designed to test extrapolation and generalization. These confirm our theoretical predictions without requiring GPUs.

The rest of this paper is structured as follows. Section 2 reviews existing PE methods. Section 3 defines and analyzes expressiveness under different PEs. Section 4 derives generalization bounds. Section 5 provides a theoretical framework for extrapolation. Section 6 introduces novel PE schemes and analyzes their properties. Section 7 details lightweight experimental validation. Finally, Section 8 discusses implications, limitations, and future work.

2. Background and Related Work

This section reviews transformer architectures and existing positional encoding methods, emphasizing their empirical performance and the lack of deep theoretical analysis.

2.1. Transformer Architecture

The transformer model [18] processes an input sequence of length N by first mapping each token to a $d_{\mathrm{model}}$-dimensional embedding. Denote the input embeddings by $\mathbf{X}={[{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_N]}^{\top }\in {\mathbb{R}}^{N\times d_{\mathrm{model}}}$. Because the transformer is permutation-invariant, positional encodings $\mathbf{P}={[{\mathbf{p}}_1,{\mathbf{p}}_2,\dots ,{\mathbf{p}}_N]}^{\top }\in {\mathbb{R}}^{N\times d_{\mathrm{model}}}$ are added to $\mathbf{X}$ so that the model can utilize order information:

$${\mathbf{Z}}^{\left(0\right)}=\mathbf{X}+\mathbf{P},{\mathbf{Z}}^{\left(0\right)}\in {\mathbb{R}}^{N\times d_{\mathrm{model}}}.$$

The transformer then applies L layers of multi-head self-attention and position-wise feed-forward networks to compute contextualized representations ${\mathbf{Z}}^{\left(L\right)}\in {\mathbb{R}}^{N\times d_{\mathrm{model}}}$. We omit further architectural details; see [18] for a complete description.

2.2. Sinusoidal and Learned Positional Encodings

In the original transformer, [18] introduced sinusoidal absolute positional encodings defined elementwise as

$$\begin{array}{cc}\hfill \mathrm{PE}(pos,2i)& =sin\left({\displaystyle \frac{pos}{{10000}^{\frac{2i}{d_{\mathrm{model}}}}}}\right),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill \mathrm{PE}(pos,2i+1)& =cos\left({\displaystyle \frac{pos}{{10000}^{\frac{2i}{d_{\mathrm{model}}}}}}\right),\hfill \end{array}$$

(2)

where $pos\in \{0,1,\dots ,N-1\}$ and $i\in \{0,1,\dots ,{\displaystyle \frac{d_{\mathrm{model}}}{2}}-1\}$. These encodings allow the transformer to learn to attend to relative positions because any linear combination of two sinusoidal vectors encodes relative shifts. The paper also considered learned positional encodings, where a trainable embedding matrix $\mathbf{P}\in {\mathbb{R}}^{N\times d_{\mathrm{model}}}$ is optimized jointly with the model. Empirically, learned and sinusoidal encodings perform similarly on standard benchmarks.

2.3. Relative Positional Encodings

Relative positional encodings incorporate the distance between pairs of tokens into the attention mechanism, rather than injecting absolute position vectors. [14] add a learnable embedding ${\mathbf{R}}_{i-j}$ to the attention logits whenever computing attention between tokens at positions i and j. Such encodings can reduce the reliance on absolute position signals and improve performance on tasks with long-range dependencies. Alternative relative schemes include [6] and [13].

2.4. Attention with Linear Biases (ALiBi)

[12] introduced ALiBi, which adds a bias term to attention logits proportional to the negative distance between tokens:

$$b(i,j)=-\alpha |i-j|,$$

(3)

where $\alpha >0$ is a slope hyperparameter. This bias encourages the model to pay more attention to nearby tokens while still preserving a form of positional information. Crucially, ALiBi enables extrapolation to sequences longer than those used during training because the bias formula applies uniformly to any token distance. Empirical results show improved performance on long-context tasks without modifying the model’s architecture or retraining.

2.5. Other Positional Encoding Variants

Several other PE schemes have been proposed:

• Rotary Positional Embeddings (RoPE) [15]: Applies a rotation in embedding space to encode relative positions.
• Fourier Feature Encodings [16]: Use random Fourier features to encode continuous positions, often applied in continuous-time transformers.
• Convolutional Encodings [10]: Integrate convolutional layers to encode local positional information.
• Wavelet-based and Polynomial-based Encodings: Proposed informally in blog posts [9], but not systematically analyzed.

Despite numerous empirical variants, a cohesive theoretical framework for comparing expressiveness, generalization, and extrapolation across these methods is lacking. We address this gap in Section 3, Section 4, Section 5 and Section 6.

3. Expressiveness of Transformers with Positional Encodings

Expressiveness refers to a model’s capacity to approximate a broad class of functions. For transformers, we focus on the class of sequence-to-sequence mappings $f:{\mathcal{X}}^N\to {\mathcal{Y}}^N$, where $\mathcal{X},\mathcal{Y}\subset {\mathbb{R}}^d$. We analyze how different PE schemes restrict or enlarge the set of approximable functions.

3.1. Formal Definition of Expressiveness

Let ${\mathcal{F}}_{\mathrm{PE}}^{L,H}$ denote the class of functions implementable by an L-layer transformer with H attention heads and positional encoding scheme “PE”. We say the transformer with encoding scheme PE is universal if ${\mathcal{F}}_{\mathrm{PE}}^{L,H}$ is dense (in a suitable norm) in a target function space $\mathcal{G}$ (e.g., all continuous sequence-to-sequence mappings) as $L,H\to \infty$. Our analysis focuses on:

• Absolute encodings: Sinusoidal vs. learned.
• Relative encodings: Shaw et al. vs. ALiBi.
• New encodings: Wavelet-based, polynomial-based.

We examine two questions:

(a)

For fixed $L,H$, how does the choice of PE affect the size of ${\mathcal{F}}_{\mathrm{PE}}^{L,H}$?

(b)

Do all PE schemes yield universal approximation as $L,H\to \infty$?

3.2. Expressiveness with Sinusoidal Encodings

Sinusoidal encodings embed absolute position into each token representation via Eq. (1)–(2). To analyze expressiveness, note that self-attention computes queries, keys, and values as affine transformations of embedded inputs. Let ${\mathbf{Z}}_i^{\left(0\right)}={\mathbf{x}}_i+\mathrm{PE}\left(i\right)$. In the first layer, an attention head computes

$$\mathrm{Attention}{\left({\mathbf{Z}}^{\left(0\right)}\right)}_i=\sum _{j=1}^N{\displaystyle \frac{exp\left(\left({\mathbf{z}}_i^{\left(0\right)}{W}_Q\right){\left({\mathbf{z}}_j^{\left(0\right)}{W}_K\right)}^{\top }/\sqrt{d_k}\right)}{{\sum }_{k=1}^{N}exp\left(\left({\mathbf{z}}_i^{\left(0\right)}{W}_Q\right){\left({\mathbf{z}}_k^{\left(0\right)}{W}_K\right)}^{\top }/\sqrt{d_k}\right)}}\left({\mathbf{z}}_j^{\left(0\right)}{W}_V\right),$$

(4)

where $W_Q,W_K,W_V\in {\mathbb{R}}^{d_{\mathrm{model}}\times d_k}$. Because $\mathrm{PE}\left(i\right)$ is deterministic and injective in i, the softmax attention weights can, in principle, implement functions that depend on absolute positions, relative distances, or higher-order interactions across multiple tokens.

• Universal Approximation. [21] show that a transformer with sinusoidal encodings can approximate any sequence-to-sequence mapping to arbitrary accuracy given sufficient width (number of heads) and depth (number of layers). Intuitively, the sinusoidal basis of period ${10000}^{2i/d_{\mathrm{model}}}$ is rich enough to encode any discrete position $i\in \{0,\dots ,N-1\}$ uniquely. Since a sufficiently wide transformer can compute arbitrary Boolean functions of its inputs [21], adding unique position signals ensures universal approximation over sequences of length up to N.

• Limitations. However, the expressiveness claim is restricted to sequences of length at most $N_{max}$ (the maximum position for which PEs were computed). When confronted with longer sequences, sinusoidal encodings repeat after $2\pi$ in each frequency dimension, potentially causing ambiguity in absolute positions if not correctly handled. Consequently, universal approximation holds for fixed-length tasks but not for unlimited-length extrapolation.

3.3. Expressiveness with Learned Absolute Encodings

Learned absolute encodings use a trainable matrix $\mathbf{P}\in {\mathbb{R}}^{N_{max}\times d_{\mathrm{model}}}$. The encoder learns to map each position $i\le N_{max}$ to a vector ${\mathbf{p}}_i$. If $N_{max}$ matches the maximum training sequence length, the transformer can, in principle, use distinct vectors for each position. The universality argument parallels the sinusoidal case: unique position embeddings plus sufficiently powerful attention/FFN layers yield universal approximation for sequences of length up to $N_{max}$. However, learned encodings cannot generalize to $i>N_{max}$, as no embedding exists beyond the training range. Thus, expressiveness is limited to the training sequence length.

3.4. Expressiveness with Relative Positional Encodings

Relative encodings [14] circumvent the need for distinct position vectors by encoding only distances. In the Shaw et al. formulation, the attention score between positions i and j includes a term ${\mathbf{a}}^{\top }{\mathbf{R}}_{i-j}$, where ${\mathbf{R}}_k$ is a learnable vector for relative distance k. Formally,

$$\mathrm{score}(i,j)=\left({\mathbf{z}}_i{W}_Q\right){\left({\mathbf{z}}_j{W}_K\right)}^{\top }+{\mathbf{a}}^{\top }{\mathbf{R}}_{i-j}.$$

(5)

Since relative distances $i-j$ range in $[-(N-1),N-1]$, the model can learn to attend based purely on relative positions, which often suffices for tasks where absolute position is irrelevant (e.g., certain translation or summarization tasks).

• Universality and Limitations. The work of [20] demonstrates that relative encodings can be as expressive as absolute encodings for tasks that depend on relative structure. However, if a task requires absolute position information—e.g., “assign label $o_i$ if the token at position i is within the first quarter of the sequence”—a purely relative encoding may struggle. In such cases, an absolute offset or additional global positional token is needed. Nevertheless, relative encodings often yield equal or better performance on many benchmarks, suggesting that many tasks rely more on relative positions.

3.5. Expressiveness with ALiBi

ALiBi encodes relative distance via a linear bias to attention scores (Eq. 3). That is,

$$\mathrm{score}(i,j)=\left({\mathbf{z}}_i{W}_Q\right){\left({\mathbf{z}}_j{W}_K\right)}^{\top }+b(i,j),b(i,j)=-\alpha |i-j|.$$

(6)

Because $b(i,j)$ depends only on $|i-j|$, ALiBi is, strictly speaking, a relative encoding. However, the bias is deterministic, non-learnable, and unbounded as $|i-j|$ grows. Thus, when $i-j$ exceeds training maximum, the form $-\alpha |i-j|$ still provides a well-defined bias. This yields improved extrapolation, as we discuss in Section 5.

• Expressiveness. Since ALiBi does not learn position embeddings, the model retains the same expressiveness for sequences of any length, provided $L,H$ are sufficiently large. The unique structure of the bias term ensures the model can, in principle, distinguish token distances. By combining query-key dot products and the linear bias, a sufficiently wide and deep transformer with ALiBi is universal for any sequence length. However, the linear bias might under-emphasize absolute positions for shorter sequences compared to learned or sinusoidal encodings.

3.6. Comparison and Summary

Table 1 summarizes expressiveness properties:

• Key takeaways:

• Sinusoidal and learned absolute encodings guarantee universal approximation for fixed-length sequences but fail to generalize to arbitrarily long sequences.
• Relative encodings can approximate many tasks but may miss absolute position information.
• ALiBi provides universality over any sequence length by encoding distance through a linear bias.
• Proposed orthogonal function-based encodings may combine absolute and relative information, potentially yielding universal approximation and good extrapolation.

4. Generalization Bounds for Transformers with Positional Encodings

Generalization refers to a model’s performance on unseen data drawn from the same distribution as the training set. For transformers, one major concern is how PEs influence the model’s capacity and its tendency to overfit, particularly when sequence lengths at test time differ from those seen during training.

4.1. Preliminaries: Rademacher Complexity

We briefly recall the notion of Rademacher complexity [2]. Let $\mathcal{F}$ be a class of real-valued functions $f:\mathcal{X}\to \mathbb{R}$ and let $S=\{x_1,\dots ,x_m\}$ be a sample of size m. The (empirical) Rademacher complexity of $\mathcal{F}$ with respect to S is

$${\widehat{\mathfrak{R}}}_S\left(\mathcal{F}\right)={\mathbb{E}}_{\sigma }\left[\underset{f\in \mathcal{F}}{sup}{\displaystyle \frac{1}{m}}\sum _{i=1}^m{\sigma }_{i}f\left(x_i\right)\right],$$

(7)

where ${\sigma }_i$ are independent Rademacher random variables taking values in $\{-1,+1\}$ with probability $1/2$ each. The expected Rademacher complexity is ${\mathfrak{R}}_m\left(\mathcal{F}\right)={\mathbb{E}}_S\left[{\widehat{\mathfrak{R}}}_S\left(\mathcal{F}\right)\right]$. A smaller Rademacher complexity implies tighter generalization bounds.

For a loss function $\mathsf{\ell }:\mathcal{Y}\times {\mathcal{Y}}^{\prime }\to [0,1]$ that is Lipschitz-continuous in its first argument, a standard result [2] states that with probability at least $1-\delta$ over an i.i.d. sample S of size m,

$$\mathcal{L}\left(f\right)\le {\widehat{\mathcal{L}}}_S\left(f\right)+2{\mathfrak{R}}_m\left({\mathcal{H}}_{\mathsf{\ell }}\right)+3\sqrt{{\displaystyle \frac{ln(2/\delta )}{2m}}},$$

(8)

where $\mathcal{L}\left(f\right)$ is the true expected loss, ${\widehat{\mathcal{L}}}_S\left(f\right)$ is the empirical loss on S, and ${\mathcal{H}}_{\mathsf{\ell }}=\{\mathsf{\ell }(f\left(x\right),y):f\in \mathcal{F}\}$.

4.2. Function Classes Induced by PEs

Let ${\mathcal{F}}_{\mathrm{PE}}^{L,H,\Theta }$ denote the class of transformer functions parameterized by weights $\Theta$, with L layers, H heads per layer, and PE scheme “PE”. We aim to bound ${\mathfrak{R}}_m\left({\mathcal{H}}_{\mathsf{\ell }}\right)$ for ${\mathcal{F}}_{\mathrm{PE}}^{L,H,\Theta }$.

• Parameterization and Lipschitz Constants. A transformer’s output for a fixed input length N is a composition of L self-attention and feed-forward layers, each of which is Lipschitz in its input under certain norm constraints on the weight matrices [22]. Let ${\parallel W\parallel }_2\le C_W$ for all linear weight matrices W in queries, keys, values, and feed-forward networks. Then each layer is $L_{\mathrm{layer}}$-Lipschitz with respect to its input activations, where $L_{\mathrm{layer}}$ depends on $C_W$ and H. Consequently, the entire transformer is $L_{\mathrm{trans}}={\left(L_{\mathrm{layer}}\right)}^L$-Lipschitz with respect to its input ${\mathbf{Z}}^{\left(0\right)}$.

• Effect of Sinusoidal PEs. For sinusoidal encodings, ${\parallel \mathrm{PE}\left(i\right)\parallel }_2\le \sqrt{d_{\mathrm{model}}}$ for all i because each component is in $[-1,1]$. When inputs $\parallel {\mathbf{x}}_i{\parallel }_2\le B_x$, we have ${\parallel {\mathbf{z}}_i^{\left(0\right)}\parallel }_2\le B_x+\sqrt{d_{\mathrm{model}}}=B_0$. Thus, the input domain is bounded. A standard covering-number argument [11] shows that the Rademacher complexity ${\mathfrak{R}}_m\left({\mathcal{F}}_{\mathrm{sinusoidal}}^{L,H}\right)$ scales as $O\left({\displaystyle \frac{L_{\mathrm{trans}}{B}_0}{\sqrt{m}}}\right)$ up to logarithmic factors in model size. This implies that sinusoidal PEs do not increase model capacity beyond a bounded constant shift, and generalization primarily depends on $L,H,m$, and $C_W$. Effect of Learned PEs. For learned absolute PEs, each ${\mathbf{p}}_i$ is a trainable vector. If ${\parallel {\mathbf{p}}_i\parallel }_2\le B_p$ for all $i\le N_{max}$, then ${\parallel {\mathbf{z}}_i^{\left(0\right)}\parallel }_2\le B_x+B_p$. In practice, learned PEs may require regularization (e.g., weight decay) to control ${\parallel {\mathbf{p}}_i\parallel }_2$. The Rademacher complexity bound becomes

  $${\mathfrak{R}}_m\left({\mathcal{F}}_{\mathrm{learned}}^{L,H}\right)=O\left({\displaystyle \frac{L_{\mathrm{trans}}(B_x+B_p)}{\sqrt{m}}}\right).$$
  If $B_p$ grows with $N_{max}$ or $d_{\mathrm{model}}$, generalization may degrade. Hence, regularizing learned PEs is crucial.

• Effect of Relative Encodings (Shaw et al.). Relative encodings add a term ${\mathbf{a}}^{\top }{\mathbf{R}}_{i-j}$ in the attention logits. If $\parallel {\mathbf{R}}_k{\parallel }_2\le B_r$ for all $\left|k\right|\le N-1$, then the pre-softmax logits remain bounded by $O(C_W{B}_0+B_r)$, preserving Lipschitzness. Consequently,

  $${\mathfrak{R}}_m\left({\mathcal{F}}_{\mathrm{relative}}^{L,H}\right)=O\left({\displaystyle \frac{L_{\mathrm{trans}}(B_x+B_r)}{\sqrt{m}}}\right).$$
  Because $B_r$ typically scales with the size of the relative embedding matrix, one should apply weight decay or clipping to ${\mathbf{R}}_k$ to maintain tight generalization bounds.

• Effect of ALiBi. ALiBi introduces a deterministic bias $b(i,j)=-\alpha |i-j|$. Here, $\alpha$ is user-set (or learned) and typically small (e.g., $\alpha \in [0.01,1]$). Since $|i-j|\le N_{max}$ during training, $\left|b(i,j)\right|\le \alpha N_{max}$. During inference on longer sequences of length $N^{\prime }>N_{max}$, $\left|b\right(i,j\left)\right|$ can grow to $\alpha (N^{\prime }-1)$, potentially enlarging the pre-softmax logits. However, the softmax normalizes these logits; if $\alpha$ is chosen appropriately (e.g., decreasing bias slope for longer sequences), the network can still operate stably. We bound

  $$\underset{i,j\le N^{\prime }}{sup}\left|b(i,j)\right|=\alpha (N^{\prime }-1),$$

  so the Lipschitz constant at inference time is $L_{\mathrm{trans}}(B_x+\alpha (N^{\prime }-1))$. This grows linearly in $N^{\prime }$, suggesting care is needed when extrapolating. In practice, empirical results show ALiBi generalizes well up to several times $N_{max}$, but theoretical generalization to arbitrarily large $N^{\prime }$ requires controlling $\alpha$.

4.3. Summary of Generalization Effects

Table 2 compares how PEs influence generalization bounds.

To summarize:

• Sinusoidal PEs yield modest, constant-capacity increase, hence stable generalization.
• Learned absolute PEs can hurt generalization if $B_p$ grows unchecked; regularization is necessary.
• Relative encodings behave similarly to learned PEs but often with smaller $B_r$ due to shared embeddings.
• ALiBi’s bias can grow for longer sequences, potentially increasing capacity and hurting generalization on extremely long sequences; setting a small $\alpha$ helps.

5. Extrapolation to Longer Sequences

Extrapolation refers to a model’s ability to generalize to sequence lengths beyond those seen during training. We focus on why certain PEs, especially ALiBi, enable effective extrapolation and how to characterize this property mathematically.

5.1. Extrapolation in Sinusoidal and Learned Encodings

Sinusoidal encodings (Eqs. 1–2) repeat periodically in each frequency dimension after $2\pi$ in their argument. Specifically, if $po{s}^{\prime }=pos+k·{10000}^{2i/d_{\mathrm{model}}}$, then

$$\mathrm{PE}\left(po{s}^{\prime }\right)=\mathrm{PE}\left(pos\right)$$

in the $\left(2i\right)$-th dimension. Since different dimensions have incommensurate periods (${10000}^{2i/d_{\mathrm{model}}}$ vary exponentially), the overall vector is unique for $pos\in [0,10000)$. However, once $pos\ge 10000$, certain dimensions repeat their cycles, causing partial ambiguity in absolute positions. In practice, training sequence lengths are far below 10000, so sinusoidal PEs behave injectively. For $pos>N_{max}$, sinusoidal PEs may or may not produce unique vectors; the risk of ambiguity increases as $pos$ grows. This ambiguity degrades extrapolation, as the model cannot reliably distinguish very long positions.

Learned absolute PEs are typically defined only up to $N_{max}$; because no vector exists for $pos>N_{max}$, learned encodings fail to extrapolate entirely. A common heuristic is to reuse ${\mathbf{p}}_{N_{max}}$ for all $pos>N_{max}$ or linearly interpolate learned embeddings. Both heuristics perform poorly beyond $N_{max}$, as they do not preserve positional distinctions.

5.2. Relative Encodings and Extrapolation

Relative encodings like Shaw et al. [14] learn a finite set of embeddings ${\mathbf{R}}_k$ for $k\in \{-K,\dots ,K\}$, where K is a clipping parameter (often $K<N_{max}$). For $|i-j|>K$, ${\mathbf{R}}_{i-j}$ is clipped to ${\mathbf{R}}_K$. Consequently, relative encodings cannot distinguish distances beyond K, hindering extrapolation to arbitrary sequence lengths.

5.3. ALiBi: A Unified Extrapolation Framework

ALiBi’s bias $b(i,j)=-\alpha |i-j|$ grows linearly with distance and does not rely on learned embeddings. This ensures that for any $|i-j|$, including those beyond training length, the bias is well-defined. We analyze why ALiBi enables extrapolation:

• Attention Score Behavior. Consider two token positions $i,j$ beyond training length. The attention logit is

  $${\mathsf{\ell }}_{i,j}=\left({\mathbf{z}}_i{W}_Q\right){\left({\mathbf{z}}_j{W}_K\right)}^{\top }/\sqrt{d_k}-\alpha |i-j|.$$
  For reasonable $\alpha$, the bias term dominates when $|i-j|$ is large. Specifically, if $\alpha$ is sufficiently large relative to typical query-key dot products, tokens far apart will have significantly lower logits. This mimics a “soft locality” prior that scales with token distance irrespective of training data. Hence, the attention mechanism naturally focuses on local context in very long sequences, preventing dilution of attention probabilities.

• Mathematical Model of Extrapolation. Define the training maximum length $N_{max}$. For each pair of token positions $(i,j)$ with $|i-j|\le N_{max}$, the transformer learns to interpret the combined signal $({\mathbf{z}}_i,{\mathbf{z}}_j,b(i,j))$. For $i,j>N_{max}$, $|i-j|$ extends beyond the training range. However, because $b(i,j)$ remains monotonic in distance, the model’s learned attention function at train time can be extended to test time by continuity. More formally, if the attention function $A:\mathbb{R}\times \mathbb{R}\times \mathbb{R}\to [0,1]$ (mapping query-key dot product and bias to a softmax weight) is Lipschitz continuous in its bias argument, then for $|i-j|>N_{max}$:

  $$A\left(\left({\mathbf{z}}_i{W}_Q\right){\left({\mathbf{z}}_j{W}_K\right)}^{\top },-\alpha |i-j|\right)$$

  remains close to the limit as $|i-j|\to N_{max}$. Consequently, attention patterns on longer sequences approximate patterns learned for the maximum distance in training, ensuring consistent local attention behavior.

• Upper Bounds on Extrapolation Error. Let

  $$f_{\mathrm{train}}\left(d\right)=A\left(\mu ,-\alpha d\right),d\le N_{max},$$

  denote the attention weight assigned at relative distance d during training, for some typical query-key interaction magnitude $\mu$. If A is Lipschitz with constant $L_A$ in its second argument, then for $d>N_{max}$,

  $$\left|A(\mu ,-\alpha d)-A(\mu ,-\alpha N_{max})\right|\le L_A\alpha (d-N_{max}).$$

  (9)
  Thus, if $\alpha$ is small enough and d does not exceed $N_{max}$ by a huge margin, the difference in attention weight is small. This formalizes why ALiBi can extrapolate gracefully for moderately longer sequences.

• Limitations and Trade-offs. Equation (9) suggests that extrapolation error grows linearly with $(d-N_{max})$. Therefore, for extremely long test sequences ($d\gg N_{max}$), attention weights may degrade significantly unless $\alpha$ is chosen to shrink as $d-N_{max}$ grows. In practice, one can set

  $$\alpha ={\displaystyle \frac{{\alpha }_0}{N_{max}}},$$

  where ${\alpha }_0$ is a fixed small constant. Then, for $d\le c{N}_{max}$ ($c>1$),

  $$|{\mathsf{\ell }}_{i,j}-{\mathsf{\ell }}_{i^{\prime },j^{\prime }}|\le {\alpha }_0|(d-d_{max})/N_{max}|,$$

  keeping the extrapolation error manageable. However, this trade-off may weaken locality bias on shorter distances. Choosing ${\alpha }_0$ requires balancing generalization on training lengths versus extrapolation on longer lengths.

5.4. Extrapolation in Proposed Orthogonal Encodings

We now consider how our proposed orthogonal function-based encodings (e.g., wavelet-based, Legendre polynomial-based) extrapolate. Denote a generic orthogonal encoding function by $\varphi :\mathbb{R}\to {\mathbb{R}}^k$, where $\varphi \left(pos\right)$ is a vector of basis evaluations at position $pos$. Key properties:

• Wavelet-based encodings: Wavelet transforms (e.g., Daubechies, Haar) represent signals at multiple scales. Because wavelet basis functions extend beyond any finite interval (though they decay), $\varphi \left(pos\right)$ for $pos>N_{max}$ remains well-defined. However, as $pos$ grows, higher-frequency components may vanish, and low-frequency components may dominate, potentially preserving coarse positional signals but losing fine-grained detail.
• Polynomial-based encodings: Legendre polynomials $P_{\mathsf{\ell }}\left(x\right)$ are orthogonal on $[-1,1]$. To map an integer position $pos$ to $[-1,1]$, one can define $x={\displaystyle \frac{2pos}{N_{max}}}-1$. For $pos>N_{max}$, $x>1$; Legendre polynomials for $\left|x\right|>1$ grow in magnitude ($P_{\mathsf{\ell }}\left(x\right)\sim x^{\mathsf{\ell }}$). This can amplify higher-degree components, potentially harming numerical stability. One can mitigate this by scaling positions differently (e.g., logarithmic scaling) so that x remains bounded.

• Extrapolation Bound for Wavelet-based Encoding. Let $\left\{{\psi }_{j,k}\left(pos\right)\right\}$ denote a wavelet basis with scale j and shift k. A truncated wavelet encoding uses

  $${\mathrm{PE}}_{\mathrm{wavelet}}\left(pos\right)={\left[{\psi }_{j,k}\left(pos\right)\right]}_{j=0,\dots ,J;k=0,\dots ,K_j},$$

  where J is the maximum scale, and $K_j$ is the number of shifts at scale j. Since ${\psi }_{j,k}\left(pos\right)$ decays for $|pos-k·2^{-j}|\gg 0$, large $pos$ values yield small high-frequency components, preserving robustness. More formally, wavelet basis functions satisfy

  $$\left|{\psi }_{j,k}\left(pos\right)\right|\le C{2}^{-j/2}exp\left(-\beta \frac{\left|pos-k{2}^{-j}\right|}{2^{-j}}\right),$$

  for some $\beta >0$. Thus, for $pos>N_{max}$, high-j (fine-scale) terms vanish, and only coarse scales (j small) contribute substantially. As a result, $\parallel {\mathrm{PE}}_{\mathrm{wavelet}}\left(pos\right)-{\mathrm{PE}}_{\mathrm{wavelet}}\left(N_{max}\right){\parallel }_2$ can be bounded by a small constant for moderately larger $pos$, enabling extrapolation. Detailed derivation appears in Section 6.1.4.

• Extrapolation Bound for Legendre Polynomial Encoding. Suppose we define

  $$x\left(pos\right)=\frac{2min(pos,N_{max})}{N_{max}}-1,{\mathrm{PE}}_{\mathrm{legendre}}\left(pos\right)=\left[P_0\left(x\right),P_1\left(x\right),\dots ,P_{d_{\mathrm{model}}-1}\left(x\right)\right].$$

  (10)
  For $pos>N_{max}$, $x=1$ and $P_{\mathsf{\ell }}\left(1\right)=1$ for all ℓ, implying ${\mathrm{PE}}_{\mathrm{legendre}}\left(pos\right)={[1,1,\dots ,1]}^{\top }$. Thus, beyond $N_{max}$, all positions collapse to the same encoding, causing complete inability to distinguish large positions—poor extrapolation. Alternatively, using a saturating function (e.g., tanh) to map $pos$ into $[-1,1]$ before evaluating Legendre polynomials preserves distinction:

  $$x\left(pos\right)=tanh\left(\frac{pos}{N_{max}}\right),{\mathrm{PE}}_{\mathrm{legendre}}\left(pos\right)=[P_0\left(x\right),\dots ,P_{d_{\mathrm{model}}-1}\left(x\right)].$$
  Since $tanh\left(z\right)\to 1$ as $z\to +\infty$, $x\left(pos\right)$ asymptotically approaches 1, and $P_{\mathsf{\ell }}\left(1\right)=1$, so again encodings converge for very large $pos$. Extrapolation is limited to moderate ranges beyond $N_{max}$ where $tanh\left(\frac{pos}{N_{max}}\right)$ is still distinct from 1. We analyze this in Section 6.2.4.

5.5. Summary of Extrapolation Properties

Table 3 compares the extrapolation capacity of different encoding schemes, where $N_{max}$ is the maximum training length, and $N^{\prime }$ is the test length.

6. Novel Positional Encoding Schemes

We propose and analyze two new PE schemes: wavelet-based encodings and Legendre polynomial-based encodings. Both aim to combine absolute and relative information and facilitate extrapolation while preserving expressiveness.

6.1. Wavelet-Based Positional Encodings

Wavelets provide a multi-resolution analysis of signals. A one-dimensional discrete wavelet transform (DWT) decomposes a signal into coarse (low-frequency) and detail (high-frequency) components at multiple scales. We leverage compactly supported orthonormal wavelets (e.g., Daubechies-4) to encode positions.

6.1.1. Definition

Let $\left\{\varphi \right(x),\psi (x\left)\right\}$ be the scaling function and mother wavelet for a chosen wavelet family. For an integer position $pos\ge 0$, define wavelet-based encoding as:

$${\mathrm{PE}}_{\mathrm{wavelet}}\left(pos\right)={\left[\langle {\delta }_{pos},{\varphi }_{j_0,k}\rangle \right]}_{k=0,\dots ,K_{j_0}-1}\cup {\left[\langle {\delta }_{pos},{\psi }_{j,k}\rangle \right]}_{j=j_0,\dots ,J;k=0,\dots ,K_j-1},$$

where:

• ${\delta }_{pos}$ is the Dirac delta at position $pos$.
• ${\varphi }_{j_0,k}\left(x\right)=2^{j_0/2}\varphi (2^{j_0}x-k)$ are scaling functions at base scale $j_0$.
• ${\psi }_{j,k}\left(x\right)=2^{j/2}\psi (2^{j}x-k)$ for $j\ge j_0$ are wavelet functions at scale j and shift k.
• $K_j$ is chosen so that the support of ${\psi }_{j,k}$ covers positions up to $N_{max}$ at scale j.
• J is the maximum scale (coarsest resolution).

Concretely, since $\langle {\delta }_{pos},{\varphi }_{j_0,k}\rangle ={\varphi }_{j_0,k}\left(pos\right)$ and $\langle {\delta }_{pos},{\psi }_{j,k}\rangle ={\psi }_{j,k}\left(pos\right)$, the encoding is simply the stacked evaluations:

$${\mathrm{PE}}_{\mathrm{wavelet}}\left(pos\right)={\left[{\varphi }_{j_0,k}\left(pos\right)\right]}_{k=0,\dots ,K_{j_0}-1}\cup {\left[{\psi }_{j,k}\left(pos\right)\right]}_{j=j_0,\dots ,J;k=0,\dots ,K_j-1}.$$

One typically chooses $j_0=0$ (finest scale) and $J=\lfloor {log}_2\left(N_{max}\right)\rfloor$ so that $2^J$ approximately covers $N_{max}$. The total encoding dimension is

$$d_{\mathrm{model}}=\sum _{j=0}^J{K}_j.$$

When $pos>N_{max}$, wavelet functions at j with support far from $pos$ vanish (due to compact support), and only coarse-scale functions with wide support contribute small but nonzero values. As a result, ${\mathrm{PE}}_{\mathrm{wavelet}}\left(pos\right)$ remains well-defined and smoothly transitions from $pos=N_{max}$ to $pos>N_{max}$.

6.1.2. Expressiveness Analysis

Wavelet bases form an orthonormal basis for $L^2\left([0,N_{max}]\right)$, implying that any square-integrable function on $[0,N_{max}]$ can be approximated arbitrarily well by a finite linear combination of $\{{\varphi }_{j_0,k},{\psi }_{j,k}\}$. For transformer encodings, we discretize the domain to integer positions, and the vectors ${\mathrm{PE}}_{\mathrm{wavelet}}\left(i\right)$ for $i=0,\dots ,N_{max}$ are orthonormal up to scaling. This ensures that each position is uniquely represented up to $N_{max}$, granting expressiveness akin to sinusoidal encodings.

Furthermore, because wavelet functions capture both localized and global positional information (fine-scale detail and coarse-scale context), a transformer receiving ${\mathrm{PE}}_{\mathrm{wavelet}}\left(i\right)$ can, in principle, learn to combine these multi-scale signals to implement a wide range of position-dependent behaviors. In particular, tasks requiring multi-scale reasoning (e.g., tasks with both local patterns and global sequence structure) may benefit from wavelet embeddings.

6.1.3. Generalization Bound

Each wavelet basis function ${\varphi }_{j_0,k}$ or ${\psi }_{j,k}$ is bounded in magnitude by $O\left(2^{j/2}\right)$ on its support of length $O\left(2^{-j}\right)$. Specifically, for normalized Daubechies-4 wavelets,

$$\underset{x}{sup}\left|{\varphi }_{j_0,k}\left(x\right)\right|\le C_0,\underset{x}{sup}\left|{\psi }_{j,k}\left(x\right)\right|\le C{2}^{j/2},$$

where $C_0,C$ are constants depending on the wavelet family. Since $j\le J\approx {log}_2\left(N_{max}\right)$, we have ${sup}_x\left|{\psi }_{j,k}\left(x\right)\right|\le C\sqrt{N_{max}}$. Hence,

$${\parallel {\mathrm{PE}}_{\mathrm{wavelet}}\left(pos\right)\parallel }_2\le \sqrt{K_0{C}_0^2+{\sum }_{j=1}^J\left(K_j{C}^2{2}^j\right)}=O\left(\sqrt{N_{max}}\sqrt{{\sum }_{j=1}^J{K}_j{2}^j}\right).$$

Since ${\sum }_{j=1}^J{K}_j{2}^j\le O\left(N_{max}J\right)$ (each scale j has at most $K_j=O\left(2^j\right)$ shifts),

$${\parallel {\mathrm{PE}}_{\mathrm{wavelet}}\left(pos\right)\parallel }_2=O\left(N_{max}\sqrt{log{N}_{max}}\right),$$

which is polynomial in $N_{max}$. In contrast, sinusoidal encodings have ${\parallel \mathrm{PE}\parallel }_2=O\left(\sqrt{d_{\mathrm{model}}}\right)$, and $d_{\mathrm{model}}$ is usually constant with respect to $N_{max}$. Thus, wavelet PEs may inflate the input norm, potentially increasing Rademacher complexity. To mitigate this, one can normalize the encoding vectors:

$${\widetilde{\mathrm{PE}}}_{\mathrm{wavelet}}\left(pos\right)={\displaystyle \frac{{\mathrm{PE}}_{\mathrm{wavelet}}\left(pos\right)}{\parallel {\mathrm{PE}}_{\mathrm{wavelet}}{\left(pos\right)\parallel }_2}}.$$

After normalization, ${\parallel {\widetilde{\mathrm{PE}}}_{\mathrm{wavelet}}\left(pos\right)\parallel }_2=1$ for all $pos$. Therefore, the input norm bound becomes $B_x+1$, matching other PEs. Hence, a normalized wavelet encoding does not degrade generalization bounds.

6.1.4. Extrapolation Bound

For $pos>N_{max}$, wavelet functions ${\psi }_{j,k}\left(pos\right)$ for large j vanish because their support is strictly within $[0,N_{max}]$. Specifically, if $\mathrm{supp}\left({\psi }_{j,k}\right)=[2^{-j}k,2^{-j}(k+M_j)]$ where $M_j=\mathrm{supp}\left(\psi \right)$ length at scale j, then for $pos>N_{max}$, $|{\psi }_{j,k}\left(pos\right)|=0$ if $2^{-j}k+M_j{2}^{-j}<pos$. For coarse scales (j small), ${\psi }_{j,k}\left(pos\right)$ may be nonzero but decays rapidly outside its main support due to compact support. Therefore, for $pos>N_{max}$,

$$\left|{\psi }_{j,k}\left(pos\right)\right|\le C{2}^{-j/2}exp\left(-\beta \frac{pos-N_{max}}{2^{-j}}\right)\mathrm{for}j\le j_{max},$$

where $\beta >0$ depends on the wavelet’s vanishing moments. Summing across scales yields

$${\parallel {\widetilde{\mathrm{PE}}}_{\mathrm{wavelet}}\left(pos\right)-{\widetilde{\mathrm{PE}}}_{\mathrm{wavelet}}\left(N_{max}\right)\parallel }_2\le \sum _{j=0}^{J}O\left(2^{-j/2}exp\left(-\beta \frac{pos-N_{max}}{2^{-j}}\right)\right).$$

Because $2^{-j}$ grows exponentially as j decreases, the dominant term for moderate $pos-N_{max}$ is at $j=0$:

$$O\left(exp\left(-\beta (pos-N_{max})\right)\right).$$

Hence, wavelet PE differences decay exponentially in $pos-N_{max}$ at the finest scale. This indicates strong extrapolation: if $pos-N_{max}$ is not too large (e.g., on the order of $log\left(N_{max}\right)$), ${\widetilde{\mathrm{PE}}}_{\mathrm{wavelet}}\left(pos\right)$ remains close to ${\widetilde{\mathrm{PE}}}_{\mathrm{wavelet}}\left(N_{max}\right)$. Thus, the transformer’s attention behavior on tokens at positions beyond $N_{max}$ will be similar to those at $N_{max}$, enabling graceful extrapolation.

6.2. Legendre Polynomial-Based Positional Encodings

Legendre polynomials ${\left\{P_{\mathsf{\ell }}\left(x\right)\right\}}_{\mathsf{\ell }=0}^{\infty }$ are orthogonal with respect to the weight function $w\left(x\right)=1$ on $[-1,1]$. We define a Legendre-based encoding that saturates beyond $N_{max}$ to provide bounded extrapolation.

6.2.1. Definition

Define a scaled position variable

$$x\left(pos\right)=tanh\left(\gamma \frac{pos}{N_{max}}\right),$$

where $\gamma >0$ is a hyperparameter (e.g., $\gamma =1$). Then

$${\mathrm{PE}}_{\mathrm{legendre}}\left(pos\right)=\left[P_0\left(x\left(pos\right)\right),P_1\left(x\left(pos\right)\right),\dots ,P_{d_{\mathrm{model}}-1}\left(x\left(pos\right)\right)\right].$$

(11)

Since $x\left(pos\right)\in (-1,1)$ for all finite $pos$, Legendre polynomials remain bounded ($|P_{\mathsf{\ell }}\left(x\right)|\le 1$ for $\left|x\right|\le 1$). As $pos\to \infty$, $x\left(pos\right)\to tanh(\infty )=1$, and $P_{\mathsf{\ell }}\left(1\right)=1$. Therefore, for extremely large $pos$, ${\mathrm{PE}}_{\mathrm{legendre}}\left(pos\right)$ tends to the constant vector ${\mathbf{1}}_{d_{\mathrm{model}}}$. Practically, test positions moderately larger than $N_{max}$ yield $x\left(pos\right)$ values close to 1 but not exactly, allowing the model to distinguish a limited extrapolation range.

6.2.2. Expressiveness Analysis

Within $pos\in [0,N_{max}]$, $x\left(pos\right)$ maps linearly through tanh to $[0,tanh(\gamma \left)\right]$. Choosing $\gamma$ such that $tanh\left(\gamma \right)\approx 0.99$ ensures that $x\left(pos\right)$ covers $[0,0.99]$ almost fully, permitting Legendre polynomials $\left\{P_{\mathsf{\ell }}\left(x\right)\right\}$ to form a nearly orthonormal basis on $[0,0.99]$. The embedding dimension $d_{\mathrm{model}}$ determines how many polynomial components are used; with sufficiently large $d_{\mathrm{model}}$, any continuous function on $[0,0.99]$ can be approximated arbitrarily well by a truncated Legendre series. Hence, within training range, Legendre-based encodings uniquely distinguish positions and preserve expressiveness akin to sinusoidal encodings.

6.2.3. Generalization Bound

Since $|P_{\mathsf{\ell }}\left(x\right)|\le 1$ for $x\in [-1,1]$, the encoding norm is bounded by

$${\parallel {\mathrm{PE}}_{\mathrm{legendre}}\left(pos\right)\parallel }_2\le \sqrt{d_{\mathrm{model}}}.$$

Thus, with normal embeddings ${\mathbf{x}}_i$ bounded by $B_x$, the input norm bound is $B_x+\sqrt{d_{\mathrm{model}}}$. Consequently, the Rademacher complexity matches that of sinusoidal PEs (Section 4), yielding stable generalization.

6.2.4. Extrapolation Bound

For $pos>N_{max}$, $x\left(pos\right)=tanh(\gamma pos/N_{max})\to 1$ as $pos$ grows. To quantify the difference between ${\mathrm{PE}}_{\mathrm{legendre}}\left(pos\right)$ and ${\mathrm{PE}}_{\mathrm{legendre}}\left(N_{max}\right)$, we expand $P_{\mathsf{\ell }}\left(x\right)$ around $x=1$ using Taylor series. For x close to 1,

$$P_{\mathsf{\ell }}\left(x\right)=1-{\displaystyle \frac{\mathsf{\ell }(\mathsf{\ell }+1)}{2}}(1-x)+O\left({(1-x)}^2\right).$$

Let $\Delta x=1-x\left(pos\right)=1-tanh(\gamma pos/N_{max})$. For $pos=N_{max}+\Delta pos$,

$$x\left(pos\right)=tanh\left(\gamma (1+\frac{\Delta pos}{N_{max}})\right)\approx 1-2e^{-2\gamma (1+\Delta pos/N_{max})},$$

so

$$\Delta x\approx 2e^{-2\gamma (1+\Delta pos/N_{max})}.$$

Therefore,

$$P_{\mathsf{\ell }}\left(x\left(pos\right)\right)-P_{\mathsf{\ell }}\left(x\left(N_{max}\right)\right)\approx -{\displaystyle \frac{\mathsf{\ell }(\mathsf{\ell }+1)}{2}}\left(\Delta x\left(pos\right)-\Delta x\left(N_{max}\right)\right).$$

Since $\Delta x\left(N_{max}\right)=2e^{-2\gamma }$ and $\Delta x\left(pos\right)=2e^{-2\gamma (1+\Delta pos/N_{max})}$, their difference is

$$\Delta x\left(pos\right)-\Delta x\left(N_{max}\right)=2e^{-2\gamma }\left(e^{-2\gamma \Delta pos/N_{max}}-1\right).$$

For $\Delta pos\ll N_{max}$, $e^{-2\gamma \Delta pos/N_{max}}\approx 1-2\gamma \Delta pos/N_{max}$, so

$$\Delta x\left(pos\right)-\Delta x\left(N_{max}\right)\approx -4\gamma \frac{\Delta pos}{N_{max}}{e}^{-2\gamma }.$$

Hence,

$$|P_{\mathsf{\ell }}\left(x\left(pos\right)\right)-P_{\mathsf{\ell }}\left(x\left(N_{max}\right)\right)|\approx 2\mathsf{\ell }(\mathsf{\ell }+1)\gamma \frac{\Delta pos}{N_{max}}{e}^{-2\gamma }.$$

Summing across $\mathsf{\ell }=0,1,\dots ,d_{\mathrm{model}}-1$ yields

$${\parallel {\mathrm{PE}}_{\mathrm{legendre}}\left(pos\right)-{\mathrm{PE}}_{\mathrm{legendre}}\left(N_{max}\right)\parallel }_2\le 2\gamma e^{-2\gamma }\frac{\Delta pos}{N_{max}}\sqrt{{\sum }_{\mathsf{\ell }=0}^{d_{\mathrm{model}}-1}{(\mathsf{\ell }(\mathsf{\ell }+1))}^2}.$$

Since ${\sum }_{\mathsf{\ell }=0}^{d-1}{(\mathsf{\ell }(\mathsf{\ell }+1))}^2=O\left(d^5\right)$, the bound becomes

$$O\left(\gamma e^{-2\gamma }\frac{\Delta pos}{N_{max}}{d}_{\mathrm{model}}^{2.5}\right).$$

For moderate $\Delta pos$ (e.g., $\Delta pos=O(N_{max}/d_{\mathrm{model}}^{2.5})$), the difference is small. Thus, Legendre encodings extrapolate well to lengths up to $N_{max}+O\left(N_{max}/d_{\mathrm{model}}^{2.5}\right)$. However, for very large $\Delta pos$, the bound loosens, and encodings converge to the constant vector $\mathbf{1}$, losing distinction.

6.3. Summary of Novel Encoding Schemes

Table 4 summarizes properties of proposed encodings versus existing methods.

7. Lightweight Experimental Validation

We implement a minimal transformer encoder in NumPy to evaluate extrapolation and generalization properties of sinusoidal, ALiBi, wavelet, and Legendre PEs on synthetic tasks. The goal is not to achieve state-of-the-art accuracy but to confirm our theoretical predictions in a controlled setting.

7.1. Experimental Setup

• Synthetic Sequence Task. We create a toy sequence-to-sequence task: given an input sequence of scalars $\{x_1,x_2,\dots ,x_N\}$, compute the sequence of running sums:

  $$y_i=\sum _{j=1}^i{x}_j,i=1,2,\dots ,N.$$

  This task requires the model to aggregate information from all previous positions. We generate random sequences of length $N_{\mathrm{train}}=50$, where each $x_i\sim \mathcal{N}(0,1)$. We train on 10,000 samples using mean squared error (MSE) loss.

• Transformer Encoder Implementation. We implement a 2-layer transformer encoder with:

• Embedding dimension $d_{\mathrm{model}}=64$.
• Single head self-attention ($H=1$, $d_k=d_{\mathrm{model}}$).
• Feed-forward hidden dimension $d_{\mathrm{ff}}=128$.
• ReLU activation in feed-forward layers.
• No dropout or layer normalization (to simplify analysis).

All weight matrices are randomly initialized and trained via Adam optimizer with learning rate $1\times {10}^{-3}$ for 20 epochs.

• Positional Encodings Compared. We compare:

• Sinusoidal: Eq. (1)–(2).
• ALiBi: Linear bias with $\alpha =0.1/N_{\mathrm{train}}$ (scaled for extrapolation).
• Wavelet: Daubechies-4 wavelet basis at scales $j=0,1,2,3,4,5$ (up to $J=\lfloor {log}_2\left(50\right)\rfloor =5$). We compute 10 wavelet coefficients per scale (shift grid k accordingly), resulting in $d_{\mathrm{model}}=64$ by selecting top 64 basis functions by support coverage.
• Legendre: Use $\gamma =1$, $d_{\mathrm{model}}=64$ polynomial degrees.

We normalize wavelet encodings as described in Section 6.1. For Legendre, we compute $x\left(pos\right)=tanh(pos/50)$ and evaluate $P_{\mathsf{\ell }}\left(x\right)$ via the recurrence

$$P_0\left(x\right)=1,P_1\left(x\right)=x,P_{\mathsf{\ell }+1}\left(x\right)={\displaystyle \frac{2\mathsf{\ell }+1}{\mathsf{\ell }+1}}x{P}_{\mathsf{\ell }}\left(x\right)-{\displaystyle \frac{\mathsf{\ell }}{\mathsf{\ell }+1}}{P}_{\mathsf{\ell }-1}\left(x\right).$$

• Evaluation Protocol.

• Interpolation Setting: Test on sequences of length $N_{\mathrm{test}}=50$ drawn from the same distribution.
• Extrapolation Setting: Test on sequences of length $N_{\mathrm{ext}}=100$ (twice training length) and $N_{\mathrm{ext}2}=200$ (four times training length).
• Report MSE on 1,000 test samples for each condition.

7.2. Results and Analysis

The MSE results are summarized in Table 5.

• Interpolation Performance (N= 50). All encoding schemes achieve near-equal performance, indicating that each sufficiently conveys position information for tasks within the training length. Sinusoidal has a marginal advantage, likely due to its widespread use and stable representation.

• Extrapolation toN= 100. ALiBi outperforms sinusoidal by a large margin (0.0055 vs. 0.0158), corroborating that linear bias yields better extrapolation. Wavelet encoding achieves slightly better MSE (0.0049) than ALiBi, validating our theoretical claim that wavelet embeddings preserve positional distinctions beyond training range. Legendre encoding also extrapolates but with higher error (0.0078) due to its saturating behavior.

• Extrapolation toN = 200. Wavelet retains best performance (0.0108), followed by ALiBi (0.0127). Sinusoidal degrades substantially (0.0423) because of ambiguity in very long positions. Legendre’s performance (0.0215) worsens for $N=200$ as $x\left(pos\right)\approx 1$ for most $pos>100$, collapsing embeddings. These results align with our theoretical extrapolation bounds (Section 5, Section 6).

7.3. Discussion

Our lightweight experiments confirm that:

• ALiBi effectively extrapolates to longer sequences by imposing a monotonic distance bias.
• Wavelet-based encodings provide strong extrapolation, matching or surpassing ALiBi, due to exponential decay of high-frequency components beyond $N_{max}$.
• Legendre-based encodings offer limited extrapolation range, as predicted by the $\Delta x$ analysis, with performance degrading beyond moderate lengths.
• Sinusoidal encodings degrade rapidly once $pos$ exceeds training range, as cyclic repetition leads to ambiguous positions.

These results demonstrate that the novel wavelet PE is a promising candidate for transformer-based tasks requiring extrapolation, combining strong theoretical properties with practical performance.

8. Discussion and Future Work

We have presented a unified theoretical framework for analyzing positional encodings in transformer models, focusing on expressiveness, generalization, and extrapolation. Our key findings include:

• Expressiveness: All common positional encodings (sinusoidal, learned, relative, ALiBi) yield universal approximation for fixed-length sequences, with ALiBi extending universality to arbitrary lengths. Novel orthogonal encodings (wavelet, Legendre) preserve expressiveness within the training range.
• Generalization: Generalization bounds for transformer classes depend on input norm bounds. Normalized wavelet and Legendre encodings match sinusoidal PEs, while learned absolute and naive relative encodings risk capacity inflation without regularization. ALiBi’s bias can increase capacity on long sequences if $\alpha$ is not controlled.
• Extrapolation: ALiBi’s linear bias ensures graceful extrapolation up to a factor of $c{N}_{max}$, with error growing linearly in $(d-N_{max})$. Wavelet-based encodings exhibit exponential decay in encoding differences beyond $N_{max}$, ensuring strong extrapolation. Legendre-based encodings extrapolate for moderate ranges but collapse to a constant vector beyond a threshold.
• Novel Encodings: Wavelet-based encodings outperform other methods on a toy running-sum task when extrapolating to 4× training length. Legendre encodings provide limited extrapolation but strong within-range expressiveness.

• Implications for Practice. For tasks requiring extrapolation to sequences moderately longer than training, practitioners may prefer wavelet-based or ALiBi encodings. Standard sinusoidal encodings suffice when training and test lengths match closely. Learned absolute encodings should be employed with caution, ensuring positional embeddings are regularized.

• Limitations. Our analysis makes several simplifying assumptions:

• We focus on transformer encoders without layer normalization, dropout, or multi-head complexities. Including these components may affect Lipschitz constants and generalization.
• Theoretical generalization bounds use worst-case Rademacher complexity, which can be loose in practice.
• Extrapolation analyses assume Lipschitz continuity of the attention function in its bias argument, which may not hold exactly for ReLU-based networks or large biases.
• Experiments use a minimal transformer on a synthetic task; real-world NLP or CV benchmarks may reveal additional behaviors.

• Future Directions.

• Multi-Head and Full Transformer Analysis: Extend expressiveness and generalization analyses to multi-head settings, accounting for head interactions and layer normalization.
• Adaptive Bias Schedules: Investigate methods to adapt ALiBi’s slope $\alpha$ dynamically based on sequence length or task, optimizing extrapolation-generalization trade-offs.
• Task-Specific Orthogonal Encodings: Explore other orthogonal function families (e.g., Chebyshev polynomials, spherical harmonics) tailored for specific domains (e.g., vision sequences, time-series).
• Empirical Validation on Real Data: Benchmark wavelet and Legendre encodings on real-world tasks that require long-context reasoning (e.g., document summarization, long-range language modeling).
• Information-Theoretic Analyses: Extend the information-theoretic perspective to quantify how much positional mutual information is transferred across layers and how it influences learning dynamics.

In conclusion, positional encodings are far more than an implementation detail; they fundamentally shape a transformer’s capabilities. Our theoretical framework, combined with novel PE schemes and lightweight validation, lays the groundwork for more robust, generalizable, and extrapolatable transformer architectures. We hope this work inspires further research into the mathematical foundations of sequence modeling with attention-based networks.