A Unified Perspective on Efficient Attention: Generalized Memory and Kernel Function Selection in Transformers

1. Introduction

Early research by James A. Anderson [1] and Leon N. Cooper [2] established foundational concepts for neural network-based memory models. They proposed that memories are not stored in specific locations but are distributed across a network of interconnected neurons. In his 1985 paper “Distributed Memory” [3] Leon N. Cooper proposed a mathematical model explaining how memory remains stable even as brain cells die. He argued that memories are not stored in individual neurons but are distributed across the network of connections (synapses) between them, much like a hologram encodes an image across its entire surface. This distributed system acts as a “content-addressable” memory, allowing a complete memory to be recalled from a partial cue. The model uses a matrix of synaptic strengths that are modified based on neural activity. Cooper also suggested this framework could explain both short-term memory (via temporary synaptic changes) and long-term memory (via permanent ones) within a single, unified system.

While James Anderson and Leon Cooper’s early models established that memory could be distributed across a network, John Hopfield’s 1982 work introduced a crucial difference: dynamics and error correction. Hopfield introduced a recurrent, nonlinear network where stored memories are stable states in an “energy landscape” [4] When presented with a partial or noisy cue, the network dynamically evolves, like a ball rolling downhill, until it settles into the nearest complete, stored memory. This iterative process makes Hopfield networks a true content-addressable memory system, capable of robust pattern completion and error correction, a significant advance over the earlier linear models.

The Transformer architecture, introduced in the groundbreaking 2017 paper [5] “Attention Is All You Need,” revolutionized natural language processing. Its core innovation was replacing traditional recurrent and convolutional layers with a mechanism based on self-attention. This design choice was pivotal: by removing the inherently sequential computations of recurrent models, the Transformer could be massively parallelized, dramatically reducing training times on modern hardware. This parallel self-attention mechanism enables the model to directly assess the importance of all words in an input sequence relative to each other, making it exceptionally effective at capturing complex context and long-range dependencies.

2. Scaled-Dot-Product Attention in Transformer

In this discussion, we focus on the self-attention mechanism [5], some mathematical symbols are adopted from [6].

We assume that q, $\mathit{v}$, k are d-dimensional random vectors respectively, where q is a query vector, $\mathit{v}$ is a value vector, and k is a key vector. The vectors q and k reside in the same d-dimensional space ${\mathit{R}}^d$ respectively, while the vector $\mathit{v}$ may reside in a different d-dimensional space.

The vectors q and $\mathit{v}$ are projections of a high-dimensional input vector x, which lies in m-dimensional space ${\mathit{R}}^m$. The dimension m is much larger than the dimension d. The space ${\mathit{R}}^d$ is a d-dimensional subspace within the larger space ${\mathit{R}}^m$.

For simplicity, we will drop the dependent variable x in vectors $\mathit{v}$ and q in the following discussion.

Let T denote the sequence length. We denote a sequence of query vectors as $\left\{{\mathit{q}}_1,{\mathit{q}}_2,{\mathit{q}}_3,...,{\mathit{q}}_{\mathit{T}}\right\}$. Similarly, we denote a sequence of value vectors as $\left\{{\mathit{v}}_1,{\mathit{v}}_2,{\mathit{v}}_3,...,{\mathit{v}}_{\mathit{T}}\right\}$.

The vectors ${\mathit{q}}_{\mathit{i}}$ and ${\mathit{v}}_{\mathit{i}}$ are two low-dimensional projections of an input vector ${\mathit{x}}_{\mathit{i}}$. The vector ${\mathit{x}}_{\mathit{i}}$ lives in m-dimensional space ${\mathit{R}}^m$.

We also have a sequence of key vectors denoted as $\left\{{\mathit{k}}_1,{\mathit{k}}_2,{\mathit{k}}_3,...,{\mathit{k}}_{\mathit{T}}\right\}$. Each key vector ${\mathit{k}}_{\mathit{i}}$ represents an address memory unit, which is also a low-dimensional projection of the input vector ${\mathit{x}}_{\mathit{i}}$. Each key vector ${\mathit{k}}_{\mathit{i}}$ can also be viewed as a hidden state associated a value vector ${\mathit{v}}_{\mathit{i}}$ which represents a content or feature memory unit.

A query vector q will match each address memory unit ${\mathit{k}}_{\mathit{i}}$ to compute a similarity score. The similarity score can be represented as the cross-correlation value, which is the inner product of the two vectors q and ${\mathit{k}}_{\mathit{i}}$ denoted as ${\mathit{q}}^{\mathit{t}}{\mathit{k}}_{\mathit{i}}$. Note that the vector q is one of vectors in the sequence $\left\{{\mathit{q}}_1,{\mathit{q}}_2,{\mathit{q}}_3,...,{\mathit{q}}_{\mathit{T}}\right\}$.

This self-attention mechanism allows the model to dynamically attend to different parts of the input sequence when computing the output value vector.

The definition of the softmax weighting function is described in [6]:

$$w_i(\mathit{q})={\displaystyle \frac{exp\left(\alpha {\mathit{q}}^{\mathit{t}}{\mathit{k}}_i\right)}{\sum _{j=1}^{T}exp\left(\alpha {\mathit{q}}^{\mathit{t}}{\mathit{k}}_j\right)}}$$

(1)

where:

• $\alpha$is a positive scaling factor.
• t is a transpose operator, which transforms a column vector to a row vector.
• ${\mathit{q}}^{\mathit{t}}{\mathit{k}}_{\mathit{i}}$ or the dot product denoted as <q,${\mathit{k}}_{\mathit{i}}$> is the inner product between the query vector $\mathit{q}$ and i-th key vector ${\mathit{k}}_{\mathit{i}}.$
• The denominator $\sum _{j=1}^{T}exp\left(\alpha {\mathit{q}}^{\mathit{t}}{\mathit{k}}_j\right)$ is the sum of the exponentials of all the inner products between the query q and the set of key vectors $\left\{{\mathit{k}}_1,{\mathit{k}}_2,{\mathit{k}}_3,...,{\mathit{k}}_{\mathit{T}}\right\}$.
• $w_i\left(\mathit{q}\right)$ is the softmax weight corresponding to the i-th key vector ${\mathit{k}}_{\mathit{i}}$

The context vector $\widehat{\mathit{v}}$ is estimated by weighted average of all vectors ${\mathit{v}}_{\mathit{i}}$:

$$\widehat{\mathit{v}}=\sum _{i=1}^T{w}_{\mathrm{i}}\left(\mathit{q}\right){\mathit{v}}_{\mathbf{i}}$$

(2)

3. Tensor Representation of Distributed Memory

Leon N. Cooper’s distributed memory model [3] provided a framework, demonstrating how associative memory could be encoded in a synaptic matrix (A). This 2nd-order tensor (a matrix) was mathematically elegant and biologically plausible for storing pairwise associations: a specific input pattern (f) is linked to a corresponding output pattern (g). The memory is built by summing the outer products of these vector pairs. Retrieval is then a simple matrix-vector multiplication, where an input cue probes the memory to recall its associated output.

3.1. Pairwise Associations

Cooper’s Model: Stores associations between two vectors, (f, g). The memory is a 2nd-order tensor (matrix A).

Cooper’s Model: The memory matrix is constructed by summing outer products: A = Σ g ⊗ f.

The outer product g ⊗ f multiplies two vectors to create a matrix. Each element (i, j) of the resulting matrix is the product of the i-th element of column vector g and the j-th element of row vector f

Cooper’s Model: Retrieval of h is done by multiplying the matrix A with an input vector q: h = A * q.

3.2. Extending Cooper’s Distributed Memory for Multi-Way Associations

We begin by defining the total memory, M, analogous to A, as a sequence of memory states or operators over time:

$$\mathit{M}=\left(\mathit{M}\right(1), \mathit{M}(2),\dots ,\mathit{M}(T\left)\right)$$

(3)

Here, each M(i) represents the complete memory operator at a specific time step i. This formulation introduces a dynamic aspect, allowing the memory to evolve.

Following Cooper’s representation, each memory operator can be composed of two distinct components:

$$\mathit{M}\left(i\right)=\left(\mathit{L}\right(i),\mathbf{S}(i\left)\right)$$

(4)

Where $\mathit{L}\left(i\right)$ is a long-term memory, which represents the stable, context-independent, and foundational knowledge of the unit i. It is the a priori semantic identity or pattern that the system has learned and stored over time. This memory may be static and may not change based on the immediate context. It answers the question: “What is this concept in isolation?”

Where $\mathbf{S}\left(i\right)$is a short-term memory, which represents the transient, context-dependent understanding of the item, which emerges from its interaction with its neighbors and its position in the sequence.

3.3. The Key-Value Representation of the Memory Operator

To make this model computationally concrete, we adopt the key-value structure from the attention mechanism. The memory operator M(i) at any given time is represented as a pair:

$$\mathit{M}\left(i\right) =({\mathit{v}}_{\mathit{i}},{\mathit{k}}_{\mathit{i}})$$

(5)

The key vector ${\mathit{k}}_{\mathit{i}}$ serves as the content-addressable index of the memory. It is the pattern to be matched against. In the context of our dual-memory model, the key can be a function of both long-term knowledge $\mathit{L}\left(i\right)$ and short-term context $\mathbf{S}\left(i\right)$. It answers the question, “What information do I represent?”

The value vector ${\mathit{v}}_{\mathit{i}}$ is the content to be retrieved. It holds the actual information or signal associated with its corresponding key. It answers the question, “What information should I provide if I am selected?”

This key-value pairing is a powerful evolution of Cooper’s model. Instead of a monolithic matrix A storing all associations implicitly, the memory is now a structured set of (content, address) pairs, making the retrieval process more explicit and flexible.

3.4. Associative Retrieval via Query

Memory retrieval is no longer a simple matrix multiplication but an interactive, query-based process. A query vector $\mathit{q}$ represents the current focus of attention or the information need of the system. The interaction between the memory and the query $\mathit{q}$ is defined by the triplet:

$$\left(\mathit{M}\right(i),\mathit{q})=\left({\mathit{v}}_{\mathit{i}},{\mathit{k}}_{\mathit{i}},\mathit{q}\right)$$

(6)

This interaction is governed by an operator F, which takes the full memory M and the query q to produce a final output context vector $\widehat{\mathit{v}}$.

The retrieval operator R acts ($\mathit{M},\mathit{q})$ is to estimate the context vector $\widehat{\mathit{v}}$:

$$\widehat{\mathit{v}}=\mathit{R}(\mathit{M},\mathit{q})$$

(7)

The standard and most powerful way to define this operator is through a weighted sum, where the weights $G({\mathit{k}}_{\mathit{j}},\mathit{q})$ are determined by the similarity between the query and each key:

$$\widehat{\mathit{v}}=\sum _{i=1}^T{\mathit{v}}_{\mathbf{i}}G({\mathit{k}}_{\mathit{i}},\mathit{q})$$

(8a)

Or with normalized term as:

$$\widehat{\mathit{v}}=\sum _{i=1}^T{\mathit{v}}_{\mathbf{i}}G({\mathit{k}}_{\mathit{i}},\mathit{q})/\sum _{j=1}^{T}G({\mathit{k}}_{\mathit{j}},\mathit{q})$$

(8b)

This formula is the essence of the attention mechanism. The function $G({\mathit{k}}_{\mathit{j}},\mathit{q})$ is a kernel that measures the similarity. One form kernel is the exponential function, which makes it a universal kernel, capable of learning any continuous similarity function:$G({\mathit{k}}_{\mathit{j}},\mathit{q})=exp\left(\alpha {\mathit{q}}^{\mathit{t}}{\mathit{k}}_i\right)=exp\left(\alpha {\mathit{k}}_{\mathit{i}}^{\mathit{t}}\mathit{q}\right)$

This triplet forms the fundamental computational unit of the associative retrieval process, mirroring the Scaled Dot-Product Attention mechanism:

Matching: The query $\mathit{q}$ is compared against every ${\mathit{k}}_{\mathit{i}}$ in the memory sequence M. This is typically performed using a dot product, the inner product value ${(\mathit{k}}_{\mathit{i}}^{\mathit{t}}\mathit{q})$ measures the similarity or “resonance” between the current need and the memory’s address.

Weighting: The raw similarity scores are normalized (e.g., via a softmax function) to produce an attention distribution, $exp\left(\alpha {\mathit{q}}^{\mathit{t}}{\mathit{k}}_i\right)$. These weights represent the relevance of each memory item M(i) to the query.

Retrieval: The final output is a context vector of weighted sum of all value vectors defined as Equation 1 or Equation 8a.

While powerful, this exponential kernel is computationally expensive compared with pure linear kernel. We can explore different simplification of kernels.

4. The Linear Kernel: A Connection to RNNs

In the linear case, the retrieval operator R simplifies significantly. Instead of a complex, query-dependent weighting, the operation becomes a direct matrix-vector multiplication.

$$\mathit{R}(\mathit{M},\mathit{q})=\mathit{H}\left(\mathit{M}\right)\mathit{q}$$

(9)

Here, the term $\mathit{H}\left(\mathit{M}\right)$ represents a single, collective-hidden memory matrix that is constructed from the entire sequence:

$$\mathit{H}(\mathit{M})=\sum _{i=1}^T{\mathit{v}}_{\mathbf{i}}{\mathit{k}}_{\mathit{i}}^{\mathit{t}}$$

(10)

This formulation is a direct echo of Cooper’s distributed memory theory, defined as the equation: A = Σ g ⊗ f, where g is value vector, f is key vector. It forges a single memory matrix by summing the outer products of all individual memories (key-value pairs). The system stores a superimposed representation of all information in this one matrix. Retrieval is then performed by probing this matrix with the query. Substitute the Equation 10 into Equation 9, we have the following equation:

$$\mathit{R}(\mathit{M},\mathit{q})=\mathit{H}(\mathit{M})\mathit{q}=\sum _{i=1}^T{\mathit{v}}_{\mathbf{i}}{(\mathit{k}}_{\mathit{i}}^{\mathit{t}}\mathit{q})$$

(11)

The final expression shows that this is equivalent to a weighted sum of the value vectors, where the weight is simply the raw dot product (or inner product) <q,${\mathit{k}}_{\mathit{i}}$> between the query and each key denoted as ${\mathit{q}}^{\mathit{t}}{\mathit{k}}_{\mathit{i}}$. For linear kernel $G({\mathit{k}}_{\mathit{i}},\mathit{q})={\mathit{q}}^{\mathit{t}}{\mathit{k}}_i$, normalizing the output yields:

$$\widehat{\mathit{v}}=\sum _{i=1}^T{\mathit{v}}_{\mathbf{i}}{(\mathit{k}}_{\mathit{i}}^{\mathit{t}}\mathit{q})/\sum _{j=1}^T{(\mathit{k}}_{\mathit{i}}^{\mathit{t}}\mathit{q})$$

(12a)

It is important to note that the denominator,$\sum _{j=1}^T{(\mathit{k}}_{\mathit{i}}^{\mathit{t}}\mathit{q})$ being a simple sum of dot products, can be close to zero, which could lead to numerical instability [7]. The unnormalized estimation of the context vector $\widehat{\mathit{v}}$ is given by:

$$\widehat{\mathit{v}}=\sum _{i=1}^T{\mathit{v}}_{\mathbf{i}}{(\mathit{k}}_{\mathit{i}}^{\mathit{t}}\mathit{q})$$

(12b)

5. The Recursive Nature of Linear Memory

This linear formulation reveals a powerful insight: the memory matrix $\mathit{H}\left(\mathit{M}\right)$,defined in the Equation 10, which we will now call ${\mathbf{A}}_T=\mathit{H}\left(\mathit{M}\right)$ given by:

$${\mathbf{A}}_T=\sum _{i=1}^T{\mathit{v}}_{\mathbf{i}}{\mathit{k}}_{\mathit{i}}^{\mathit{t}}$$

(13)

Let ${\mathit{b}}_{\mathit{T}}^{\mathit{t}}$ be the sum of all row key vectors, used for normalization:

$${\mathit{b}}_{\mathit{T}}^{\mathit{t}}=\sum _{i=1}^T{\mathit{k}}_{\mathit{i}}^{\mathit{t}}$$

(14)

When a new item ${\mathit{v}}_{\mathit{T}+1},{\mathit{k}}_{\mathit{T}+1}$ arrives, we don’t need to recompute the entire memory from scratch. We can simply update the previous state:

$${\mathbf{A}}_{T+1}={\mathbf{A}}_T+{\mathit{v}}_{\mathit{T}+1}{\mathit{k}}_{\mathit{T}+1}^{\mathit{t}}$$

(15)

${\mathit{b}}_{\mathit{T}+1}^{\mathit{t}}$ can be updated recursively the following equation:

$${\mathit{b}}_{\mathit{T}+1}^{\mathit{t}}={\mathit{b}}_{\mathit{T}}^{\mathit{t}}+{\mathit{k}}_{\mathit{T}+1}^{\mathit{t}}$$

(16)

This incremental update is conceptually identical to the state transition in a Recurrent Neural Network (RNN). An RNN updates its hidden state by combining the previous state with the new input, here, we update the memory matrix ${\mathbf{A}}_{T+1}$ by incorporating the new key-value pair, we update ${\mathit{b}}_{\mathit{T}+1}^{\mathit{t}}$ by incorporating the new key. This reveals that linear attention is a form of recurrent state-space model.

For context vector estimation, let’s break down the retrieval at step T and T+1.

The unnormalized output ${\widehat{\mathit{v}}}_T$ is the memory matrix ${\mathbf{A}}_T$ probed by the query ${\mathit{q}}_{\mathbf{T}}$:

$${\widehat{\mathit{v}}}_T={\mathbf{A}}_T{\mathit{q}}_{\mathbf{T}}=\sum _{i=1}^T{\mathit{v}}_{\mathbf{i}}{\mathit{k}}_{\mathit{i}}^{\mathit{t}}{\mathit{q}}_{\mathbf{T}}$$

(17)

By defining ${\mathit{d}}_{\mathbf{i},\mathbf{T}}={\mathit{k}}_{\mathit{i}}^{\mathit{t}}{\mathit{q}}_{\mathbf{T}}$, the above equation becomes:

$${\mathbf{A}}_T{\mathit{q}}_{\mathbf{T}}=\sum _{i=1}^T{\mathit{d}}_{\mathbf{i},\mathbf{T}}{\mathit{v}}_{\mathbf{i}}$$

(18)

The normalization factor $z_{\mathrm{T}}$ is defined as the sum of all similarities:

$${{\mathit{z}}_{\mathbf{T}}=\mathit{b}}_{\mathit{T}}^{\mathit{t}}{\mathit{q}}_{\mathbf{T}}=\sum _{i=1}^T{\mathit{k}}_{\mathit{i}}^{\mathit{t}}{\mathit{q}}_{\mathbf{T}}=\sum _{i=1}^T{\mathit{d}}_{\mathbf{i},\mathbf{T}}$$

(19)

The normalized vector ${\widehat{\mathit{v}}}_T$ is their ratio:

$${\widehat{\mathit{v}}}_T={\mathit{u}}_T/z_{\mathrm{T}}={\mathbf{A}}_T{\mathit{q}}_{\mathbf{T}}/z_{\mathrm{T}}=\sum _{i=1}^T{\mathit{d}}_{\mathbf{i},\mathbf{T}}{\mathit{v}}_{\mathbf{i}}/\sum _{i=1}^T{\mathit{d}}_{\mathbf{i},\mathbf{T}}$$

(20)

Note that ${\widehat{\mathit{v}}}_T$ $\mathrm{c}\mathrm{a}\mathrm{n} \mathrm{b}\mathrm{e} \mathrm{d}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{d} \mathrm{b}\mathrm{y}$ ${\mathbf{A}}_T{\mathit{q}}_{\mathbf{T}}$ can be computed with linear time complexity. Likewise, the normalization factor $z_{\mathrm{T}}$ can also be computed in linear time.

For the next time step, T+1, the context vector is similarly estimated with normalized weights as follows:

$${\widehat{\mathit{v}}}_{T+1}={\mathit{u}}_{T+1}/z_{T+1}={\mathbf{A}}_{T+1}{\mathit{q}}_{\mathbf{T}+1}/z_{T+1}$$

(21)

Where ${\mathit{u}}_{T+1}$ and $z_{\mathrm{T}+1}$ are defined as follows:

$${{\mathit{u}}_{T+1}=\mathbf{A}}_{T+1}{\mathit{q}}_{\mathbf{T}+1}=({\mathbf{A}}_T+{\mathit{v}}_{\mathit{T}+1}{\mathit{k}}_{\mathit{T}+1}^{\mathit{t}}){\mathit{q}}_{\mathbf{T}+1}$$

$${{\mathit{u}}_{T+1}=\mathbf{A}}_{T+1}{\mathit{q}}_{\mathbf{T}+1}=({\mathbf{A}}_T{\mathit{q}}_{\mathbf{T}+1}+{\mathit{v}}_{\mathit{T}+1}{\mathit{k}}_{\mathit{T}+1}^{\mathit{t}}{\mathit{q}}_{\mathbf{T}+1})$$

$${z_{\mathrm{T}+1}=\mathit{b}}_{\mathit{T}+1}^{\mathit{t}}{\mathit{q}}_{\mathbf{T}+1}$$

The unnormalized context vector estimation is defined as:

$${\widehat{\mathit{v}}}_{T+1}={\mathit{u}}_{T+1}$$

(22)

6. Kernel Function Selection

We discuss how to select kernels of $G({\mathit{k}}_{\mathit{j}},\mathit{q})$ as defined in Equation 8a, and Equation 8b to reduce computation in attention model.

In the standard Transformer, the core of the associative retrieval mechanism is the dot-product attention, which can be viewed as applying a universal kernel function, $G({\mathit{k}}_{\mathit{j}},\mathit{q})$ = $exp\left(\alpha {\mathit{k}}_{\mathit{j}}^{\mathit{t}}\mathit{q}\right)$, followed by normalization. This is standard Scaled-Dot-Product function or softmax function. In the following discussion, $\mathit{x}$ and $\mathit{y}$ refer to ${\mathit{k}}_{\mathit{j}}$ and $\mathit{q}$, respectively.

Consider a kernel of the form k($\mathit{x}$, $\mathit{y}$) = ψ(<$\mathit{x}$, $\mathit{y}$ >), where < $\mathit{x}$, $\mathit{y}$ > is the inner product of the vectors $\mathit{x}$ and $\mathit{y}$, and ψ is a real-valued function. If we let z = < $\mathit{x}$, $\mathit{y}$ >, the function ψ(z) = exp(αz) is a universal kernel [11] when α > 0.

The most direct simplification of the kernel function is to replace exp(αz) with the linear kernel, ψ(z) = αz.

Another way to reduce the kernel’s computation is to use a feature map, φ, which maps each component of a vector to a non-negative value. For instance, φ(r) could be elu(r) + 1 or relu(r) [8], where r is a real value. In this way, the kernel G(x, y) = k($\mathit{x}$, $\mathit{y}$)=G($\mathit{x}$, $\mathit{y}$) = ${\mathsf{\phi }\left(\mathit{x}\right)}^t\mathsf{\phi }\left(\mathit{y}\right)$ is a valid kernel. However, it is not a universal kernel because its feature map is not sufficiently expressive. Because The mapping, such as elu(r) + 1 or relu(r), compresses or loses information from the negative components of a vector because it either compresses these values or discards them entirely.

Note that when computing the normalized weights in Equation 8b with the RBF kernel listed in Table 1, where k($\mathit{x}$, $\mathit{y})=G({\mathit{k}}_{\mathit{j}},\mathit{q})$, $\mathit{x}={\mathit{k}}_{\mathit{j}}$, and $\mathit{y}=\mathit{q}$, $G({\mathit{k}}_{\mathit{j}},\mathit{q})={exp(-‖\mathit{q}‖}^2/\gamma \left)exp\right({\mathit{k}}_{\mathit{j}}^{\mathit{t}}(2\mathit{q}-{\mathit{k}}_{\mathit{j}})/\gamma )$, the factor of ${exp(-‖\mathit{q}‖}^2/\gamma )$, appears in both the numerator and the denominator. This factor cancels out, which results in the normalized weights being computed using the softmax function.

Table 1. Kernels. 

Name	Expression	Type
Exponential	ψ (z)=exp(z)	Universal
Linear	ψ (z)=z	Valid
Geometric Series	ψ (z)=1/(1-z)	Conditionally universal if its input, z, is within in range of (-1,1). This condition is met when z is computed from cosine kernel
Cosine	k($\mathit{x}$, $\mathit{y}$) =< $\mathit{x}$, $\mathit{y}$> /$\sqrt{(\alpha +{‖\mathit{x}‖}^2)(\alpha +{‖\mathbf{y}‖}^2)}$	Valid when $\alpha \ge 0$ such as $\alpha =1.e-3$
Shift Cosine	k($\mathit{x}$, $\mathit{y}$) =(1+z)	Valid when z is computed from cosine kernel
RBF	k($\mathit{x}$, $\mathit{y}$) =exp(${-‖\mathit{x}-\mathit{y}‖}^2/\gamma$)	Universal when $\gamma >0$
Clipped Geometric Series	k($\mathit{x}$, $\mathit{y}$) =1/(1+eps-c(<$\mathit{x}$, $\mathit{y}$>)/M)	Conditionally universal, c(<$\mathit{x}$, $\mathit{y}$>) =M when <$\mathit{x}$, $\mathit{y}$> >M, where M>0, such as M=300, eps is small positive number such as eps=1.e-4
Relu	k($\mathit{x}$, $\mathit{y})$ =${\mathsf{\phi }\left(\mathit{x}\right)}^t\mathsf{\phi }\left(\mathit{y}\right)$+ eps	Valid, where $\mathsf{\phi }=\mathrm{R}\mathrm{e}\mathrm{l}\mathrm{u}$, and eps is small positive number such as eps=1.e-4
Relu2	k($\mathit{x}$, $\mathit{y})$ =${\mathsf{\phi }\left(\mathit{x}\right)}^t\mathsf{\phi }\left(\mathit{y}\right)$+ eps	Valid, where $\mathsf{\phi }={\mathrm{R}\mathrm{e}\mathrm{l}\mathrm{u}}^2$, and eps is small positive number such as eps=1.e-4
Elu	k($\mathit{x}$, $\mathit{y})$ =${\mathsf{\phi }\left(\mathit{x}\right)}^t\mathsf{\phi }\left(\mathit{y}\right)$	Valid, where $\mathsf{\phi }=\mathrm{E}\mathrm{l}\mathrm{u}+1$

Table 1. Kernels. 

Name	Expression	Type
Exponential	ψ (z)=exp(z)	Universal
Linear	ψ (z)=z	Valid
Geometric Series	ψ (z)=1/(1-z)	Conditionally universal if its input, z, is within in range of (-1,1). This condition is met when z is computed from cosine kernel
Cosine	k($\mathit{x}$, $\mathit{y}$) =< $\mathit{x}$, $\mathit{y}$> /$\sqrt{(\alpha +{‖\mathit{x}‖}^2)(\alpha +{‖\mathbf{y}‖}^2)}$	Valid when $\alpha \ge 0$ such as $\alpha =1.e-3$
Shift Cosine	k($\mathit{x}$, $\mathit{y}$) =(1+z)	Valid when z is computed from cosine kernel
RBF	k($\mathit{x}$, $\mathit{y}$) =exp(${-‖\mathit{x}-\mathit{y}‖}^2/\gamma$)	Universal when $\gamma >0$
Clipped Geometric Series	k($\mathit{x}$, $\mathit{y}$) =1/(1+eps-c(<$\mathit{x}$, $\mathit{y}$>)/M)	Conditionally universal, c(<$\mathit{x}$, $\mathit{y}$>) =M when <$\mathit{x}$, $\mathit{y}$> >M, where M>0, such as M=300, eps is small positive number such as eps=1.e-4
Relu	k($\mathit{x}$, $\mathit{y})$ =${\mathsf{\phi }\left(\mathit{x}\right)}^t\mathsf{\phi }\left(\mathit{y}\right)$+ eps	Valid, where $\mathsf{\phi }=\mathrm{R}\mathrm{e}\mathrm{l}\mathrm{u}$, and eps is small positive number such as eps=1.e-4
Relu2	k($\mathit{x}$, $\mathit{y})$ =${\mathsf{\phi }\left(\mathit{x}\right)}^t\mathsf{\phi }\left(\mathit{y}\right)$+ eps	Valid, where $\mathsf{\phi }={\mathrm{R}\mathrm{e}\mathrm{l}\mathrm{u}}^2$, and eps is small positive number such as eps=1.e-4
Elu	k($\mathit{x}$, $\mathit{y})$ =${\mathsf{\phi }\left(\mathit{x}\right)}^t\mathsf{\phi }\left(\mathit{y}\right)$	Valid, where $\mathsf{\phi }=\mathrm{E}\mathrm{l}\mathrm{u}+1$

There are other kernels, but a comprehensive list is beyond the scope of this paper.

7. Experiments

Experiments were conducted on the Shakespeare character-level dataset [12] using a 6-layer GPT-style Transformer (6 attention heads, 256 context length, 384 embedding dimension). This architecture provides a resource-efficient testbed for our analysis.

Our findings demonstrate that all tested kernels defined in the Table 1, as described in the ‘Kernel Function Selection’ section, enable the model to converge successfully, confirmed by a steady reduction in the training loss. Significantly, even the pure linear kernel proved sufficient for model convergence, establishing a strong and efficient baseline for attention approximation.

In our experiment, we also observed that the model successfully converges even when the query and key vectors are constrained to be identical. This modification implies that a single source of feature vectors is utilized for both functions. A direct consequence of this approach is a 50% reduction in the memory required for storing the key and query vectors, a significant efficiency gain, particularly for models with large sequence lengths where memory complexity can be a bottleneck.

Furthermore, a comparative analysis of kernel functions was conducted. The results indicate that the Radial Basis Function (RBF) kernel, yields a marginal performance improvement over the original exponential kernel.

Specifically, with the Clipped Geometric Series, as defined in Table 1, the model also demonstrates convergence with normalized weights.

8. Conclusion

In this work, we have established a new theoretical foundation for Scaled-Dot-Product Attention (SDPA) by reframing it as a Generalized Memory Model. This perspective, which extends classical distributed memory theory, provides a new lens to understand the core associative properties of attention. Our empirical findings strongly support this model. We demonstrated the robustness of a simple linear kernel, which achieves convergence even without normalization, establishing it as a powerful and efficient baseline for attention approximation. Furthermore, we introduced a novel class of simplified, exponential-free kernels designed to reduce computational demands on hardware lacking specific acceleration.

While our experiments on the Shakespeare dataset provide a strong proof-of-concept, scaling these findings to larger corpora is a crucial next step. Future research should focus on characterizing these kernels’ performance profiles on dedicated hardware accelerators. Ultimately, by connecting modern attention mechanisms to classical memory theory, this research offers both a practical roadmap for developing more efficient sequence models and a deeper theoretical understanding of how they work.