Deep Learning 2.0.1: Mind and Cosmos - Towards Cosmos-Inspired Interpretable Neural Networks

1. Introduction

Deep learning practitioners are often forced into a false dichotomy: to measure alignment (via the dot product) or to measure proximity (via Euclidean distance). Models that require sensitivity to both must rely on complex, multi-layered architectures to approximate this relationship. This work challenges that paradigm by introducing a primitive operator that unifies these concepts.

At the heart of this dichotomy lies the standard model underpinning most deep learning systems: the dot product for linear interaction, followed by a non-linear activation function. The dot product serves as the primary mechanism for measuring similarity and interaction between neural units, a practice dating back to the perceptron’s introduction [1,2,3,4,5,6]. Non-linear activation functions, such as the Rectified Linear Unit (ReLU) [7,8,9], are then applied to enable the network to learn complex patterns, as underscored by the universal approximation theorem [3,4]. Without such non-linearities, a deep stack of layers would mathematically collapse into an equivalent single linear transformation, severely curtailing its representational capacity.

However, this ubiquitous approach has a significant cost: a loss of geometric fidelity and the need for additional components like normalization layers. The dot product itself is a geometrically impoverished measure, primarily capturing alignment while conflating magnitude with direction and often obscuring more complex structural and spatial relationships [10,11,12,13,14]. Furthermore, the way current activation functions achieve non-linearity can exacerbate this issue. For instance, ReLU ($f\left(x\right)=max(0,x)$) maps all negative pre-activations, which can signify a spectrum of relationships from weak dissimilarity to strong anti-alignment, to a single zero output. This thresholding, while promoting sparsity, means the network treats diverse inputs as uniformly orthogonal or linearly independent for onward signal propagation. Such a coarse-graining of geometric relationships leads to a tangible loss of information regarding the degree and nature of anti-alignment or other negative linear dependencies. This information loss, coupled with the inherent limitations of the dot product, highlights a fundamental challenge.

This raises a central question: Can we develop a single computational operator that possesses intrinsic non-linearity while being inherently geometrically aware, thereby preserving geometric fidelity without the need for separate activation functions?

This paper proposes an elegant answer: the Yat-product (pronounced Yat-product), a novel neural operator. The intuition behind the Yat-product is the unification of alignment and proximity, inspired by fundamental principles observed in physical systems, particularly the concept of interaction fields governed by inverse-square laws [15,16,17,18,19,20]. In physics, the strength of interactions (like gravity or electrostatic force) depends not only on intrinsic properties (like mass or charge) but critically on the inverse square of the distance between entities.

To this end, we introduce the Yat-product, defined as:

$$\mathrm{Yat}(\mathbf{w},\mathbf{x}):={\displaystyle \frac{{\langle \mathbf{w},\mathbf{x}\rangle }^2}{{\parallel \mathbf{w}-\mathbf{x}\parallel }^2+ϵ}}$$

(1)

where $\mathbf{w}$ is a weight vector, $\mathbf{x}$ is an input vector, and $ϵ$ is a small positive constant for numerical stability. This operator, inspired by physical inverse-square laws, unifies alignment and proximity in a single, non-linear computation. See Section 3 for a detailed analysis, comparison with standard operators, and information-theoretic interpretation.

The Yat-product is intrinsically non-linear and self-regulating, and we prove that networks built from it are universal approximators (see Section 3 and Appendix G.6).

The Yat-product can be naturally extended to convolutional operations for processing structured data like images. The Yat-Convolution (Yat-Conv) is defined as:

$${\mathrm{Yat}}^{*}(\mathbf{W},\mathbf{X}):={\displaystyle \frac{{\left({\sum }_{i,j}{w}_{ij}{x}_{ij}\right)}^2}{{\sum }_{i,j}{(w_{ij}-x_{ij})}^2+ϵ}}$$

(2)

where $\mathbf{W}$ and $\mathbf{X}$ represent local patches (e.g., a convolutional kernel and an input patch, respectively), and $w_{ij}$ and $x_{ij}$ are their corresponding elements. This formulation allows for patch-wise computation of the Yat-product, integrating its geometric sensitivity into convolutional architectures.

Building upon the Yat-product, we propose Neural-Matter Networks (NMNs) and Convolutional NMNs (CNMNs). NMNs are designed to preserve input topology by leveraging the Yat-product’s geometric awareness and avoiding aggressive, dimension-collapsing non-linearities typically found in standard architectures. In NMNs, each neuron, through its learned weight vector $\mathbf{w}$, effectively defines an interaction field. It "attracts" or responds to input vectors $\mathbf{x}$ based on the dual criteria of learned alignment and spatial proximity, analogous to how bodies with mass create gravitational fields. This approach aims to maintain critical geometric relationships, fostering more interpretable models and robust learning.

The primary contributions of this work are:

• The introduction of the Yat-product, a novel, physics-grounded neural operator that unifies directional sensitivity with an inverse-square proximity measure, designed for geometrically faithful similarity assessment.
• The proposal of Neural-Matter Networks (NMNs), a new class of neural architectures based on the Yat-product, which inherently incorporate non-linearity and are designed to preserve input topology.
• A commitment to open science through the release of all associated code and models under the Affero GNU General Public License.

By reconceiving neural computation through the lens of physical interaction fields, this work seeks to bridge the empirical successes of contemporary machine learning with the structural understanding and interpretability afforded by principles derived from physics.

2. Theoretical Background

2.1. Revisiting Core Computational Primitives and Similarity Measures

The computational primitives used in deep learning are fundamental to how models represent and process information. This section revisits key mathematical operations and similarity measures, such as the dot product, convolution, cosine similarity, and Euclidean distance, that form the bedrock of many neural architectures. We will explore their individual properties and how they contribute to tasks like feature alignment, localized feature mapping, and quantifying spatial proximity. Furthermore, we will delve into the role of neural activation functions in enabling the non-linear transformations crucial for complex pattern recognition. Understanding these core concepts and their inherent characteristics is crucial for appreciating the motivation behind developing novel operators, as explored in this work, that aim to capture more nuanced relationships within data [5].

2.1.1. The Dot Product: A Measure of Alignment

The dot product, or scalar product, remains a cornerstone of neural computation, serving as the primary mechanism for quantifying the interaction between vectors, such as a neuron’s weights and its input. For two vectors $\mathbf{a}=[a_1,a_2,\dots ,a_n]$ and $\mathbf{b}=[b_1,b_2,\dots ,b_n]$, it is defined as:

$$\mathbf{a}·\mathbf{b}=\sum _{i=1}^n{a}_i{b}_i=a_1{b}_1+a_2{b}_2+\dots +a_n{b}_n$$

(3)

Geometrically, the dot product is proportional to the cosine of the angle between the vectors and their Euclidean magnitudes: $\mathbf{a}·\mathbf{b}=\parallel \mathbf{a}\parallel \parallel \mathbf{b}\parallel cos\left(\theta \right)$. Its sign indicates the general orientation (acute, obtuse, or orthogonal angle), and its magnitude reflects the degree of alignment scaled by vector lengths. In machine learning, dot product scores are pervasively used to infer similarity, relevance, or the strength of activation. However, as noted in Section 1, its conflation of magnitude and directional alignment can sometimes obscure more fine-grained geometric relationships, motivating the exploration of operators that offer a more comprehensive assessment of vector interactions.

2.1.2. The Convolution Operator: Localized Feature Mapping

The convolution operator is pivotal in processing structured data, particularly in Convolutional Neural Networks (CNNs). It applies a kernel (or filter) across an input to produce a feature map, effectively an operation on two functions, f (input) and g (kernel), yielding a third that expresses how one modifies the shape of the other. For discrete signals, such as image patches and kernels, it is:

$$(f*g)\left[n\right]=\sum _{m=-\infty }^{\infty }f\left[m\right]g[n-m]$$

(4)

In CNNs, convolution performs several critical roles:

• Feature Detection: Kernels learn to identify localized patterns (edges, textures, motifs) at various abstraction levels.
• Spatial Hierarchy: Stacking layers allows the model to build complex feature representations from simpler ones.
• Parameter Sharing: Applying the same kernel across spatial locations enhances efficiency and translation equivariance.

The core computation within a discrete convolution at a specific location involves an element-wise product sum between the kernel and the corresponding input patch, which is, in essence, a dot product. Consequently, the resulting activation at each point in the feature map reflects the local alignment between the input region and the kernel. If an input patch and a kernel are orthogonal (i.e., their element-wise product sums to zero, akin to a zero dot product if they were vectorized), the convolution output at that position will be zero, indicating no local match for the feature encoded by the kernel. This reliance on dot product-like computations means that standard convolutions primarily assess feature alignment, potentially overlooking other geometric aspects of the data.

2.1.3. Cosine Similarity: Normalizing for Directional Agreement

Cosine similarity refines the notion of alignment by isolating the directional aspect of vector relationships, abstracting away from their magnitudes. It measures the cosine of the angle between two non-zero vectors $\mathbf{A}$ and $\mathbf{B}$:

$$cos\left(\theta \right)={\displaystyle \frac{\mathbf{A}·\mathbf{B}}{\parallel \mathbf{A}\parallel \parallel \mathbf{B}\parallel }}={\displaystyle \frac{{\sum }_{i=1}^n{A}_i{B}_i}{\sqrt{{\sum }_{i=1}^n{A}_i^2}\sqrt{{\sum }_{i=1}^n{B}_i^2}}}$$

(5)

Scores range from -1 (perfectly opposite) to 1 (perfectly aligned), with 0 signifying orthogonality (decorrelation). By normalizing for vector lengths, cosine similarity provides a pure measure of orientation. This is particularly useful when the magnitude of vectors is not indicative of their semantic relationship, such as in document similarity tasks. While it effectively captures directional agreement, it explicitly discards information about vector magnitudes and, like the dot product, does not inherently account for the spatial proximity between the vectors themselves if they are points in a space [13,14].

2.1.4. Euclidean Distance: Quantifying Spatial Proximity

In contrast to measures of alignment, Euclidean distance quantifies the "ordinary" straight-line separation between two points (or vectors) $\mathbf{p}=(p_1,\dots ,p_n)$ and $\mathbf{q}=(q_1,\dots ,q_n)$ in an n-dimensional Euclidean space:

$$d(\mathbf{p},\mathbf{q})=\sqrt{{\sum }_{i=1}^n{(q_i-p_i)}^2}$$

(6)

This metric is fundamental in various machine learning algorithms, including k-Nearest Neighbors and k-Means clustering, and forms the basis of loss functions like Mean Squared Error. Euclidean distance measures dissimilarity based on spatial proximity; a smaller distance implies greater similarity in terms of location within the vector space. Unlike cosine similarity, it is sensitive to vector magnitudes and their absolute positions. However, Euclidean distance alone does not directly convey information about the relative orientation or alignment of vectors, only their nearness.

The distinct characteristics of these foundational measures, alignment (dot product, cosine similarity) versus proximity (Euclidean distance), highlight an opportunity. These foundational measures force a choice: one can measure alignment (dot product, cosine similarity) or spatial proximity (Euclidean distance), but no single, primitive operator in conventional use effectively unifies both. Neural operators that can synergistically combine these aspects, assessing not only if vectors point in similar directions but also if they are close in the embedding space, could offer a richer, more geometrically informed way to model interactions. This perspective underpins the development of the Yat-product introduced in Section 3.

2.2. The Role and Geometric Cost of Non-Linear Activation

While the core computational primitives provide tools to measure similarity and interaction, their inherent linearity limits the complexity of functions they can represent. To overcome this, deep neural networks employ non-linear activation functions. These are the standard method for introducing non-linearity, a necessary step for modeling intricate data patterns. However, this "fix" is imperfect, as it introduces its own set of problems, particularly concerning the preservation of the input data’s geometric integrity. The remarkable expressive power of deep neural networks hinges on their capacity to model complex, non-linear relationships. This ability to approximate any continuous function to an arbitrary degree of accuracy is formally captured by the universal approximation theorem.

Theorem 1 

(Universal Approximation Theorem [3,4,21,22]). Let σ be any continuous, bounded, and nonconstant activation function. Let $I_m$ denote the m-dimensional unit hypercube ${[0,1]}^m$. The space of continuous functions on $I_m$ is denoted by $C\left(I_m\right)$. Then, for any $f\in C\left(I_m\right)$ and any $ϵ>0$, there exists an integer N, real constants $v_i,b_i\in \mathbb{R}$, and real vectors $w_i\in {\mathbb{R}}^m$ for $i=1,\dots ,N$, such that the function $F:I_m\to \mathbb{R}$ defined by

$$F\left(x\right)=\sum _{i=1}^N{v}_i\sigma (w_i^{T}x+b_i)$$

(7)

satisfies $\left|F\right(x)-f(x\left)\right|<ϵ$ for all $x\in I_m$. In simpler terms, a single hidden layer feedforward network with a sufficient number of neurons employing a suitable non-linear activation function can approximate any continuous function on compact subsets of ${\mathbb{R}}^m$ to any desired degree of accuracy.

This theorem underscores the critical role of non-linear activation functions. Without such non-linearities, a deep stack of layers would mathematically collapse into an equivalent single linear transformation, severely curtailing its representational capacity. Activation functions are thus not mere auxiliaries; they are the pivotal components that unlock the hierarchical and non-linear feature learning central to deep learning’s success. They determine a neuron’s output based on its aggregated input, and in doing so, introduce crucial selectivity: enabling the network to preferentially respond to certain patterns while attenuating or ignoring others.

2.2.1. Linear Separability and the Limitations of the Inner Product

The fundamental computation within a single artificial neuron (perceptron) is an affine transformation followed by a non-linear activation function $\sigma$:

$$y=\sigma (\langle \mathbf{w},\mathbf{x}\rangle +b),$$

(8)

where $\mathbf{w}$ is the weight vector, $\mathbf{x}$ is the input vector, and b is the bias term. The decision boundary of this neuron is implicitly defined by the hyperplane where the argument to $\sigma$ is zero:

$$\{\mathbf{x}\in {\mathbb{R}}^d\mid \langle \mathbf{w},\mathbf{x}\rangle +b=0\}.$$

(9)

This hyperplane partitions the input space ${\mathbb{R}}^d$ into two half-spaces. Consequently, a single neuron can only implement linearly separable functions. This is a direct consequence of the linear nature of the inner product, which can only define a linear decision boundary. While this allows for efficient computation, it severely restricts the complexity of functions that can be learned.

A classic counterexample is the XOR function, whose truth table cannot be satisfied by any single linear decision boundary. Specifically, for inputs $\mathbf{x}\in \{(0,0),(0,1),(1,0),(1,1)\}\subset {\mathbb{R}}^2$, there exist no $\mathbf{w}\in {\mathbb{R}}^2$ and $b\in \mathbb{R}$ such that $sign(\langle \mathbf{w},\mathbf{x}\rangle +b)$ matches the XOR output (0, 1, 1, 0 respectively). This limitation stems directly from the linear nature of the inner product operation defining the separating boundary [5].

2.2.2. Non-Linear Feature Space Transformation via Hidden Layers and its Geometric Cost

Multi-layer perceptrons (MLPs) overcome this limitation by cascading transformations. A hidden layer maps the input $\mathbf{x}$ to a new representation $\mathbf{h}$ through a matrix-vector product and an element-wise activation function $\varphi$:

$$\mathbf{h}=\varphi (W\mathbf{x}+\mathbf{b}).$$

(10)

Here, $W\in {\mathbb{R}}^{m\times d}$ is the weight matrix, $\mathbf{b}\in {\mathbb{R}}^m$ is the bias vector, and m is the number of hidden neurons. Each row ${\mathbf{w}}_i^{\top }$ of W corresponds to the weight vector of the i-th hidden neuron, computing $h_i=\varphi (\langle {\mathbf{w}}_i,\mathbf{x}\rangle +b_i)$. This transforms the input space ${\mathbb{R}}^d$ into a feature space ${\mathbb{R}}^m$. The introduction of the non-linear activation function $\varphi$ is what allows the network to learn non-linear decision boundaries. However, this gain in expressive power comes at a cost: the potential loss of geometric fidelity.

2.2.3. Topological Distortions and Information Loss via Activation Functions

While hidden layers using the transformation $\mathbf{h}=\varphi (W\mathbf{x}+\mathbf{b})$ enable the learning of non-linear functions, the introduction of the element-wise non-linear activation function $\varphi$, often crucial for breaking linearity, can significantly alter the topological and geometric structure of the data representation, potentially leading to information loss [5]. This is a critical trade-off: gaining non-linear modeling capability while potentially discarding valuable geometric information.

Consider the mapping $T:{\mathbb{R}}^d\to {\mathbb{R}}^m$ defined by $T\left(\mathbf{x}\right)=\varphi (W\mathbf{x}+\mathbf{b})$. The affine part, $A\left(\mathbf{x}\right)=W\mathbf{x}+\mathbf{b}$, performs a linear transformation (rotation, scaling, shear, projection) followed by a translation. While this affine map distorts metric properties (distances and angles, unless W is proportional to an orthogonal matrix), it preserves basic topological features like connectedness and maps lines to lines (or points) [5].

However, the subsequent application of a typical non-linear activation $\varphi$ element-wise often leads to more drastic topological changes:

• Non-Injectivity and Collapsing Regions: Many common activation functions render the overall mapping T non-injective.
  • ReLU ($\varphi \left(z\right)=max(0,z)$): Perhaps the most prominent example. For each hidden neuron i, the entire half-space defined by $\{\mathbf{x}\in {\mathbb{R}}^d\mid \langle {\mathbf{w}}_i,\mathbf{x}\rangle +b_i\le 0\}$ is mapped to $h_i=0$. Distinct points ${\mathbf{x}}_1,{\mathbf{x}}_2$ within this region, potentially far apart, become indistinguishable along the i-th dimension of the hidden space. This constitutes a significant loss of information about the relative arrangement of data points within these collapsed regions. The mapping is fundamentally many-to-one. For instance, consider two input vectors that are anti-aligned with a neuronś weight vector to different degrees, one strongly and one weakly. A ReLU activation function would map both resulting negative dot products to zero, rendering their distinct geometric opposition indistinguishable to subsequent layers. This information is irretrievably discarded.
  • Sigmoid/Tanh: While smooth, these functions saturate. Inputs ${\mathbf{z}}_1=A\left({\mathbf{x}}_1\right)$ and ${\mathbf{z}}_2=A\left({\mathbf{x}}_2\right)$ that are far apart but both fall into the saturation regime (e.g., large positive or large negative values) will map to ${\mathbf{h}}_1\approx {\mathbf{h}}_2$. This ’squashing’ effect can merge distinct clusters from the input space if they map to saturated regions in the hidden space, again losing discriminative information and distorting the metric structure.
• Distortion of Neighborhoods: The relative distances between points can be severely distorted. Points close in the input space ${\mathbb{R}}^d$ might be mapped far apart in ${\mathbb{R}}^m$, or vice-versa (especially due to saturation or the zero-region of ReLU). This means the local neighborhood structure is not faithfully preserved. Formally, the mapping T is generally not a homeomorphism onto its image, nor is it typically bi-Lipschitz (which would provide control over distance distortions).

Figure 1. Illustration of how non-linear activation functions can distort the geometric structure of the input data manifold, leading to potential information loss. The original manifold (left) is transformed into a distorted representation after applying a non-linear activation functions.

Figure 1. Illustration of how non-linear activation functions can distort the geometric structure of the input data manifold, leading to potential information loss. The original manifold (left) is transformed into a distorted representation after applying a non-linear activation functions.

While these distortions are precisely what grant neural networks their expressive power to warp the feature space and create complex decision boundaries, they come at the cost of potentially discarding information present in the original geometric configuration of the data. The network learns which information to preserve and which to discard based on the optimization objective, but the mechanism relies on potentially non-smooth or non-injective transformations introduced by $\varphi$. This highlights the conflation of magnitude and direction in the dot product, the information loss from activation functions, and the lack of a unified measure for proximity and alignment, setting the stage for the Yat-product.

3. Methodology: A Framework for Geometry-Aware Computation

3.1. The Yat-Product: A Unified Operator for Alignment and Proximity

The methodological innovations presented in this work are fundamentally rooted in the Yat-product, introduced in Section 1. This single operator serves as the foundation for subsequent layers and networks.

The Yat-product is formally defined as $Yat(\mathbf{w},\mathbf{x})={\displaystyle \frac{{\left({\mathbf{w}}^{\top }\mathbf{x}\right)}^2}{{\parallel \mathbf{w}-\mathbf{x}\parallel }^2+ϵ}}$ [19,20]. It exhibits a unique form of non-linearity. Unlike conventional activation functions (e.g., ReLU, sigmoid) which are often applied as separate, somewhat heuristic, transformations to introduce non-linearity after a linear operation, the non-linearity in the Yat-product arises directly from its mathematical structure. It is a function of the squared dot product (capturing alignment) and the inverse squared Euclidean distance (capturing proximity) between the weight vector $\mathbf{w}$ and the input vector $\mathbf{x}$. This formulation provides a rich, explainable non-linearity based on fundamental geometric and algebraic relationships, rather than an imposed, "artificial" non-linear mapping. The interaction between the numerator and the denominator allows for complex responses that are inherently tied to the geometric interplay of the input vectors.

As visualized in Figure 2, the Yat-product creates a potential well around the weight vector $\mathbf{w}$, reflecting both alignment and proximity.

3.2. Comparison to Standard Similarity and Distance Metrics

To further appreciate the unique characteristics of the Yat-product, it is instructive to compare it with other common similarity or distance metrics [13,14,23,24]:

• Dot Product (${\mathbf{w}}^{\top }\mathbf{x}$): The dot product measures the projection of one vector onto another, thus capturing both alignment and magnitude. A larger magnitude in either vector, even with constant alignment, leads to a larger dot product. While useful, its direct sensitivity to magnitude can sometimes overshadow the pure geometric alignment.
• Cosine Similarity (${\displaystyle \frac{{\mathbf{w}}^{\top }\mathbf{x}}{\parallel \mathbf{w}\parallel \parallel \mathbf{x}\parallel }}$): Cosine similarity normalizes the dot product by the magnitudes of the vectors, yielding the cosine of the angle between them. This makes it purely a measure of alignment, insensitive to vector magnitudes. However, as pointed out, this means it loses information about true distance or scale; two vectors can have perfect cosine similarity (e.g., value of 1) even if one is very distant from the other, as long as they point in the same direction.
• Euclidean Distance ($\parallel \mathbf{w}-\mathbf{x}\parallel$): This metric computes the straight-line distance between the endpoints of two vectors. It is a direct measure of proximity. However, it does not inherently capture alignment. For instance, if $\mathbf{w}$ is a reference vector, all vectors $\mathbf{x}$ lying on the surface of a hypersphere centered at $\mathbf{w}$ will have the same Euclidean distance to $\mathbf{w}$, regardless of their orientation relative to $\mathbf{w}$.
• Yat-Product (${\mathcal{K}}_{Y}at(\mathbf{w},\mathbf{x})={\displaystyle \frac{{\left({\mathbf{w}}^{\top }\mathbf{x}\right)}^2}{{\parallel \mathbf{w}-\mathbf{x}\parallel }^2+ϵ}}$): The Yat-product uniquely combines aspects of both alignment and proximity in a non-linear fashion. The numerator, ${\left({\mathbf{w}}^{\top }\mathbf{x}\right)}^2$, emphasizes strong alignment (being maximal when vectors are collinear and zero when orthogonal) and is sensitive to magnitude. The denominator, ${\parallel \mathbf{w}-\mathbf{x}\parallel }^2+ϵ$, heavily penalizes large distances between $\mathbf{w}$ and $\mathbf{x}$. This synergy allows the Yat-product to be highly selective. It seeks points that are not only well-aligned with the weight vector $\mathbf{w}$ but also close to it. Unlike cosine similarity, it distinguishes between aligned vectors at different distances. Unlike Euclidean distance alone, it differentiates based on orientation. This combined sensitivity allows the Yat-product to identify matches with a high degree of specificity, akin to locating a point with "atomic level" precision, as it requires both conditions (alignment and proximity) to be met strongly for a high output.

Beyond its geometric interpretation, the Yat-product has a profound connection to information theory when its arguments are probability distributions [25,26,27] (see Appendix G.7 for formal results). In this context, it can be viewed as a signal-to-noise ratio, where the "signal" ${(\mathbf{p}·\mathbf{q})}^2$ measures distributional alignment and the "noise" ${\parallel \mathbf{p}-\mathbf{q}\parallel }^2$ quantifies their dissimilarity.

The Yat-product’s ability to discern between aligned vectors at varying distances, as well as its sensitivity to the angle between vectors, is illustrated in Figure 2 and Figure 3. The vector field generated by the Yat-product can be visualized as a potential well around the weight vector $\mathbf{w}$, where the strength of the interaction diminishes with distance, akin to gravitational or electrostatic fields. This visualization underscores how the Yat-product captures both alignment and proximity in a unified manner. This combined sensitivity is crucial for tasks where both the orientation and the relative position of features are important.

3.3. Design Philosophy: Intrinsic Non-Linearity and Self-Regulation

A central hypothesis underpinning our methodological choices is that the Yat-product (Section 1) possesses inherent non-linearity and self-regulating properties that can reduce or eliminate the need for conventional activation functions (e.g., ReLU, sigmoid, GeLU) and normalization layers (e.g., Batch Normalization, Layer Normalization).

This philosophy recontextualizes the fundamental components of neural computation. Neuron weights ($\mathbf{w}$) and input signals ($\mathbf{x}$) are not merely operands in a linear transformation followed by a non-linear activation; instead, they are conceptualized as co-equal vector entities inhabiting a shared, high-dimensional feature manifold. Within this framework, each vector can be viewed as an analogue to a fundamental particle or feature vector, with its constituent dimensions potentially encoding excitatory, inhibitory, or neutral characteristics relative to other entities in the space. The Yat-product (Section 1) then transcends simple similarity assessment; it functions as a sophisticated interaction potential, ${\mathcal{K}}_{Y}at(\mathbf{w},\mathbf{x})={\displaystyle \frac{{\left({\mathbf{w}}^{\top }\mathbf{x}\right)}^2}{{\parallel \mathbf{w}-\mathbf{x}\parallel }^2+ϵ}}$, quantifying the ’field effects’ between these vector entities. This interaction is reminiscent of n-body problems in physics. In machine learning, it draws parallels with, yet distinctively evolves from, learned metric spaces in contrastive learning, particularly those employing a triplet loss framework. While triplet loss aims to pull positive pairs closer and push negative pairs apart in the embedding space, our Yat-product seeks a more nuanced relationship: ’positive’ interactions (high Yat-product value) require both strong alignment (high ${\left({\mathbf{w}}^{\top }\mathbf{x}\right)}^2$) and close proximity (low ${\parallel \mathbf{w}-\mathbf{x}\parallel }^2$). Conversely, ’negative’ or dissimilar relationships are not merely represented by distance, but more significantly by orthogonality (leading to a vanishing numerator), which signifies a form of linear independence and contributes to the system’s capacity for true non-linear discrimination. Crucially, the non-linearity required for complex pattern recognition is not an external imposition (e.g., via a separate activation function) but is intrinsic to this interaction potential. The interplay between the squared dot product (alignment sensitivity) and the inverse squared Euclidean distance (proximity sensitivity) in its formulation directly sculpts a complex, non-linear response landscape without recourse to auxiliary functions.

Furthermore, this conceptualization of the Yat-product as an intrinsic interaction potential suggests inherent self-regulating properties. The distance-sensitive denominator, ${\parallel \mathbf{w}-\mathbf{x}\parallel }^2+ϵ$, acts as a natural dampening mechanism. As the ’distance’ (dissimilarity in terms of position) between interacting vector entities $\mathbf{w}$ and $\mathbf{x}$ increases, the strength of their interaction, and thus the resultant activation, diminishes quadratically. This behavior is hypothesized to inherently curtail runaway activations and stabilize learning dynamics by ensuring that responses are localized and bounded. Such intrinsic stabilization contrasts sharply with conventional approaches that rely on explicit normalization layers (e.g., Batch Normalization, Layer Normalization) to manage activation statistics post-hoc. These layers, while effective, introduce additional computational overhead, can obscure direct input-output relationships, and sometimes complicate the theoretical analysis of network behavior. The Yat-product’s formulation, therefore, offers a pathway to architectures where regulatory mechanisms are embedded within the primary computational fabric of the network.

The inherent non-linearity of the Yat-product, coupled with the self-regulating properties suggested by its formulation (and formally proven in Appendix G.3), are central to our hypothesis that it can form the basis of powerful and robust neural architectures. These intrinsic characteristics open avenues for simplifying network design, potentially reducing reliance on or even eliminating conventional activation functions and normalization layers.

Figure 4. The Vectoverse: Conceptualizing neural computation where weight vectors ($\mathbf{w}$) and input vectors ($\mathbf{x}$) are akin to fundamental particles (vectoms). The interaction force between them is quantified by the Yat-product, which measures their alignment and proximity, defining a field of influence.

Figure 4. The Vectoverse: Conceptualizing neural computation where weight vectors ($\mathbf{w}$) and input vectors ($\mathbf{x}$) are akin to fundamental particles (vectoms). The interaction force between them is quantified by the Yat-product, which measures their alignment and proximity, defining a field of influence.

3.4. Core Building Blocks

Now, we show how the Yat-product is operationalized into reusable layers.

3.4.1. The Neural Matter Network (NMN) Layer

The first and simplest application of the Yat-product is in Neural-Matter Network (NMN) layers. These networks represent a departure from traditional Multi-Layer Perceptrons (MLPs) by employing the non-linear, spatially-aware ${\mathcal{K}}_{Y}at$-kernel (derived from the Yat-product, see Section 3.1) as the primary interaction mechanism, instead of the conventional linear projection ($\langle \mathbf{w},\mathbf{x}\rangle$).

An NMN layer transforms an input vector ($\mathbf{x}\in {\mathbb{R}}^d$) into an output (here, we consider a scalar output h for simplicity, extendable to vector outputs $\mathbf{h}$ by aggregating the influence of multiple "neural matter" units). Each unit i is defined by a weight vector (${\mathbf{w}}_i\in {\mathbb{R}}^d$) (acting as a positional anchor or prototype) and a bias term ($b_i\in \mathbb{R}$). The layer output is computed as:

$$h\left(\mathbf{x}\right)=\left(s·\sum _{i=1}^n\left({\mathcal{K}}_{Y}at({\mathbf{w}}_i,\mathbf{x})+b_i\right)\right)=\left(s·\sum _{i=1}^n\left({\displaystyle \frac{{\left({\mathbf{w}}_i^{\top }\mathbf{x}\right)}^2}{\parallel {\mathbf{w}}_i{-\mathbf{x}\parallel }^2+ϵ}}+b_i\right)\right)$$

where:

• ${\mathbf{w}}_i$ is the weight vector of the i-th NMN unit.
• $b_i$ is the bias term for the i-th NMN unit.
• ${\mathcal{K}}_{Y}at({\mathbf{w}}_i,\mathbf{x})$ represents the Yat-product between the weight vector ${\mathbf{w}}_i$ and the input $\mathbf{x}$.
• n is the number of NMN units in the layer.
• s is a scaling factor.

This formulation allows each NMN unit to respond based on both alignment and proximity to its learned weight vector.

A key theoretical guarantee for NMNs is their capacity for universal function approximation [3,4,21,22,28,29]. This is significant because, unlike traditional neural networks that depend on separate, often heuristically chosen, activation functions (e.g., ReLU, sigmoid) to introduce non-linearity, the approximation power of NMNs is an intrinsic property of the ${\mathcal{K}}_{Y}at$-kernel itself. This finding validates the Yat-product as a self-contained computational primitive powerful enough to form the basis of expressive neural architectures, distinguishing NMNs from classical MLP-based designs and supporting our core hypothesis that effective, geometry-aware computation is possible without separate activation functions [10,11,12].

3.4.2. Convolutional Neural-Matter Networks (CNMNs) and the Yat-Convolution Layer

To extend the principles of Neural-Matter Networks (NMNs) (Section 3.4.1) and the Yat-product (Section 1) to spatially structured data like images, we introduce the Yat-Convolution (Yat-Conv) layer. This layer adapts the Yat-product to operate on local receptive fields, analogous to standard convolutional layers. The Yat-Conv operation is defined as:

$${\left(\mathrm{Yat}-\mathrm{Conv}(K,I)\right)}_{i,j}=Ya{t}^{*}(K,I_{i,j})={\displaystyle \frac{{\langle K,I_{i,j}\rangle }^2}{\parallel K-I_{i,j}{\parallel }^2+ϵ}}$$

(11)

where K is the convolutional kernel and $I_{i,j}$ is the input patch at location $(i,j)$ corresponding to the receptive field of the kernel.

3.4.3. The Yat-Attention Mechanism

To extend the Yat-product’s application to sequence modeling, we propose the Yat-Attention mechanism. This mechanism serves as an alternative to the standard scaled dot-product attention found in transformer architectures by replacing the dot product used for calculating query-key similarity with the Yat-product (Section 3.1). Given Query (Q), Key (K), and Value (V) matrices, Yat-Attention is computed as:

$$\mathrm{Yat}-\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(s·\left(QYat{K}^T\right)\right)V$$

(12)

where the operation $QYat{K}^T$ signifies applying the Yat-product element-wise between query and key vectors (e.g., the $(i,j)$-th element is $Yat(q_i,k_j)$), and s is a scaling factor.

3.5. Architectural Implementations

The development of architectures like AetherResNet and AetherGPT without standard components (like separate activation and normalization layers) is a deliberate effort to test the hypothesis outlined in Section 3.3. Key architectural distinctions driven by this philosophy include:

• Fundamental Operator Replacement: The standard dot product is replaced by the Yat-product. This is manifested as Yat-Convolution (Equation 11) in convolutional networks and Yat-Attention (Equation 12) in transformer-based models.
• Feed-Forward Networks (FFNs): The FFNs within are constructed using NMN layers (Section 3.4.1) without explicit non-linear activation functions.
• Omission of Standard Layers: Consistent with this design philosophy, explicit activation functions and standard normalization layers are intentionally omitted.

Additionally, in all NMN-based architectures, we use a scaling factor $s={\left({\displaystyle \frac{n}{log(1+n)}}\right)}^{\alpha }$, where n is the number of NMN units and $\alpha$ is a learnable parameter. This scaling is designed to adaptively control the overall magnitude of the layer outputs as a function of network width.

By minimizing reliance on these traditional layers, we aim to explore simpler, potentially more efficient, and interpretable models where the primary computational operator itself handles these crucial aspects of neural processing. Furthermore, this principle of substituting the dot product with the Yat-product is not limited to the architectures presented and holds potential for enhancing other neural network paradigms.

3.5.1. Convolutional NMNs:

AetherResNet is a Convolutional Neural-Matter Network (CNMN) built by replacing all standard convolutions in a ResNet18 architecture with the Yat-Conv layers. Building upon the Yat-Conv layer, CNMNs adapt conventional convolutional architectures by employing the Yat-Conv layer as the primary feature extraction mechanism. The core idea is to leverage the geometric sensitivity and inherent non-linearity of the Yat-product within deep convolutional frameworks. Consistent with the philosophy of Section 3.3, AetherResNet omits Batch Normalization and activation functions [9,30,31,32]. The design relies on the hypothesis that the Yat-product itself provides sufficient non-linearity and a degree of self-regulation.

In each CNMN residual block, we use a YatConv layer (with input dimension n and output dimension m) followed by a linear Conv layer (with input and output dimension m), without any activation functions or normalization layers (see Figure A15).

3.5.2. YatFormer: AetherGPT

AetherGPT is a YatFormer model that uses Yat-Attention (from Section 3.4.3) for sequence interaction and NMN layers (from Section 3.4.1) in its feed-forward blocks. Building upon the Yat-Attention mechanism, which forms its cornerstone, we introduce YatFormer, a family of transformer-based models. As a specific instantiation for our investigations, we developed AetherGPT. This model adapts the architectural principles of GPT-2. Again, following the philosophy of Section 3.3, it omits standard normalization and activation layers.

In YatFormer/AetherGPT, we remove the projection layer after the attention mechanism, as the dot product between the attention map and the value matrix V already serves as a linear projection (see Fig. A16). Furthermore, in the MLP block, we do not multiply the first layer’s width by 4 as in standard transformers. Instead, we use a YatNMN layer with input and output dimensions equal to the embedding dimension, followed by a linear layer, also with input and output dimensions equal to the embedding dimension (see Fig. A17).

3.6. Output Processing for Non-Negative Scores

The Yat-product and its derivatives, such as the ${\mathcal{K}}_{Y}at$-kernel, naturally yield non-negative scores. In many machine learning contexts, particularly when these scores need to be interpreted as probabilities, attention weights, or simply normalized outputs, it is essential to apply a squashing function to map them to a desired range (e.g., [0, 1] or ensuring a set of scores sum to 1).

Squashing functions for non-negative scores can be broadly categorized into two types:

• Competitive (Vector-Normalizing) Functions: These functions normalize a set of scores collectively, producing a distribution over the vector. Each output depends on the values of all dimensions, allowing for competitive interactions among them. This is useful for attention mechanisms or probability assignments where the sum of outputs is meaningful.
• Individualistic (Per-Dimension) Functions: These functions squash each score independently, without reference to other values in the vector. Each output depends only on its corresponding input, making them suitable for bounding or interpreting individual activations.

Traditional squashing functions, however, present challenges when applied to non-negative inputs:

• Standard Sigmoid Function ($\sigma \left(x\right)={\displaystyle \frac{1}{1+e^{-x}}}$): When applied to non-negative inputs ($x\ge 0$), the standard sigmoid function produces outputs in the range $[0.5,1)$. The minimum value of $0.5$ for $x=0$ renders it unsuitable for scenarios where small non-negative scores should map to values close to 0.
• Standard Softmax Function ($\mathrm{softmax}{\left(\mathbf{x}\right)}_i={\displaystyle \frac{e^{x_i}}{{\sum }_j{e}^{x_j}}}$): The use of the exponential function in softmax can lead to hard distributions, where one input value significantly dominates the output, pushing other probabilities very close to zero. While this is often desired for classification, it can be too aggressive if a softer assignment of probabilities or attention is preferred. Additionally, softmax can suffer from numerical instability for large input values due to the exponentials.

Given these limitations and the non-negative nature of Yat-product scores, we consider alternative squashing functions more suited to this domain:

• softermax (Competitive): This function normalizes a score $x_k$ (optionally raised to a power $n>0$) relative to the sum of a set of non-negative scores $\left\{x_i\right\}$ (each raised to n), with a small constant $ϵ>0$ for numerical stability. It is defined as:

  $${\mathrm{softermax}}_n(x_k,\left\{x_i\right\})={\displaystyle \frac{x_k^n}{ϵ+{\sum }_i{x}_i^n}}$$

  (13)
  Unlike softmax, softermax does not use exponentials, which avoids numerical instability for large inputs and provides a more direct, interpretable translation of the underlying scores into a normalized distribution. The power n controls the sharpness of the distribution: $n=1$ recovers the original Softermax, while $n>1$ makes the distribution harder (more peaked), and $0<n<1$ makes it softer.
• soft-sigmoid (Individualistic): This function squashes a single non-negative score $x\ge 0$ (optionally raised to a power $n>0$) into the range $[0,1)$. It is defined as:

  $${\mathrm{soft}-\mathrm{sigmoid}}_n\left(x\right)={\displaystyle \frac{x^n}{1+x^n}}$$

  (14)
  The power n modulates the softness: higher n makes the function approach zero faster for large x, while $n<1$ makes the decay slower.
• soft-tanh (Individualistic): This function maps a non-negative score $x\ge 0$ (optionally raised to a power $n>0$) to the range $[-1,1)$ by linearly transforming the output of soft-sigmoid. It is defined as:

  $${\mathrm{soft}-\tan \mathrm{h}}_n\left(x\right)=2·({\mathrm{soft}-\mathrm{sigmoid}}_n\left(x\right)-\frac{1}{2})={\displaystyle \frac{x^n-1}{1+x^n}}$$

  (15)
  The power n again controls the transition sharpness: higher n makes the function approach $-1$ more quickly for large x.

Figure 5. Visualization of the softermax, soft-sigmoid, and soft-tanh functions. These functions are designed to handle non-negative inputs from the Yat-product and its derivatives, providing appropriate squashing mechanisms that maintain sensitivity across the range of non-negative inputs.

Figure 5. Visualization of the softermax, soft-sigmoid, and soft-tanh functions. These functions are designed to handle non-negative inputs from the Yat-product and its derivatives, providing appropriate squashing mechanisms that maintain sensitivity across the range of non-negative inputs.

These functions are particularly well-suited for the outputs of Yat-product-based computations, as they maintain sensitivity across the range of non-negative inputs while avoiding the pitfalls of standard activation functions [7,8,9].

The main role of these squashing functions can be categorized into two main categories:

• Collective Communication and Space Splitting: The softermax function allows for a comparative analysis of scores, reflecting their orthogonality and spatial proximity to an input vector. A higher score indicates that a vector is more aligned and closer to the input, while a lower score suggests greater orthogonality. This facilitates a competitive interaction where vectors vie for influence based on their geometric relationship with the input. The power parameter n, analogous to the temperature in softmax, controls the sharpness of the gravitational potential well’s slope.
• Individual Score Squashing: The soft-sigmoid and soft-tanh functions are used to squash individual non-negative scores into a bounded range, typically $[0,1)$ for soft-sigmoid and $[-1,1)$ for soft-tanh. They are particularly useful when the output needs to be interpreted as a probability or when a bounded response is required, as each score is processed independently of the others. The power parameter controls the steepness of the function, while the minimum value can be interpreted as an orthogonality score.

3.7. Mathematical Guarantees of the Yat-Product and NMNs

The Yat-product and the resulting Neural-Matter Networks (NMNs) are supported by several key mathematical properties, each formally proven in the appendices:

• Mercer Kernel Property: The Yat-product is a symmetric, positive semi-definite Mercer kernel, enabling its use in kernel-based learning methods (see Appendix G.2).
• Universal Approximation: NMNs with Yat-product activations can approximate any continuous function on a compact set, establishing their expressive power (see Appendix G.6).
• Self-Regulation: The output of a Yat-product neuron is naturally bounded and converges to a finite value as input magnitude increases, ensuring stable activations (see Appendix G.3).
• Stable Gradient: The gradient of the Yat-product with respect to its input vanishes for distant inputs, preventing large, destabilizing updates from outliers (see Appendix G.5).
• Information-Theoretic Duality: The Yat-product unifies geometric and information-theoretic notions of similarity and orthogonality, with formal theorems connecting it to KL divergence and cross-entropy (see Appendix G.7).

4. Results and Discussion

The Yat-product’s non-linearity is not merely a mathematical curiosity; it has practical implications for neural computation. By integrating alignment and proximity into a single operator, the Yat-product allows for more nuanced feature learning. It can adaptively respond to inputs based on their geometric relationships with learned weight vectors, enabling the network to capture complex patterns without the need for separate activation functions.

Consider the classic XOR problem, which is not linearly separable and thus cannot be solved by a single traditional neuron (linear perceptron). The inputs are $(0,0)\to 0$, $(0,1)\to 1$, $(1,0)\to 1$, and $(1,1)\to 0$. A single Yat-product unit can, however, solve this. Let the weight vector be $\mathbf{w}={[1,-1]}^{\top }$.

For $\mathbf{x}={[0,0]}^{\top }$: ${\mathbf{w}}^{\top }\mathbf{x}=0$, so ${\mathcal{K}}_{Y}at(\mathbf{w},\mathbf{x})=0$.

For $\mathbf{x}={[1,1]}^{\top }$: ${\mathbf{w}}^{\top }\mathbf{x}=1-1=0$, so ${\mathcal{K}}_{Y}at(\mathbf{w},\mathbf{x})=0$.

For $\mathbf{x}={[0,1]}^{\top }$: ${\mathbf{w}}^{\top }\mathbf{x}=-1$. ${\parallel \mathbf{w}-\mathbf{x}\parallel }^2={\parallel {[1,-2]}^{\top }\parallel }^2=5$. So ${\mathcal{K}}_{Y}at(\mathbf{w},\mathbf{x})={\displaystyle \frac{{(-1)}^2}{5+ϵ}}={\displaystyle \frac{1}{5+ϵ}}>0$.

For $\mathbf{x}={[1,0]}^{\top }$: ${\mathbf{w}}^{\top }\mathbf{x}=1$. ${\parallel \mathbf{w}-\mathbf{x}\parallel }^2={\parallel {[0,-1]}^{\top }\parallel }^2=1$. So ${\mathcal{K}}_{Y}at(\mathbf{w},\mathbf{x})={\displaystyle \frac{1^2}{1+ϵ}}={\displaystyle \frac{1}{1+ϵ}}>0$.

Thus, the Yat-product unit with an appropriate weight vector (such as one where components have opposite signs, reflecting the XOR logic) naturally separates the XOR patterns, effectively acting as a mathematical kernel. We have formally proven that the Yat-product is a valid Mercer kernel in Appendix G.2 [33,34,35,36,37,38].

To understand its behavior during learning, we analyze its gradient. A key property for stable training is that the gradient with respect to the input, ${\nabla }_{\mathbf{x}}{\mathcal{K}}_{\mathrm{Yat}}$, diminishes as the input $\mathbf{x}$ moves far from the weight vector $\mathbf{w}$. This ensures that distant outliers do not cause large, destabilizing updates. We have formally proven this property in Appendix G.5, demonstrating that ${lim}_{\parallel \mathbf{x}\parallel \to \infty }\parallel {\nabla }_{\mathbf{x}}{\mathcal{K}}_{\mathrm{Yat}}(\mathbf{w},\mathbf{x})\parallel =0$.

The presence of $ϵ$ in the denominator ensures that the derivative remains well-defined, avoiding division by zero and contributing to numerical stability. This contrasts with activation functions like ReLU, which have a derivative of zero for negative inputs, potentially leading to "dead neurons." The smooth and generally non-zero gradient of the Yat-product is hypothesized to contribute to more stable and efficient learning dynamics, reducing the reliance on auxiliary mechanisms like complex normalization schemes. The non-linearity is thus not an add-on but an intrinsic property derived from the direct mathematical interaction of vector projection (alignment, via the ${\left({\mathbf{w}}^{\top }\mathbf{x}\right)}^2$ term) and vector distance (proximity, via the ${\parallel \mathbf{w}-\mathbf{x}\parallel }^2+ϵ$ term). This provides a mathematically grounded basis for feature learning, as the unit becomes selectively responsive to inputs that exhibit specific geometric relationships, both in terms of angular alignment and spatial proximity, to its learned weight vector $\mathbf{w}$. Consequently, $\mathbf{w}$ can be interpreted as a learned feature template or prototype that the unit is tuned to detect, with the Yat-product quantifying the degree of match in a nuanced, non-linear fashion.

The gradient of the Yat-product, being responsive across the input space, actively pushes the neuron’s weights away from configurations that would lead to a zero output (neuron death, e.g., at an input of $[0,0]$ for this problem if weights were also near zero). This contrasts with a simple dot product neuron where the gradient might vanish or lead to a global minimum at zero output for certain problems. For instance, when considering gradient-based optimization, the loss landscape "seen" by the Yat-product neuron in the XOR context would exhibit a peak or high loss at $[0,0]$ (if that were the target for non-zero outputs), encouraging weights to move towards a state that correctly classifies. Conversely, a simple dot product neuron might present a loss landscape where a gradient-based optimizer could find a stable (but incorrect) minimum at zero output. This ability to avoid such dead zones and actively shape the decision boundary makes it helpful to solve problems like XOR with a single unit, leveraging its inherent non-linearity as a mathematical kernel.

Figure 6. Comparison of the loss landscape for a simple dot product neuron and a Yat-product neuron. The dot product neuron has a stable minimum at zero output which doesn’t solve the xor problem and cause the neuron death, while the Yat-product neuron resides in a valley of orthogonality, allowing it to avoid dead zones and actively shape the decision boundary. This illustrates the Yat-product’s ability to solve problems like XOR with a single unit, leveraging its inherent non-linearity as a mathematical kernel.

Figure 6. Comparison of the loss landscape for a simple dot product neuron and a Yat-product neuron. The dot product neuron has a stable minimum at zero output which doesn’t solve the xor problem and cause the neuron death, while the Yat-product neuron resides in a valley of orthogonality, allowing it to avoid dead zones and actively shape the decision boundary. This illustrates the Yat-product’s ability to solve problems like XOR with a single unit, leveraging its inherent non-linearity as a mathematical kernel.

Conceptually, the decision boundary or vector field generated by a simple dot product neuron is linear, forming a hyperplane that attempts to separate data points. In contrast, the Yat-product generates a more complex, non-linear vector field. This field can be visualized as creating a series of potential wells or peaks centered around the weight vector $\mathbf{w}$, with the strength of influence decaying with distance. The condition ${\mathbf{w}}^{\top }\mathbf{x}=0$ defines a "valley" of zero output where vectors are orthogonal to the weight vector. This structure allows for more nuanced and localized responses, akin to a superposition of influences rather than a single linear division, enabling the capture of intricate patterns in the data.

4.1. Your Neuron is a secret Vortex

We begin by analyzing the fundamental learning dynamics that emerge in both conventional and our proposed architectures. In artificial intelligence, competitive learning manifests in various forms, whether through linear classification using dot products or clustering using Euclidean distances. Both approaches involve partitioning the feature space between neurons, which can be conceptualized as prototype learning where each neuron claims a territorial “field” in the representation space.

Figure 7. Comparison of decision boundaries formed by a conventional linear model (left) and our proposed Yat-product method (right). The conventional model’s prototypes grow unbounded, while our method learns more representative prototypes that better capture class distributions.

Figure 7. Comparison of decision boundaries formed by a conventional linear model (left) and our proposed Yat-product method (right). The conventional model’s prototypes grow unbounded, while our method learns more representative prototypes that better capture class distributions.

In this experiment, we analyze the learning dynamics of a linear model on a synthetic dataset, comparing the formation of decision boundaries by conventional neurons employing standard dot products with those generated by our proposed Yat-product method.

In a conventional linear model, the logit for each class i is computed as:

$$z_i={\mathbf{w}}_i^T\mathbf{x}$$

(16)

where ${\mathbf{w}}_i$ is the weight vector (prototype) for class i, and $\mathbf{x}$ is the input vector. The softmax function then normalizes these logits into probabilities:

$$p_i={\displaystyle \frac{exp\left(z_i\right)}{{\sum }_{j=1}^{C}exp\left(z_j\right)}}$$

(17)

The decision boundary between any two classes i and j forms a linear hyperplane defined by:

$${({\mathbf{w}}_i-{\mathbf{w}}_j)}^T\mathbf{x}=0$$

(18)

During training via gradient descent, each prototype ${\mathbf{w}}_i$ is updated to maximize its alignment with the data distributions of its assigned class. This optimization process often leads to an unbounded increase in prototype magnitudes, as $\parallel {\mathbf{w}}_i\parallel \to \infty$ directly amplifies the logit $z_i$, thereby increasing the model’s confidence. However, the decision boundaries themselves remain linear hyperplanes, creating rigid geometric separations in the feature space.

In contrast, the non-linear Yat-product allows neurons to learn more representative prototypes for each class, leading to the formation of more nuanced decision boundaries. For the Yat-product, the response of neuron i to input $\mathbf{x}$ is given by:

$$z_i=\mathrm{Yat}({\mathbf{w}}_i,\mathbf{x})={\displaystyle \frac{{\langle {\mathbf{w}}_i,\mathbf{x}\rangle }^2}{\parallel {\mathbf{w}}_i{-\mathbf{x}\parallel }^2+ϵ}}$$

(19)

This formulation embodies the signal-to-noise ratio interpretation established in our theoretical framework (Appendix G.7), where the squared dot product ${\langle {\mathbf{w}}_i,\mathbf{x}\rangle }^2$ represents the "signal" of distributional alignment, and $\parallel {\mathbf{w}}_i{-\mathbf{x}\parallel }^2$ quantifies the "noise" of dissimilarity. The Yat-product thus provides a principled geometric measure that balances similarity and proximity in a theoretically grounded manner.

Similarly to conventional neurons, the Yat-product outputs are normalized using the softmax function:

$$p_i={\displaystyle \frac{exp\left(z_i\right)}{{\sum }_{j=1}^{C}exp\left(z_j\right)}}={\displaystyle \frac{exp\left(\frac{{\langle {\mathbf{w}}_i,\mathbf{x}\rangle }^2}{\parallel {\mathbf{w}}_i{-\mathbf{x}\parallel }^2+ϵ}\right)}{{\sum }_{j=1}^{C}exp\left(\frac{{\langle {\mathbf{w}}_j,\mathbf{x}\rangle }^2}{\parallel {\mathbf{w}}_j{-\mathbf{x}\parallel }^2+ϵ}\right)}}$$

(20)

This softmax normalization serves a crucial role in the competitive dynamics of Yat-product neurons. The softmax function acts as a transformation that maps from the real-valued Yat-product responses $z_i\in \mathbb{R}$ to a delta distribution ${\delta }_i$ in probability space. This softmax distribution over Yat-product scores can be interpreted as the posterior responsibility of each prototype (neuron) for the input, drawing a direct connection to Gaussian Mixture Models (GMMs) and expectation-maximization frameworks.

The softmax can also be viewed as computing a categorical distribution proportional to exponentiated log-likelihoods, which in this case derive from a geometric Yat-product similarity rather than traditional probabilistic assumptions. This bridges the gap between probabilistic views (such as EM algorithms and classification) and our geometric formulation, providing a principled foundation for the competitive dynamics.

As training progresses and the differences between Yat-product responses become more pronounced, the softmax transformation approaches a delta distribution, where the winning neuron (with the highest Yat-product response) approaches probability 1 while all others approach 0. This winner-take-all mechanism enables competitive learning dynamics where each neuron competes to "take over" regions of the input space based on their vortex-like attraction fields.

The decision boundary between two neurons with prototypes ${\mathbf{w}}_i$ and ${\mathbf{w}}_j$ is defined by the condition where their responses are equal:

$$\mathrm{Yat}({\mathbf{w}}_i,\mathbf{x})=\mathrm{Yat}({\mathbf{w}}_j,\mathbf{x})$$

(21)

Expanding this condition:

$${\displaystyle \frac{{\langle {\mathbf{w}}_i,\mathbf{x}\rangle }^2}{\parallel {\mathbf{w}}_i{-\mathbf{x}\parallel }^2+ϵ}}={\displaystyle \frac{{\langle {\mathbf{w}}_j,\mathbf{x}\rangle }^2}{\parallel {\mathbf{w}}_j{-\mathbf{x}\parallel }^2+ϵ}}$$

(22)

Cross-multiplying and rearranging:

$${\langle {\mathbf{w}}_i,\mathbf{x}\rangle }^2(\parallel {\mathbf{w}}_j{-\mathbf{x}\parallel }^2+ϵ)={\langle {\mathbf{w}}_j,\mathbf{x}\rangle }^2(\parallel {\mathbf{w}}_i-\mathbf{x}{\parallel }^2+ϵ)$$

(23)

This equation defines a complex, non-linear decision boundary that depends on both the alignment (through the squared dot products) and the proximity (through the squared distances) between the input and each prototype. Unlike the linear hyperplane formed by conventional dot product neurons, the Yat-product creates what we term a vortex phenomenon or gravitational potential well around each prototype.

The space partitioning behavior of the Yat-product exhibits several key properties that create this vortex-like effect:

• Gravitational Attraction: The inverse-square relationship in the denominator creates a field where points are more strongly attracted to nearby prototypes, similar to gravitational fields in physics.
• Alignment Amplification: The squared dot product in the numerator creates a strong response for well-aligned inputs, while the vortex effect pulls data points toward the prototype center.
• Bounded Potential Wells: Each neuron creates a localized potential well with bounded depth, preventing the unbounded growth seen in linear neurons. This boundedness is theoretically guaranteed by the Minimal and Maximal Similarity Characterizations (Theorems A7 and A8), which establish that $0\le \mathrm{Yat}({\mathbf{w}}_i,\mathbf{x})\le \infty$ with well-defined extremal conditions.
• Curved Decision Boundaries: The resulting decision boundaries are non-linear curves that wrap around the data distribution, creating vortex-like territorial regions for each neuron.

This vortex phenomenon allows each Yat-product neuron to create a territorial "field" in the representation space, where data points are pulled toward the dominant prototype based on both similarity and proximity metrics. The field each neuron occupies can indeed be considered a vortex, where the strength of attraction follows an inverse-square law, creating more natural and geometrically faithful decision boundaries.

The combination of the Yat-product’s vortex-like attraction and the softmax’s competitive normalization creates a powerful space partitioning mechanism. Each neuron’s vortex field competes with others through the softmax transformation, and the neuron with the strongest local attraction (highest Yat-product response) wins that region. Over time, this leads to a natural tessellation of the input space, where each neuron’s territory is defined by the regions where its vortex field dominates. The softmax transformation ${\mathbb{R}}^C\to {\Delta }^{C-1}$ (where ${\Delta }^{C-1}$ is the $(C-1)$-dimensional probability simplex) ensures that these territorial boundaries are sharp and well-defined, transforming the continuous real-valued responses into discrete delta distributions that clearly assign each input to its dominant neuron.

Orthogonality and Competitive Dynamics: The competitive learning behavior observed in practice is theoretically grounded in our Orthogonality-Entropy Connection. When two prototypes ${\mathbf{w}}_i$ and ${\mathbf{w}}_j$ develop disjoint support regions, they become Euclidean orthogonal (${\mathbf{w}}_i\perp {\mathbf{w}}_j$), which corresponds to:

$$\mathrm{Yat}({\mathbf{w}}_i,{\mathbf{w}}_j)=0\mathrm{and}H({\mathbf{w}}_i,{\mathbf{w}}_j)=\infty$$

(24)

This geometric-probabilistic duality explains why neurons naturally develop specialized, non-overlapping representations during competitive learning. The infinite cross-entropy between orthogonal prototypes creates strong pressure for territorial separation, preventing the collapse to identical representations that can plague conventional competitive learning systems.

These prototypes are optimized to maximize parallelism and minimize distance to all points within their class distribution. When minimizing distance becomes challenging, the properties of the Yat-product enable the prototype to exist in a superposition state, prioritizing the maximization of parallelism over strict distance minimization.

4.2. Do you even MNIST bro?

Having established the theoretical foundation of the vortex phenomenon in Section 4.1, we now validate these insights on the canonical MNIST dataset. This experiment serves as a bridge between our geometric theory and practical applications, demonstrating how the vortex-like territorial dynamics translate into improved prototype learning on real data.

In our MNIST experiments, the network consists of $C=10$ neurons, each corresponding to one of the digit classes (0–9). Each neuron’s prototype is represented as a vector ${\mathbf{w}}_i\in {\mathbb{R}}^{784}$, where $i=1,\dots ,10$. The input images $\mathbf{x}\in {\mathbb{R}}^{784}$ are obtained by flattening the original $28\times 28$ pixel images, so each neuron’s prototype ${\mathbf{w}}_i$ has the same dimensionality as the input, i.e., ${\mathbf{w}}_i=(w_{i,1},w_{i,2},\dots ,w_{i,784})$. This structure allows each neuron to learn class-specific features in the full image space.

The MNIST dataset provides an ideal testbed for examining the vortex phenomenon because its 10-class structure allows clear visualization of how different neurons compete for territorial control in the feature space. We specifically investigate whether the bounded attraction fields and territorial partitioning predicted by our theory manifest as improved prototype quality and learning dynamics in practice.

Experimental Design: We train both conventional linear classifiers and our Yat-product networks on MNIST, analyzing three key aspects that directly relate to the vortex phenomenon:

• Prototype Evolution Dynamics: How do prototypes evolve during training under different competitive mechanisms?
• Territorial Boundary Formation: Do we observe the predicted non-linear decision boundaries and vortex-like attraction fields?
• Representational Quality: How does the theoretical prediction of bounded, concentrated prototypes translate to interpretability?

The prototype evolution during training reveals the fundamental differences between conventional unbounded growth and our bounded vortex dynamics. Figure 9 shows the final learned prototypes, providing striking empirical confirmation of our vortex theory. The conventional linear model produces prototypes that exhibit the unbounded growth predicted by our analysis—these prototypes become increasingly diffuse and less interpretable as they grow to maximize margin separation. The resulting digit representations are blurry and lack the fine-grained features necessary for robust classification.

In stark contrast, the Yat-product method produces prototypes that perfectly exemplify the bounded vortex fields described in our theory. Each digit prototype exhibits:

• Localized Concentration: Sharp, well-defined features that correspond to the bounded potential wells predicted by our Minimal and Maximal Similarity Characterizations (Theorems A7 and A8)
• Class-Specific Territorial Structure: Each prototype captures unique digit characteristics, reflecting the competitive territorial dynamics where each neuron’s vortex field dominates specific regions of the input space
• Geometric Fidelity: The prototypes maintain geometric coherence with actual digit structure, confirming that the signal-to-noise ratio optimization preserves meaningful visual patterns

Figure 8. Prototypes learned by the conventional linear model (top) and our proposed Yat-product method (bottom) on the MNIST dataset. The prototypes from our method are more distinct and representative of the digit classes, capturing finer details and class-specific characteristics.

Figure 8. Prototypes learned by the conventional linear model (top) and our proposed Yat-product method (bottom) on the MNIST dataset. The prototypes from our method are more distinct and representative of the digit classes, capturing finer details and class-specific characteristics.

Superposition and Prototype Inversion: A unique property of the Yat-product neuron is its ability to exist in a superposition state, which can be empirically demonstrated by inverting the learned prototype. Specifically, if $\mathbf{w}$ is a learned prototype, we consider the effect of replacing $\mathbf{w}$ with $-\mathbf{w}$ (i.e., multiplying by $-1$) at test time, without any retraining. For a conventional dot product neuron, this operation flips the sign of the logit:

$$z={\mathbf{w}}^T\mathbf{x}⟶z^{\prime }={(-\mathbf{w})}^T\mathbf{x}=-z$$

(25)

This sign flip causes the softmax output to assign high probability to the incorrect class, resulting in a dramatic drop in accuracy (from 93% to nearly 0% in our MNIST experiments).

In contrast, for the Yat-product neuron, the response is:

$$z=\mathrm{Yat}(\mathbf{w},\mathbf{x})={\displaystyle \frac{{\left({\mathbf{w}}^T\mathbf{x}\right)}^2}{{\parallel \mathbf{w}-\mathbf{x}\parallel }^2+ϵ}}$$

(26)

Multiplying $\mathbf{w}$ by $-1$ leaves the numerator unchanged, since ${(-\mathbf{w})}^T\mathbf{x}=-{\mathbf{w}}^T\mathbf{x}$ and ${(-{\mathbf{w}}^T\mathbf{x})}^2={\left({\mathbf{w}}^T\mathbf{x}\right)}^2$. The denominator is also unchanged, as ${\parallel -\mathbf{w}-\mathbf{x}\parallel }^2={\parallel \mathbf{w}+\mathbf{x}\parallel }^2$, which is symmetric with respect to the data distribution. As a result, the Yat-product neuron’s accuracy remains nearly unchanged (dropping only slightly from 92% to 89%), demonstrating its robustness to prototype inversion and its ability to represent solutions in a superposition state.

This property allows the Yat-product neuron to yield two valid solutions to the same dataset without retraining, a phenomenon not observed in conventional dot product neurons. The table below summarizes the empirical results:

Table 1. Test accuracy on MNIST before and after prototype inversion ($\mathbf{w}\to -\mathbf{w}$) for dot product and Yat-product (yat) neurons.

Neuron Type	Original Prototype	Inverted Prototype ($\mathit{w}\to -\mathit{w}$)
Dot Product	91.88%	≈0.01%
Yat-Product (Yat)	92.18%	87.87%

Table 1. Test accuracy on MNIST before and after prototype inversion ($\mathbf{w}\to -\mathbf{w}$) for dot product and Yat-product (yat) neurons.

Neuron Type	Original Prototype	Inverted Prototype ($\mathit{w}\to -\mathit{w}$)
Dot Product	91.88%	≈0.01%
Yat-Product (Yat)	92.18%	87.87%

4.3. Aether-GPT2: The Last Unexplainable Language Model

To demonstrate the versatility of our approach beyond vision tasks, we implement Aether-GPT2, incorporating the Yat-product architecture into the GPT2 framework for language modeling. We compare the perplexity scores between our Aether-GPT2 and the standard GPT2 architecture across multiple text corpora.

The results in Table 2 demonstrate that Aether-GPT2 achieves a validation loss competitive with the standard GPT-2 baseline. While the loss is marginally higher, it is critical to note that Aether-GPT2 attains this performance despite its simplified design, which entirely omits dedicated activation functions and normalization layers. This outcome highlights a promising trade-off between raw performance and architectural simplicity, efficiency, and the inherent interpretability afforded by the Yat-product. These findings establish Aether-GPT2 as a successful proof-of-concept, suggesting that the Yat-product can serve as a viable alternative to conventional neural network components.

Figure 9. Loss Curve over the 600m tokens from fineweb trained on Kaggle TPU v3, Linear model is using standard GPT2 achitecture.

Figure 9. Loss Curve over the 600m tokens from fineweb trained on Kaggle TPU v3, Linear model is using standard GPT2 achitecture.

Table 3. Aether-GPT2 Experiment Card

Parameter	Value
Optimizer	Novograd
Learning Rate	0.003
Batch Size	32
Embedding Dimension	768
MLP Dimension	768 (No x4)
Vocabulary Size	50,257
Number of Heads	12
Number of Blocks	12

Table 3. Aether-GPT2 Experiment Card

Parameter	Value
Optimizer	Novograd
Learning Rate	0.003
Batch Size	32
Embedding Dimension	768
MLP Dimension	768 (No x4)
Vocabulary Size	50,257
Number of Heads	12
Number of Blocks	12

The results demonstrate that Aether-GPT2 consistently achieves close loss, indicating its ability to learn non-linearity without the need for activation functions.

The performance can be attributed to the Yat-product’s ability to capture more nuanced relationships between tokens, allowing the model to better understand contextual dependencies and semantic similarities in natural language.

5. Related Work

5.1. Inverse-Square Laws

The inverse-square law, where a quantity or intensity is inversely proportional to the square of the distance from its source, is a fundamental principle observed across numerous scientific disciplines [15]. This relationship signifies that as the distance from a source doubles, the intensity reduces to one-quarter of its original value.

In classical physics, this law is foundational. Newton’s Law of Universal Gravitation describes the force between two masses [16], and Coulomb’s Law defines the electrostatic force between charges [17]. The intensity of electromagnetic radiation, such as light, and the intensity of sound from a point source also diminish according to this principle. Gauss’s Law offers a unifying mathematical framework for these phenomena in fields like electromagnetism and gravitation, connecting them to the property that the divergence of such fields is zero outside the source [18]. Similarly, thermal radiation intensity from a point source adheres to this law [39].

The inverse-square law’s influence extends significantly into engineering and applied sciences. In health physics, it is critical for radiation protection, governing the attenuation of ionizing radiation [40]. Photometry applies it to illumination engineering for lighting design [41]. In telecommunications, it underpins the free-space path loss of radio signals, as described by the Friis transmission equation [42], while radar systems experience an inverse fourth-power relationship due to the signal’s two-way travel [43]. Seismology observes that seismic wave energy attenuates following this law [44], and in fluid dynamics, the velocity field from a point source in incompressible, irrotational flow also demonstrates an inverse-square dependence [45].

Beyond the physical sciences, analogous concepts are found in information theory, data science, and social sciences. For instance, the Tanimoto coefficient, used in molecular similarity analysis [23]), and the Jaccard index, a metric for set similarity [24], exhibit mathematical properties akin to inverse-square decay when viewed in feature space geometry. In economics, the gravity model of trade frequently employs an inverse-square (or similar power law) relationship to predict trade flows between entities, based on their economic "mass" (e.g., GDP) and the distance separating them [46], illustrating how these physical principles can offer insights into complex socio-economic phenomena.

5.2. Learning with Kernels

Learning with kernels has significantly influenced machine learning by enabling algorithms to handle complex, non-linear patterns efficiently. The introduction of Support Vector Machines (SVMs) [37] laid the foundation for kernel-based learning, leveraging the kernel trick to implicitly map data into high-dimensional spaces. Schölkopf formalized kernel methods, expanding their applicability to various tasks [36].

Key advancements include Kernel PCA [47] for non-linear dimensionality reduction and Gaussian Processes [48] for probabilistic modeling. Applications like Spectral Clustering[49] and One-Class SVM [50] highlight the versatility of kernel methods.

To address computational challenges, techniques like the Nyström method [51] and Random Fourier Features [52] have improved scalability. Recent work, such as the Neural Tangent Kernel (NTK) [38], bridges kernel methods and deep learning, offering insights into the dynamics of neural networks.

Furthermore, many prominent kernel functions, such as the Gaussian Radial Basis Function (RBF) kernel [53], explicitly define similarity based on the Euclidean distance between data points, effectively giving higher weight to nearby points. This inherent sensitivity to proximity allows models like Support Vector Machines with RBF kernels or Gaussian Processes to capture local structures in the data.

While these methods leverage distance to inform the kernel matrix or model covariance, our research explores a more direct architectural integration of proximity. We propose a novel neural operator where an inverse-square law with respect to feature vector distance is fundamentally incorporated into the neuron’s computation, in conjunction with measures of feature alignment. This approach differs from traditional kernel methods where the kernel function primarily serves to define a similarity measure for algorithms that operate on pairs of samples, rather than defining the intrinsic operational characteristics of individual neural units themselves.

5.3. Deep Learning

The landscape of deep learning is characterized by a continuous drive towards architectures and neural operators that offer enhanced expressivity, computational efficiency, and improved understanding of underlying data structures. Convolutional Neural Networks (CNNs) remain a foundational paradigm, particularly in vision, lauded for their proficiency in extracting hierarchical features [54]. However, the pursuit of alternative and complementary approaches remains vibrant.

A significant trajectory involves architectures leveraging dot-product mechanisms, most prominently exemplified by the Transformer architecture and its self-attention mechanism [55]. This has spurred developments like Vision Transformers (ViTs) [56], and models such as MLP-Mixer [57] and gMLP [58] which, while simplifying or eschewing self-attention, still rely on operations like matrix multiplication for feature mixing, demonstrating competitive performance, particularly in computer vision.

The quest for capturing intricate data relationships has also led to renewed interest in Polynomial Neural Networks. These networks incorporate polynomial expansions of neuron inputs, enabling the modeling of higher-order correlations [59,60], offering a different inductive bias compared to standard linear transformations followed by activation functions. Concurrently, Fourier Networks, such as FNet [61], explore the frequency domain, replacing spatial convolutions or attention with Fourier transforms for token or feature mixing, presenting an alternative for global information aggregation with potential efficiency gains.

Despite these advancements, a persistent challenge in deep learning is interpretability. The complex interplay of numerous parameters and non-linear activation functions (e.g., ReLU [7], sigmoid, tanh) often renders the internal decision-making processes of deep models opaque [27]. These diverse approaches highlight a shared pursuit for more expressive primitives. Our work contributes to this by proposing an operator that achieves non-linearity not through polynomial expansion or frequency-domain transformation, but through a unified measure of geometric alignment and proximity.

6. Conclusion

Perhaps artificial intelligence’s greatest limitation has been our stubborn fixation on the human brain as the pinnacle of intelligence. The universe itself, governed by elegant and powerful laws, demonstrates intelligence far beyond human cognition. These fundamental laws, which shape galaxies and guide quantum particles, represent a deeper form of intelligence that we have largely ignored in our pursuit of AI.

In this paper, we challenge the conventional AI paradigm. We broke free from biological metaphors by drawing direct inspiration from inverse-square law interactions in physics. The Yat-product, with its inherent non-linearity and geometric sensitivity, allows for a more nuanced understanding of vector interactions, capturing both alignment and proximity in a single operation. This approach not only simplifies the architecture of neural networks but also enhances their interpretability and robustness.

Acknowledgments

This research was supported by the MLNomads community and the broader open-source AI community. We extend special thanks to Dr. D. Sculley for his insightful feedback on kernel learning. We are also grateful to Kaggle and Colab for providing the computational resources instrumental to this research. Additionally, this work received support from the Google Developer Expert program, the Google AI/ML Developer Programs team, and Google for Startups in the form of Google Cloud Credits. We used BashNota and Weights and Biases for managing hypotheses and validating our research.

Appendix G.1. Preliminary

• Cauchy-Schwarz Inequality [62]: Used to bound the dot product and characterize equality conditions for identical vectors/distributions.
• Properties of KL Divergence and Cross-Entropy [63]: Used to show divergence for disjoint supports in probability distributions.
• Mercer’s Theorem [33,64]: Establishes that a symmetric, positive semi-definite kernel defines a valid reproducing kernel Hilbert space (RKHS).
• Schur Product Theorem [62]: States that the Hadamard (element-wise) product of two positive semi-definite matrices is also positive semi-definite.
• Bochner’s Theorem [65]: Characterizes translation-invariant kernels as positive definite if and only if their Fourier transform is non-negative.
• Universal Approximation Theorem [4,66]: Guarantees that neural networks with suitable activation functions or kernels can approximate any continuous function on a compact set.
• Universality of Polynomial and Translation-Invariant Kernels [35]: Used to argue that both the squared polynomial kernel and the translation-invariant kernel are universal.
• Laplace Transform/Integral Representation [67]: Used to express the inverse quadratic kernel as an integral over Gaussians, supporting the Bochner argument.

Appendix G.2. Proof of Mercer’s Condition for the Yat-product

Theorem A2. 

The Yat-product, defined as $K(\mathbf{w},\mathbf{x})={\displaystyle \frac{{\langle \mathbf{w},\mathbf{x}\rangle }^2}{{\parallel \mathbf{w}-\mathbf{x}\parallel }^2+ϵ}}$, is a Mercer kernel.

Proof. 

To prove that the Yat-product is a Mercer kernel, we must show that it is symmetric and positive semi-definite [33,64].

1. Symmetry

The kernel is defined as:

$$K(\mathbf{w},\mathbf{x})={\displaystyle \frac{{\langle \mathbf{w},\mathbf{x}\rangle }^2}{{\parallel \mathbf{w}-\mathbf{x}\parallel }^2+ϵ}}$$

(A1)

To check for symmetry, we evaluate $K(\mathbf{x},\mathbf{w})$:

$$K(\mathbf{x},\mathbf{w})={\displaystyle \frac{{\langle \mathbf{x},\mathbf{w}\rangle }^2}{{\parallel \mathbf{x}-\mathbf{w}\parallel }^2+ϵ}}$$

(A2)

Given that the dot product is commutative, $\langle \mathbf{w},\mathbf{x}\rangle =\langle \mathbf{x},\mathbf{w}\rangle$, and thus ${\langle \mathbf{w},\mathbf{x}\rangle }^2={\langle \mathbf{x},\mathbf{w}\rangle }^2$. Also, the squared Euclidean distance is symmetric: ${\parallel \mathbf{w}-\mathbf{x}\parallel }^2=(\mathbf{w}-\mathbf{x})·(\mathbf{w}-\mathbf{x})=\mathbf{w}·\mathbf{w}-2\mathbf{w}·\mathbf{x}+\mathbf{x}·\mathbf{x}={\parallel \mathbf{x}-\mathbf{w}\parallel }^2$. Therefore, $K(\mathbf{w},\mathbf{x})=K(\mathbf{x},\mathbf{w})$, and the kernel is symmetric.

2. Positive Semi-Definiteness

A kernel $K(\mathbf{w},\mathbf{x})$ is positive semi-definite (PSD) if for any finite set of points ${\left\{{\mathbf{x}}_i\right\}}_{i=1}^N\subset {\mathbb{R}}^d$ and any coefficients ${\left\{c_i\right\}}_{i=1}^N\subset \mathbb{R}$, the following condition holds:

$$\sum _{i=1}^N\sum _{j=1}^N{c}_i{c}_{j}K({\mathbf{x}}_i,{\mathbf{x}}_j)\ge 0$$

This is equivalent to stating that the Gram matrix G, where $G_{ij}=K({\mathbf{x}}_i,{\mathbf{x}}_j)$, is positive semi-definite.

The proof of positive semi-definiteness is established by decomposing the kernel and leveraging the Schur product theorem [62] in conjunction with Bochner’s theorem [65] for translation-invariant kernels. The proof proceeds by showing that the Yat-product is a product of two established Mercer kernels. Let the kernel be decomposed as:

$$K(\mathbf{w},\mathbf{x})=K_1(\mathbf{w},\mathbf{x})·K_2(\mathbf{w},\mathbf{x})$$

where:

• $K_1(\mathbf{w},\mathbf{x})={\langle \mathbf{w},\mathbf{x}\rangle }^2$
• $K_2(\mathbf{w},\mathbf{x})={\displaystyle \frac{1}{{\parallel \mathbf{w}-\mathbf{x}\parallel }^2+ϵ}}$

The set of Mercer kernels is closed under pointwise product. If $K_1$ and $K_2$ are Mercer kernels, then for any set of points, their Gram matrices $G_1$ and $G_2$ are PSD. The Gram matrix for K is the Hadamard (element-wise) product of $G_1$ and $G_2$. By the Schur product theorem [62], the Hadamard product of two PSD matrices is also PSD. Thus, if we can prove that $K_1$ and $K_2$ are Mercer kernels, their product K must also be a Mercer kernel.

a) $K_1(\mathbf{w},\mathbf{x})$ is a Mercer Kernel

The linear kernel $k_{\mathrm{lin}}(\mathbf{w},\mathbf{x})=\langle \mathbf{w},\mathbf{x}\rangle$ is a known Mercer kernel. As established, the set of Mercer kernels is closed under multiplication, so $K_1(\mathbf{w},\mathbf{x})=k_{\mathrm{lin}}(\mathbf{w},\mathbf{x})·k_{\mathrm{lin}}(\mathbf{w},\mathbf{x})={\langle \mathbf{w},\mathbf{x}\rangle }^2$ is also a Mercer kernel.

b) $K_2(\mathbf{w},\mathbf{x})$ is a Mercer Kernel

The kernel $K_2$ is a translation-invariant kernel, as it depends only on the difference $\mathbf{z}=\mathbf{w}-\mathbf{x}$. Let $k_2\left(\mathbf{z}\right)={(\parallel \mathbf{z}\parallel }^2{+ϵ)}^{-1}$. By Bochner’s theorem [65], a continuous translation-invariant kernel is positive definite if and only if its Fourier transform is non-negative.

The function $k_2\left(\mathbf{z}\right)$ can be represented as an integral of a positive function (a Gaussian) using the identity ${\displaystyle \frac{1}{A}}={\int }_0^{\infty }{e}^{-At}dt$ [67]:

$$\begin{array}{cc}\hfill k_2\left(\mathbf{z}\right)& ={\displaystyle \frac{1}{{\parallel \mathbf{z}\parallel }^2+ϵ}}\hfill \\ \hfill & ={\int }_0^{\infty }{e}^{-(\parallel \mathbf{z}{\parallel }^2+ϵ)t}dt\hfill \\ \hfill & ={\int }_0^{\infty }{e}^{-ϵt}{e}^{-{t\parallel \mathbf{z}\parallel }^2}dt\hfill \end{array}$$

The term $e^{-{t\parallel \mathbf{z}\parallel }^2}$ is proportional to the un-normalized density of a zero-mean Gaussian with variance ${\sigma }^2=1/\left(2t\right)$. The Fourier transform of a Gaussian is a Gaussian, which is always non-negative. Since $e^{-ϵt}$ is also non-negative for $t\ge 0$, the Fourier transform of $k_2\left(\mathbf{z}\right)$ is an integral of non-negative functions, and is therefore itself non-negative. Thus, $K_2$ is a Mercer kernel.

Since both $K_1$ and $K_2$ are Mercer kernels, their product, $K(\mathbf{w},\mathbf{x})$, is also a Mercer kernel.

Thus, the Yat-product satisfies the conditions of symmetry and positive semi-definiteness, and is therefore a Mercer kernel. □

Appendix G.3 PProof of Self-Regulation for the Yat-product

Theorem A3 (The Yat-Product is Naturally Self-Regulating) The output of a Yat-product neuron is bounded and converges to a finite value as the magnitude of the input vector approaches infinity.

Proof. Let the Yat-product be defined as:

$$\mathrm{Yat}(\mathbf{w},\mathbf{x})={\displaystyle \frac{{\langle \mathbf{w},\mathbf{x}\rangle }^2}{{\parallel \mathbf{w}-\mathbf{x}\parallel }^2+ϵ}}$$

(A3)

where $\mathbf{w}$ is a fixed weight vector and $\mathbf{x}$ is the input vector.

We want to analyze the behavior of $\mathrm{Yat}(\mathbf{w},\mathbf{x})$ as the magnitude of the input, $\parallel \mathbf{x}\parallel$, approaches infinity. We can represent any input vector $\mathbf{x}$ as $\mathbf{x}=k·\mathbf{u}$, where $k=\parallel \mathbf{x}\parallel$ is its magnitude and $\mathbf{u}$ is a unit vector in the direction of $\mathbf{x}$. The limit can be expressed as $k\to \infty$.

Substituting $\mathbf{x}=k\mathbf{u}$ into the equation and using the properties of the dot product and norm, we get:

$$\begin{array}{cc}\hfill \mathrm{Yat}(\mathbf{w},k\mathbf{u})& ={\displaystyle \frac{{\langle \mathbf{w},k\mathbf{u}\rangle }^2}{{\parallel \mathbf{w}-k\mathbf{u}\parallel }^2+ϵ}}\hfill \\ \hfill & ={\displaystyle \frac{k^2{\langle \mathbf{w},\mathbf{u}\rangle }^2}{{\parallel \mathbf{w}\parallel }^2-2\langle \mathbf{w},k\mathbf{u}\rangle +{\parallel k\mathbf{u}\parallel }^2+ϵ}}\hfill \\ \hfill & ={\displaystyle \frac{k^2{\langle \mathbf{w},\mathbf{u}\rangle }^2}{{\parallel \mathbf{w}\parallel }^2-2k\langle \mathbf{w},\mathbf{u}\rangle +k^2{\parallel \mathbf{u}\parallel }^2+ϵ}}\hfill \end{array}$$

Since $\mathbf{u}$ is a unit vector, ${\parallel \mathbf{u}\parallel }^2=1$:

$$\mathrm{Yat}(\mathbf{w},k\mathbf{u})={\displaystyle \frac{k^2{\langle \mathbf{w},\mathbf{u}\rangle }^2}{{\parallel \mathbf{w}\parallel }^2-2k\langle \mathbf{w},\mathbf{u}\rangle +k^2+ϵ}}$$

(A4)

To find the limit as $k\to \infty$, we divide the numerator and the denominator by the highest power of k, which is $k^2$:

$$\underset{k\to \infty }{lim}\mathrm{Yat}(\mathbf{w},k\mathbf{u})=\underset{k\to \infty }{lim}{\displaystyle \frac{{\langle \mathbf{w},\mathbf{u}\rangle }^2}{\frac{{\parallel \mathbf{w}\parallel }^2}{k^2}-\frac{2\langle \mathbf{w},\mathbf{u}\rangle }{k}+1+\frac{ϵ}{k^2}}}$$

(A5)

As $k\to \infty$, the terms with k in the denominator approach zero:

$$\underset{k\to \infty }{lim}\mathrm{Yat}(\mathbf{w},k\mathbf{u})={\displaystyle \frac{{\langle \mathbf{w},\mathbf{u}\rangle }^2}{0-0+1+0}}={\langle \mathbf{w},\mathbf{u}\rangle }^2$$

(A6)

By the definition of the dot product, $\langle \mathbf{w},\mathbf{u}\rangle =\parallel \mathbf{w}\parallel \parallel \mathbf{u}\parallel cos\theta =\parallel \mathbf{w}\parallel cos\theta$, where $\theta$ is the angle between $\mathbf{w}$ and $\mathbf{u}$ (the direction of $\mathbf{x}$).

Therefore, the limit is:

$$\underset{\parallel \mathbf{x}\parallel \to \infty }{lim}\mathrm{Yat}(\mathbf{w},\mathbf{x})={(\parallel \mathbf{w}\parallel cos\theta )}^2={\parallel \mathbf{w}\parallel }^2{cos}^2\theta$$

(A7)

Since ${cos}^2\theta$ is always between 0 and 1, the output of the Yat-product is bounded by $0\le \mathrm{Yat}(\mathbf{w},\mathbf{x})\le {\parallel \mathbf{w}\parallel }^2$. The limit is a finite value that depends only on the squared magnitude of the weight vector and the squared cosine of the angle between the weight and input vectors. This proves that the kernel is naturally self-regulating and does not diverge, even for inputs of arbitrarily large magnitude. □

Appendix G.4 Addressing Internal Covariate Shift

Theorem A4 (Asymptotic Independence of Score Statistics) Let $a=\mathrm{Yat}(\mathbf{w},\mathbf{x})$ be the score of a neuron for an input $\mathbf{x}$. Consider a mini-batch of inputs $\mathcal{B}=\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_N\}$, where each input is represented as ${\mathbf{x}}_i=k_i{\mathbf{u}}_i$ with magnitude $k_i=\parallel {\mathbf{x}}_i\parallel$ and direction ${\mathbf{u}}_i$. Let ${\mu }_{\mathcal{B}}\left(a\right)$ and ${\sigma }_{\mathcal{B}}^2\left(a\right)$ denote the empirical mean and variance of the scores over the mini-batch. In the limit as $k_i\to \infty$ for all $i\in \{1,\dots ,N\}$, the mean and variance of the scores converge to values that are independent of the magnitudes $k_i$:

$$\begin{array}{cc}\hfill \underset{k_1,\dots ,k_N\to \infty }{lim}{\mu }_{\mathcal{B}}\left(a\right)& ={\parallel \mathbf{w}\parallel }^2{\mathbb{E}}_{\mathbf{u}\in \mathcal{U}}\left[{cos}^2\theta (\mathbf{w},\mathbf{u})\right]\hfill \\ \hfill \underset{k_1,\dots ,k_N\to \infty }{lim}{\sigma }_{\mathcal{B}}^2\left(a\right)& ={\parallel \mathbf{w}\parallel }^4{\mathrm{Var}}_{\mathbf{u}\in \mathcal{U}}\left[{cos}^2\theta (\mathbf{w},\mathbf{u})\right]\hfill \end{array}$$

where $\mathcal{U}=\{{\mathbf{u}}_1,\dots ,{\mathbf{u}}_N\}$ is the set of direction vectors and the expectation and variance are taken over this set. This demonstrates that the scores statistics are asymptotically decoupled from input magnitudes, thus mitigating internal covariate shift.

Proof. Let a neuron in a neural network layer be defined by the Yat-product kernel, $a=\mathrm{Yat}(\mathbf{w},\mathbf{x})$, where $\mathbf{w}$ is the weight vector and $\mathbf{x}$ is the input vector from the previous layer. Internal Covariate Shift (ICS) refers to the change in the distribution of the input $\mathbf{x}$ during training, which in turn causes undesirable shifts in the distribution of the score a. We will demonstrate that the statistical moments of the score a are asymptotically independent of the input magnitude $\parallel \mathbf{x}\parallel$, thus mitigating ICS.

From the proof in Section G.3, we have established the asymptotic behavior of the Yat-product for an input $\mathbf{x}=k\mathbf{u}$ where $k=\parallel \mathbf{x}\parallel$:

$$\underset{k\to \infty }{lim}\mathrm{Yat}(\mathbf{w},k\mathbf{u})={\parallel \mathbf{w}\parallel }^2{cos}^2\theta$$

(A8)

where $\theta$ is the angle between the weight vector $\mathbf{w}$ and the input direction vector $\mathbf{u}$. For inputs with large magnitudes, which are a primary concern for training stability, the score can be approximated as:

$$\mathrm{Yat}(\mathbf{w},\mathbf{x})\approx {\parallel \mathbf{w}\parallel }^2{cos}^2\theta$$

(A9)

Consider a mini-batch of N inputs, $\mathcal{B}=\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_N\}$. The corresponding score are $a_i=\mathrm{Yat}(\mathbf{w},{\mathbf{x}}_i)$. The empirical mean of the scores over this mini-batch is:

$${\mu }_{\mathcal{B}}\left(a\right)={\displaystyle \frac{1}{N}}\sum _{i=1}^N{a}_i$$

(A10)

Assuming the inputs in the mini-batch have sufficiently large magnitudes, we can substitute the asymptotic approximation:

$${\mu }_{\mathcal{B}}\left(a\right)\approx {\displaystyle \frac{1}{N}}\sum _{i=1}^N{\parallel \mathbf{w}\parallel }^2{cos}^2{\theta }_i={\parallel \mathbf{w}\parallel }^2·{\displaystyle \frac{1}{N}}\sum _{i=1}^N{cos}^2{\theta }_i$$

(A11)

This can be expressed in terms of the empirical expectation over the mini-batch:

$${\mu }_{\mathcal{B}}\left(a\right)\approx {\parallel \mathbf{w}\parallel }^2{\mathbb{E}}_{\mathbf{x}\in \mathcal{B}}\left[{cos}^2\theta (\mathbf{w},\mathbf{x})\right]$$

(A12)

Similarly, the empirical variance of the scores is:

$${\sigma }_{\mathcal{B}}^2\left(a\right)={\displaystyle \frac{1}{N}}\sum _{i=1}^N{(a_i-{\mu }_{\mathcal{B}}\left(a\right))}^2$$

(A13)

Using the same approximation, the variance becomes:

$$\begin{array}{cc}\hfill {\sigma }_{\mathcal{B}}^2\left(a\right)& \approx {\mathbb{E}}_{\mathbf{x}\in \mathcal{B}}{\left[\right(\parallel \mathbf{w}\parallel }^2{cos}^2\theta -{\parallel \mathbf{w}\parallel }^2{\mathbb{E}}_{\mathbf{x}\in \mathcal{B}}\left[{cos}^2\theta \right]{)}^2]\hfill \end{array}$$

(A14)

$$\begin{array}{cc}\hfill & ={\parallel \mathbf{w}\parallel }^4{\mathbb{E}}_{\mathbf{x}\in \mathcal{B}}\left[{({cos}^2\theta -{\mathbb{E}}_{\mathbf{x}\in \mathcal{B}}\left[{cos}^2\theta \right])}^2\right]\hfill \end{array}$$

(A15)

$$\begin{array}{cc}\hfill & ={\parallel \mathbf{w}\parallel }^4\left({\mathbb{E}}_{\mathbf{x}\in \mathcal{B}}\left[{cos}^4\theta \right]-{\left({\mathbb{E}}_{\mathbf{x}\in \mathcal{B}}\left[{cos}^2\theta \right]\right)}^2\right)\hfill \end{array}$$

(A16)

Crucially, both the empirical mean and variance of the scores are, in the large-magnitude limit, functions of the weight vector’s magnitude $\parallel \mathbf{w}\parallel$ and the statistics of the angle $\theta$ between the weights and the inputs. They are independent of the input magnitudes $\parallel {\mathbf{x}}_i\parallel$.

During training, while the distribution of $\mathbf{x}$ (and thus the distribution of angles ${\theta }_i$) and the weight vector $\mathbf{w}$ evolve, the primary source of instability associated with ICS, namely, drastic fluctuations in the magnitudes of layer inputs, is filtered out. The evolution of the score distribution is governed by the more gradual changes in the learned weight vector and the angular structure of the data, rather than the raw input scales. This decoupling of score statistics from input magnitudes provides inherent stabilization, thus mitigating internal covariate shift. □

Appendix G.5 Proof of Stable Learning for the Yat-product

Theorem A5 (The Yat-Product Ensures Stable Learning) The gradient of the Yat-product with respect to its input, ${\nabla }_{\mathbf{x}}\mathrm{Yat}(\mathbf{w},\mathbf{x})$, approaches zero as the input vector $\mathbf{x}$ moves infinitely far from the weight vector $\mathbf{w}$.

Proof. We aim to prove that the learning signal, represented by the gradient of the Yat-product with respect to the input $\mathbf{x}$, diminishes for inputs that are distant from the learned weight vector $\mathbf{w}$. This ensures that outliers do not cause large, destabilizing updates.

The Yat-product is defined as:

$$\mathrm{Yat}(\mathbf{w},\mathbf{x})={\displaystyle \frac{{\langle \mathbf{w},\mathbf{x}\rangle }^2}{{\parallel \mathbf{w}-\mathbf{x}\parallel }^2+ϵ}}={\displaystyle \frac{N\left(\mathbf{x}\right)}{D\left(\mathbf{x}\right)}}$$

where $N\left(\mathbf{x}\right)={\langle \mathbf{w},\mathbf{x}\rangle }^2$ and $D\left(\mathbf{x}\right)={\parallel \mathbf{w}-\mathbf{x}\parallel }^2+ϵ$.

Using the quotient rule for vector calculus, the gradient ${\nabla }_{\mathbf{x}}\mathrm{Yat}$ is:

$${\nabla }_{\mathbf{x}}\mathrm{Yat}={\displaystyle \frac{\left({\nabla }_{\mathbf{x}}N\right)D-N\left({\nabla }_{\mathbf{x}}D\right)}{D^2}}$$

First, we compute the gradients of the numerator $N\left(\mathbf{x}\right)$ and the denominator $D\left(\mathbf{x}\right)$:

1. Gradient of the Numerator

$N\left(\mathbf{x}\right)={\left({\mathbf{w}}^T\mathbf{x}\right)}^2$

$${\nabla }_{\mathbf{x}}N\left(\mathbf{x}\right)=2\left({\mathbf{w}}^T\mathbf{x}\right)·{\nabla }_{\mathbf{x}}\left({\mathbf{w}}^T\mathbf{x}\right)=2\langle \mathbf{w},\mathbf{x}\rangle \mathbf{w}$$

2. Gradient of the Denominator

$D\left(\mathbf{x}\right)={\parallel \mathbf{w}-\mathbf{x}\parallel }^2+ϵ={(\mathbf{w}-\mathbf{x})}^T(\mathbf{w}-\mathbf{x})+ϵ$

$${\nabla }_{\mathbf{x}}D\left(\mathbf{x}\right)=2(\mathbf{w}-\mathbf{x})·(-1)=-2(\mathbf{w}-\mathbf{x})=2(\mathbf{x}-\mathbf{w})$$

Substituting these into the quotient rule expression:

$${\nabla }_{\mathbf{x}}\mathrm{Yat}={\displaystyle \frac{{\left(2\langle \mathbf{w},\mathbf{x}\rangle \mathbf{w}\right)(\parallel \mathbf{w}-\mathbf{x}\parallel }^2+ϵ)-\left({\langle \mathbf{w},\mathbf{x}\rangle }^2\right)\left(2(\mathbf{x}-\mathbf{w})\right)}{{(\parallel \mathbf{w}-\mathbf{x}\parallel }^2{+ϵ)}^2}}$$

To analyze the behavior for distant inputs, we examine the limit as $\parallel \mathbf{x}\parallel \to \infty$. Let $\mathbf{x}=k\mathbf{u}$, where $k=\parallel \mathbf{x}\parallel$ and $\mathbf{u}$ is a unit vector.

As $k\to \infty$:

• $\langle \mathbf{w},\mathbf{x}\rangle =k\langle \mathbf{w},\mathbf{u}\rangle \sim \mathcal{O}\left(k\right)$
• ${\parallel \mathbf{w}-\mathbf{x}\parallel }^2={\parallel \mathbf{w}\parallel }^2-2k\langle \mathbf{w},\mathbf{u}\rangle +k^2\sim \mathcal{O}\left(k^2\right)$

Let’s analyze the order of magnitude for the terms in the gradient’s numerator:

• First term: ${\left(2\langle \mathbf{w},\mathbf{x}\rangle \mathbf{w}\right)(\parallel \mathbf{w}-\mathbf{x}\parallel }^2+ϵ)\sim \mathcal{O}\left(k\right)·\mathcal{O}\left(k^2\right)=\mathcal{O}\left(k^3\right)$
• Second term: $\left({\langle \mathbf{w},\mathbf{x}\rangle }^2\right)\left(2(\mathbf{x}-\mathbf{w})\right)\sim \mathcal{O}\left(k^2\right)·\mathcal{O}\left(k\right)=\mathcal{O}\left(k^3\right)$

The numerator as a whole is of order $\mathcal{O}\left(k^3\right)$.

The denominator is ${(\parallel \mathbf{w}-\mathbf{x}\parallel }^2{+ϵ)}^2\sim {\left(\mathcal{O}\left(k^2\right)\right)}^2=\mathcal{O}\left(k^4\right)$.

Therefore, the magnitude of the gradient behaves as:

$$\parallel {\nabla }_{\mathbf{x}}\mathrm{Yat}\parallel \sim {\displaystyle \frac{\mathcal{O}\left(k^3\right)}{\mathcal{O}\left(k^4\right)}}=\mathcal{O}\left({\displaystyle \frac{1}{k}}\right)$$

As $k=\parallel \mathbf{x}\parallel \to \infty$, the magnitude of the gradient approaches zero:

$$\underset{\parallel \mathbf{x}\parallel \to \infty }{lim}\parallel {\nabla }_{\mathbf{x}}\mathrm{Yat}(\mathbf{w},\mathbf{x})\parallel =0$$

This proves that for inputs $\mathbf{x}$ that are very far from the weight vector $\mathbf{w}$, the gradient becomes vanishingly small. The learning process is therefore stable, as distant outliers will not exert a significant influence on the weight updates. □

Appendix G.6 Proof of Universal Approximation Theorem for Yat-Product Networks

Theorem A6 (Universal Approximation Theorem for NMNs) Let $C\left(K\right)$ be the space of continuous functions on a compact set $K\subset {\mathbb{R}}^d$. A single-hidden-layer Neural-Matter Network (NMN) with Yat-product activation functions can approximate any function $f\in C\left(K\right)$ to any desired precision. That is, for any $f\in C\left(K\right)$ and any $\delta >0$, there exists an NMN function $g\left(\mathbf{x}\right)={\sum }_{i=1}^N{c}_i\mathrm{Yat}({\mathbf{w}}_i,\mathbf{x})$ such that ${sup}_{\mathbf{x}\in K}|f\left(\mathbf{x}\right)-g\left(\mathbf{x}\right)|<\delta$.

Proof. The proof relies on the theory of universal kernels. A continuous kernel K on a compact set $\mathcal{X}$ is defined as universal if its associated Reproducing Kernel Hilbert Space (RKHS), ${\mathcal{H}}_K$, is dense in the space of continuous functions $C\left(\mathcal{X}\right)$ with respect to the uniform norm. The span of functions of the form $g\left(\mathbf{x}\right)={\sum }_{i=1}^N{c}_{i}K({\mathbf{w}}_i,\mathbf{x})$ is dense in ${\mathcal{H}}_K$. Therefore, if the Yat-product kernel is universal, the set of NMN functions is dense in $C\left(K\right)$, which proves the theorem.

A key result from kernel theory [35] states that the product of two universal kernels is also universal. We have previously shown in Section G.2 that the Yat-product kernel K can be expressed as the pointwise product of two kernels:

$$K(\mathbf{w},\mathbf{x})=K_1(\mathbf{w},\mathbf{x})·K_2(\mathbf{w},\mathbf{x})$$

where:

• $K_1(\mathbf{w},\mathbf{x})={\langle \mathbf{w},\mathbf{x}\rangle }^2$ (the squared polynomial kernel)
• $K_2(\mathbf{w},\mathbf{x})={(\parallel \mathbf{w}-\mathbf{x}\parallel }^2{+ϵ)}^{-1}$ (a translation-invariant kernel)

We will now show that both $K_1$ and $K_2$ are universal kernels on any compact set $K\subset {\mathbb{R}}^d$.

1. $K_1$ is a Universal Kernel

The polynomial kernel $k_p(\mathbf{w},\mathbf{x})={(\langle \mathbf{w},\mathbf{x}\rangle +c)}^p$ is known to be universal for any $p\ge 1$ and $c>0$ [35]. Our kernel $K_1$ is a specific instance of the polynomial kernel family and is also universal on any compact subset of ${\mathbb{R}}^d$.

2. $K_2$ is a Universal Kernel

A sufficient condition for a continuous, translation-invariant kernel $k\left(\mathbf{z}\right)$ (where $\mathbf{z}=\mathbf{w}-\mathbf{x}$) to be universal is that its Fourier transform must be strictly positive almost everywhere [65]. In the proof of Mercer’s condition (Section G.2), we showed that $K_2$ has a non-negative Fourier transform via its integral representation:

$$K_2(\mathbf{w},\mathbf{x})={\int }_0^{\infty }{e}^{-ϵt}{e}^{{-t\parallel \mathbf{w}-\mathbf{x}\parallel }^2}dt$$

The integrand is a strictly positive function for all $t\ge 0$. The integral of a strictly positive function is strictly positive. Therefore, the Fourier transform of $K_2$ is strictly positive everywhere, which is a stronger condition than required. Thus, $K_2$ is a universal kernel.

Since both $K_1$ and $K_2$ are universal kernels, their product, the Yat-product kernel $K(\mathbf{w},\mathbf{x})$, is also universal. This implies that the span of functions generated by the NMN is dense in $C\left(K\right)$.

Therefore, for any continuous function $f\in C\left(K\right)$ and any $\delta >0$, there exists an NMN function $g\left(\mathbf{x}\right)$ such that ${sup}_{\mathbf{x}\in K}|f\left(\mathbf{x}\right)-g\left(\mathbf{x}\right)|<\delta$, which completes the proof. □

Appendix G.7 Information-Geometric Foundations of the Yat-Product

Appendix G.7.1 Definition and Geometric Interpretation

We consider probability distributions in the simplex ${\Delta }^{n-1}=\{\mathbf{p}\in {\mathbb{R}}_{\ge 0}^n:{\sum }_{i=1}^n{p}_i=1\}$. While information geometry traditionally employs the Fisher metric, we establish a novel connection to Euclidean geometry through the Yat-product.

Definition A1 (Yat-Product: Geometric Similarity Measure)For distinct distributions $\mathbf{p},\mathbf{q}\in {\Delta }^{n-1}$, the Yat-product is defined as:

$$Yat(\mathbf{p},\mathbf{q}){\displaystyle \frac{{(\mathbf{p}·\mathbf{q})}^2}{{\parallel \mathbf{p}-\mathbf{q}\parallel }_2^2}}$$

where:

• $\mathbf{p}·\mathbf{q}={\sum }_{i=1}^n{p}_i{q}_i$ measures distributional alignment
• ${\parallel \mathbf{p}-\mathbf{q}\parallel }_2^2={\sum }_{i=1}^n{(p_i-q_i)}^2$ quantifies Euclidean dissimilarity

This ratio captures the tension between distributional agreement and geometric separation.

Remark A1 (Singularity and Invariance Properties)When $\mathbf{p}=\mathbf{q}$, we define $Yat(\mathbf{p},\mathbf{q})\infty$ via the limit:

$$\underset{\mathbf{q}\to \mathbf{p}}{lim}Yat(\mathbf{p},\mathbf{q})=\infty$$

reflecting maximal self-similarity. The Yat-product exhibits two key properties:

• Symmetry: $Yat(\mathbf{p},\mathbf{q})=Yat(\mathbf{q},\mathbf{p})$
• Scale Invariance: Invariant under index permutation

Appendix G.7.2 Extremal Similarity Theorems

Theorem A7 (Minimal Similarity and Statistical Orthogonality) For distinct $\mathbf{p},\mathbf{q}\in {\Delta }^{n-1}$:

$$Yat(\mathbf{p},\mathbf{q})=0\iff supp\left(\mathbf{p}\right)\cap supp\left(\mathbf{q}\right)=\varnothing$$

Moreover, this condition implies information-theoretic divergence:

$$\begin{array}{cccccc}\hfill \mathrm{KL}(\mathbf{p}\parallel \mathbf{q})& =\infty ,\hfill & \hfill \mathrm{KL}(\mathbf{q}\parallel \mathbf{p})& =\infty ,\hfill & \hfill H(\mathbf{p},\mathbf{q})& =\infty \hfill \end{array}$$

Proof.(⇒) Assume $Yat(\mathbf{p},\mathbf{q})=0$. Since $\mathbf{p}\ne \mathbf{q}$, ${\parallel \mathbf{p}-\mathbf{q}\parallel }_2^2>0$. Thus ${(\mathbf{p}·\mathbf{q})}^2=0$⇒$\sum p_i{q}_i=0$. By non-negativity of probabilities, $p_i{q}_i=0$$\forall i$, hence $supp\left(\mathbf{p}\right)\cap supp\left(\mathbf{q}\right)=\varnothing$.

(⇐) Disjoint supports imply $\forall i:(p_i>0\Rightarrow q_i=0)$ and vice versa. Thus $\mathbf{p}·\mathbf{q}=0$, so $Yat(\mathbf{p},\mathbf{q})=0$.

The KL divergence $\mathrm{KL}(\mathbf{p}\parallel \mathbf{q})$ contains terms $log(p_i/q_i)$ where $p_i>0$ and $q_i=0$, causing divergence. Similar reasoning applies to $\mathrm{KL}(\mathbf{q}\parallel \mathbf{p})$ and cross-entropy $H(\mathbf{p},\mathbf{q})$ [63]. □

Theorem A8 (Maximal Similarity and Distributional Identity) For $\mathbf{p},\mathbf{q}\in {\Delta }^{n-1}$:

$$Yat(\mathbf{p},\mathbf{q})=\infty \iff \mathbf{p}=\mathbf{q}$$

When satisfied, information-theoretic consistency holds:

$$\begin{array}{cccccc}\hfill \mathrm{KL}(\mathbf{p}\parallel \mathbf{q})& =0\hfill & \hfill \mathrm{and}& \hfill & \hfill H(\mathbf{p},\mathbf{q})& =H\left(\mathbf{p}\right)\hfill \end{array}$$

Proof.(⇒) Suppose $Yat(\mathbf{p},\mathbf{q})\to \infty$. By Cauchy-Schwarz [62], $\mathbf{p}·\mathbf{q}\le {\parallel \mathbf{p}\parallel }_2{\parallel \mathbf{q}\parallel }_2\le 1$. Since the numerator is bounded, ${\parallel \mathbf{p}-\mathbf{q}\parallel }_2^2\to 0$, implying $\mathbf{p}=\mathbf{q}$.

(⇐) For $\mathbf{p}=\mathbf{q}$, consider ${\mathbf{q}}^{\left(k\right)}\to \mathbf{p}$. Then:

$$\mathbf{p}·{\mathbf{q}}^{\left(k\right)}\to {\parallel \mathbf{p}\parallel }_2^2\ge \frac{1}{n}>0(\sin \mathrm{ce}{\parallel \mathbf{p}\parallel }_2^2\ge \frac{1}{n}\mathrm{by}\mathrm{Cauchy}-\mathrm{Schwarz})$$

while $\parallel \mathbf{p}-{\mathbf{q}}^{\left(k\right)}{\parallel }_2^2\to 0$, so $Yat(\mathbf{p},{\mathbf{q}}^{\left(k\right)})\to \infty$.

When $\mathbf{p}=\mathbf{q}$, $log(p_i/q_i)=0$ for all i, so $\mathrm{KL}(\mathbf{p}\parallel \mathbf{q})=0$. Cross-entropy reduces to entropy when distributions are identical. □

Remark A2 (Duality of Orthogonality Concepts)The Yat-product unifies three distinct notions of orthogonality:

$$\begin{array}{cc}\hfill \mathrm{Euclidean}\text{:}& \mathbf{p}\perp \mathbf{q}\iff \mathbf{p}·\mathbf{q}=0\hfill \\ \hfill \mathrm{Combinatorial}\text{:}& supp\left(\mathbf{p}\right)\cap supp\left(\mathbf{q}\right)=\varnothing \hfill \\ \hfill \mathrm{Information}-\mathrm{Theoretic}\text{:}& \mathrm{KL}(\mathbf{p}\parallel \mathbf{q})=\infty \hfill \end{array}$$

Theorem A7 establishes their equivalence through $Yat(\mathbf{p},\mathbf{q})=0$. This contrasts with Fisher-based orthogonality, which depends on manifold curvature.

Remark A3 (Geometric-Information Duality)The Yat-product creates a bridge between geometric and probabilistic perspectives:

Appendix G.8 Diagrams

Figure A10. The core Scaled Dot-Product Attention calculation.

Figure A10. The core Scaled Dot-Product Attention calculation.

Figure A11. The Multi-Head Attention mechanism, which runs attention in parallel.

Figure A11. The Multi-Head Attention mechanism, which runs attention in parallel.

Figure A12. The position-wise Feed-Forward Network (MLP).

Figure A12. The position-wise Feed-Forward Network (MLP).

Figure A13. The complete Encoder block, showing how Multi-Head Attention and the FFN are combined using residual connections and layer normalization.

Figure A13. The complete Encoder block, showing how Multi-Head Attention and the FFN are combined using residual connections and layer normalization.

Figure A14. A standard ResNet "Basic Block" with a residual (skip) connection. This concept of bypassing layers is a precursor to the residual connections in Transformers.

Figure A14. A standard ResNet "Basic Block" with a residual (skip) connection. This concept of bypassing layers is a precursor to the residual connections in Transformers.

Figure A15. A CNMN Residual Block as used in AetherResNet: a YatConv layer ($n\to m$) followed by a linear Conv ($m\to m$), with no activation functions or normalization layers. The skip connection includes a projection if $n\ne m$.

Figure A15. A CNMN Residual Block as used in AetherResNet: a YatConv layer ($n\to m$) followed by a linear Conv ($m\to m$), with no activation functions or normalization layers. The skip connection includes a projection if $n\ne m$.

Figure A16. A YatFormer transformer block as used in AetherGPT: Yat-Attention followed by a Yat-MLP block, each with residual connections. No projection after attention, and no normalization layers.

Figure A16. A YatFormer transformer block as used in AetherGPT: Yat-Attention followed by a Yat-MLP block, each with residual connections. No projection after attention, and no normalization layers.

Figure A17. The Yat-MLP block as used in AetherGPT: a YatNMN layer ($d\to d$) followed by a linear layer ($d\to d$), with no expansion of the hidden dimension.

Figure A17. The Yat-MLP block as used in AetherGPT: a YatNMN layer ($d\to d$) followed by a linear layer ($d\to d$), with no expansion of the hidden dimension.

Figure A18. The Yat-Attention block as used in AetherGPT: input embeddings are projected to queries, keys, and values; Yat-product similarity is computed between queries and keys; softermax is applied; and the output is the weighted sum of values. No projection layer after attention.

Figure A18. The Yat-Attention block as used in AetherGPT: input embeddings are projected to queries, keys, and values; Yat-product similarity is computed between queries and keys; softermax is applied; and the output is the weighted sum of values. No projection layer after attention.