Computational Complexity of Self-Attention Algorithms for Efficient Transformer Architectures

Self-attention is fundamental to Transformer architectures, but its computational and memory requirements grow quadratically with sequence length. This paper presents a structured narrative review of computationally efficient self-attention algorithms. It establishes the mathematical foundations of standard self-attention and examines dense, sparse/local, low-rank projection, landmark/Nyström, kernel/linear, memory-efficient exact, and hierarchical mechanisms. The review compares these approaches in terms of arithmetic complexity, memory requirements, sequence-length scaling, approximation, connectivity, I/O behaviour, implementation overhead, and hardware dependence. It distinguishes algorithmic complexity reduction from implementation-level optimization and clarifies the conditions under which complexity is linear in sequence length. Analytical crossover conditions show how sequence length, representation dimension, mechanism-specific parameters, and implementation overhead influence practical advantage. The review demonstrates that efficient attention cannot be characterized solely as quadratic versus linear complexity and identifies research gaps in multidimensional complexity modelling, approximation-error trade-offs, I/O complexity, long-range dependency preservation, Pareto efficiency, and hardware-aware analysis.