Attention Might Offer Little Benefit for Graph Node Classification

1. Introduction

In academic research, when a breakthrough algorithm emerges in one field, its underlying scientific paradigm often diffuses rapidly to others [1,2]. Such cross-domain adoption typically attracts widespread attention and follow-up studies, regardless of its ultimate effectiveness, until the initial momentum fades or a new paradigm emerges. A decade ago, convolutional neural networks (CNNs) [3] played this role in computer vision. More recently, the attention mechanism, originally developed for language modeling, has triggered a comparable wave across machine learning. Attention has achieved strong results in language modeling [4] and was subsequently shown to be effective in computer vision [5], speech processing [6], and multimodal learning [7], particularly in domains that require modeling sequential data with global connectivity to capture long-range dependencies.

In graph node classification, attention mechanisms were rapidly adopted through Graph Attention Networks (GAT) [8], which apply attention over first-order neighbors. More recently, Graph Transformers [9] have emerged as a rapidly evolving research area, offering an alternative to traditional Message Passing Neural Networks (MPNNs) through the use of full attention and spawning a growing number of model variants. However, early work by Shchur et al. [10] showed that GAT can suffer from extremely low performance under certain weight initializations, as well as lower accuracy and higher variance than non-attention-based models on some datasets. Unlike GAT, which represents a single model, Graph Transformers encompass a broad and expanding family of models. Despite attracting significant research attention in recent years, concerns have been raised regarding their empirical effectiveness [11,12,13], scalability [14], and the complexity of their preprocessing pipelines [15]. Although some researchers have shifted away from developing ever more Graph Transformer variants, the broader belief in the effectiveness of attention for graph learning remains strong, often taken for granted [15], and attention-based architectures continue to proliferate.

In this position paper, we challenge the prevailing hype around attention mechanisms in graph node classification. Our experiments across multiple datasets reveal that attention often offers negligible benefit for node classification. We discuss potential reasons for this and advocate for a more cautious, evidence-driven adoption of attention mechanisms in this domain.

Structure of This Position Paper:

• Section 2 provides background on edge weight adjustment methods, positioning attention as one specific approach among many alternatives.
• Section 3 shows that attention for 1-hop neighbors is dispensable by experimenting on GAT [8] and its variants, and extends this finding to larger neighborhoods using Inception-style models, where attention remains unnecessary.
• Section 4 examines Graph Transformers and explains the reasons behind their performance shortcomings.

Limitations:

Although the paper discusses and evaluates both homophilic and heterophilic graph settings, the scope of the conclusions is limited to node classification and does not necessarily generalize to other graph learning tasks. In addition, our experiments focus on existing node classification settings, such as content-based and traffic-based node classification, and the findings may not directly apply to novel node classification scenarios.

2. Edge Weight Adjustment

Many Graph Neural Networks (GNNs) modify edge weights (or, equivalently, the adjacency entries used for propagation). This design choice appears under various names in the literature, including aggregators [16], graph filters [17], normalization [18], and scalers [17]. Despite the differing terminology, they all fundamentally adjust edge weights, which determine the relative contribution of each neighbor during aggregation. We classify these approaches as follows:

(1) Fixed/rule-based adjustment.

A common approach is to set edge scalings deterministically, most often as a function of node degrees. Representative examples include the symmetric normalization used by GCN [18], random-walk-based normalization [16], and direction-aware normalizations in Dir-GNN [19]. We provide a detailed summary of these normalizations in Appendix A.

Not all fixed aggregators are degree-based. There are several rules that do not rely on degree, such as min/max aggregators, softmax/softmin aggregators, and standard-deviation aggregators [17], as well as pooling and LSTM-based aggregators [16]. Other choices incorporate structural priors beyond degree, e.g., centrality-based graph shift operators [20]. Notably, PNA [17] argues that a single fixed graph filter can fail in certain regimes; to mitigate such failures, it combines multiple aggregators (mean, min, max, standard deviation) together with degree-based scalers to increase robustness.

(2) Learned adjustment.

Another line of work learns edge coefficients end-to-end during training. Attention-based models, such as GAT [8] and graph transformer variants [9], compute data-dependent edge weights (often normalized across neighbors) to adaptively modulate message contributions.

(3) Using given edge weights.

When edge weights are provided by the dataset, models may directly use these values; in unweighted graphs, a standard baseline is to set all edge weights to 1.

Unified view.

Across these cases, edge-weight adjustment can be viewed as applying an element-wise reweighting to the adjacency matrix, i.e., performing a Hadamard product between the adjacency matrix and an adjustment matrix that encodes either rule-based scalings or learned coefficients [21].

3. Attention of GAT Model

3.1. GAT as Learned Edge Weights

Graph Attention Networks (GATs) [8] integrate attention into message passing, allowing each node to assign different importance to its directly connected neighbors, as shown in Figure 1(a). The layer-wise propagation rule of a Graph Attention Network (GAT) [8,22] can be written in matrix form as

$${\mathit{H}}^{(l+1)}=\sigma \left({\mathit{A}}^{\left(l\right)}{\mathit{H}}^{\left(l\right)}{\mathit{W}}^{\left(l\right)}\right)=\sigma \left(({\mathit{\alpha }}^{\left(l\right)}\odot \mathit{A}){\mathit{H}}^{\left(l\right)}{\mathit{W}}^{\left(l\right)}\right),$$

(1)

where ${\mathit{H}}^{\left(l\right)}={\overrightarrow{h}}_1,{\overrightarrow{h}}_2,...,{\overrightarrow{h}}_N\in {\mathbb{R}}^{N\times D}$ denotes the node feature matrix at layer l, containing one D-dimensional representation per node, with ${\mathit{H}}^{\left(0\right)}=\mathit{X}$, the original node features. ${\mathit{W}}^{\left(l\right)}$ is a layer-specific trainable weight matrix. The matrix ${\mathit{A}}^{\left(l\right)}\in {\mathbb{R}}^{N\times N}$ collects the normalized attention coefficients produced by the l-th attention mechanism and can be written as the element-wise product ${\mathit{\alpha }}^{\left(l\right)}\odot \mathit{A}$, where $\mathit{A}$ is the adjacency matrix and ${\mathit{\alpha }}^{\left(l\right)}$ contains the learned attention weights on edges. $\sigma (·)$ denotes an activation function.

This update rule implements a message-passing operation on the underlying graph: each node aggregates linearly transformed features from its neighbors, but the aggregation weights are dataset-dependent and learned through the attention mechanism, rather than being fixed by a predefined normalization of $\mathit{A}$, as in GCN [18] or GraphSage [16].

3.2. Attention in GAT is Dispensable

We now examine the attention matrix ${\mathit{\alpha }}^{\left(l\right)}$ in Equation 1. Each entry ${\alpha }_{ij}$ represents a normalized attention coefficient computed by the attention mechanism [8]:

$${\alpha }_{ij}=Softmax\left(LeakyReLU\left(e_{ij}\right)\right),$$

(2)

where $e_{ij}$ denotes the unnormalized attention score that measures the importance of node $\mathit{j}$’s feature to node $\mathit{i}$. Since GAT restricts attention computations to the first-order neighbors of $\mathit{i}$ (including $\mathit{i}$), the mechanism effectively implements masked attention to encode local graph structure.

From Equation 2, the attention mechanism in GAT can be decomposed into three key steps:

• Step 1: Computation of learnable weights $e_{ij}$;
• Step 2: Application of the LeakyReLU activation;
• Step 3: Softmax normalization to obtain ${\alpha }_{ij}$.

To test whether the learnable weights in Step 1 are truly necessary, we conducted an ablation study by replacing the learnable weights $e_{ij}$ with either uniform weights (all set to 1) or random weights sampled from a uniform distribution in the range [0.0001, 10000]. We further replaced the Softmax normalization in Step 3 with several alternative normalization schemes : symmetric normalization (Sym) from GCN [18], row normalization (Row) from GraphSAGE [16], directed normalization (Dir) from Dir-GNN [19], and a no-normalization configuration (None). Results are reported in Table 1. Because GAT attention weights can be negative and thus incompatible with these normalization schemes, we take their absolute values before applying normalization.

To ensure a fair comparison, all models use the same implementation. Detailed hyperparameter settings are provided in Appendix B.2.2.9. Table 1 presents the performance of GAT variants under different weight schemes and normalization strategies, showing the results for the best hyperparameter configurations in each case. The results demonstrate that model performance remains largely unaffected by the removal of learned attention weights. In most cases, uniform and random weights achieve comparable or even superior performance to attention-based weighting. Only on the Amazon-Photo and Amazon-Computer datasets do we observe marginal performance differences (less than 0.5%).

Furthermore, Table 1 reveals that the choice of normalization scheme exerts a stronger influence on model performance than the choice of weighting strategy. Specifically, the Dir-norm and Sym-norm consistently outperform Softmax-norm and No-norm on the PubMed, CS, and Physics datasets. In contrast, for the Telegram dataset, the No-norm configuration achieves a noticeably higher performance than all other normalization methods, regardless of whether the edge weights are uniform, random, or learned. These findings suggest that, for these datasets, the attention mechanism is entirely dispensable, while normalization plays a much more critical role in determining performance. The competitive results of uniform edge weighting on datasets such as PubMed, CS, and Physics further indicate that simply assigning uniform weights to edges is both sufficient and effective—making the learning of attention-based weights an unnecessary complexity.

Collectively, since the evaluated datasets cover widely used node classification benchmarks, including citation networks, co-author networks, co-purchase graphs, and social networks(Appendix B.1), they consistently indicate that GAT’s attention weights contribute minimally to overall performance. This suggests that attention mechanisms over local (1-hop) neighborhoods are largely unnecessary for effective graph node classification. Sensitivity experiments on the number of attention heads, presented in Table A4 (in the Appendix), further show that increasing the number of heads generally yields little performance improvement and can even lead to slightly worse results for some datasets.

As the GAT model [8] is primarily evaluated on homophilic datasets, the above ablation results, largely based on homophilic graphs, already call into question the necessity of attention in its original formulation. To rule out the possibility that the observed behavior is driven by homophily or heterophily, we extend our study to four heterophilic datasets, with results reported in Table A7.

Across the Chameleon, Squirrel, and Arxiv-Year datasets, competitive performance can be achieved without attention. Multi-head attention yields modest gains on these datasets, with improvements of approximately 6% on Arxiv-Year, 4% on Squirrel, and 1% on Chameleon, compared to only around 0.5% on homophilic graphs. In contrast, the Roman-Empire dataset exhibits a substantial improvement of over 30%, indicating a clear benefit from attention mechanisms. We attribute this to the nature of its node features: as dense word embeddings, they encode rich semantic information, making fine-grained neighbor discrimination more beneficial than in datasets with sparse features. Nevertheless, prior work [19,23] reports accuracies of up to 93.58% on Roman-Empire without using attention, suggesting that such gains may not be uniquely attributable to attention mechanisms. While further research is needed to fully understand this phenomenon, this evidence suggests that the benefits of attention mechanisms may be more limited than commonly assumed.

For Roman-Empire, LargeScaleNet [23] achieves state-of-the-art performance by learning on directed multi-scale learning with Jumping Knowledge Connections, reporting an accuracy of 93.58±0.24. When replacing the uniform edge weights with attention weights while keeping all other settings identical, performance changes only marginally to 93.60±0.32, as shown in Table A10 (Appendix). This negligible difference further supports the conclusion that attention mechanisms are not the determining factor, even on a dataset where simpler attention-based models show large gains.

3.3. Attention is Dispensable for Extended Neighborhoods

In Section 3, we showed attention to immediate neighbors is unnecessary. Before examining graph transformers with full attention, we evaluate DiGib [24], an inception-style framework that aggregates information from both immediate neighbors and nodes that are not directly connected, thereby expanding the receptive field, as illustrated in Figure 1(b). Specifically, DiGib connects nodes that share a common predecessor (i.e., both point to the same node) or a common successor (i.e., both are pointed to by the same node).

The DiGib variants retain the same aggregation structure but replace edge weights with learned attention weights, uniform weights, or random weights sampled from $[{10}^{-4},{10}^4]$, and apply the same normalization variants as in Section 3.

Although DiGib was originally designed for directed graphs, it is also compatible with undirected graphs. We therefore evaluate it on four directed and four undirected datasets. The Amazon-Computer dataset is excluded due to GPU memory limitations on the NVIDIA A40 platform. To ensure a fair comparison, all models share the same implementation, and detailed hyperparameter settings are provided in Appendix B.2.2.10. Table 2 reports the results for the best hyperparameter configurations in each case.

As shown in Table 2, across all datasets except Amazon-Photo (where differences are below $0.2\%$), the non-attention variants match or outperform the attention-based version. Notably, on three datasets (CoraML, CiteSeer, and Telegram), learned attention consistently underperforms uniform weighting and, in some cases, even random weighting. This suggests that the weight-learning mechanism may adversely affect node classification, potentially due to sparse connectivity and limited labeled data, as reflected in the dataset statistics reported in Table A1. Sensitivity experiments on the number of attention heads, presented in Table A5 (in the Appendix) show that increasing the number of heads generally yields little performance improvement. Experiments on heterophilic datasets are reported in Table A9(in Appendix D.

Overall, these results confirm that attention provides no tangible benefit even when the receptive field is expanded.

3.4. Why Attention Can Fail in GAT and Inception Models

Node classification benchmarks vary in which signals are most predictive of the labels [26]. For example, citation graphs are often more influenced by node attributes (e.g., high-dimensional sparse bag-of-words or one-hot encodings), whereas social networks can rely more on structural cues (e.g., degree and local connectivity patterns) [21]. This difference helps explain why attention mechanisms in GAT and Inception-style GNNs behave inconsistently across datasets.

When prediction is dominated by high-dimensional sparse features, the precise feature magnitudes are often secondary to preserving which dimensions are active. In this regime, attention-based neighbor reweighting may yield limited gains. Conversely, when labels correlate strongly with scale-sensitive structural statistics such as node degree, normalization and reweighting within message passing may inadvertently attenuate or distort these signals. For instance, in degree-driven networks such as Telegram, where strong performance can be obtained using an MLP trained solely on degree features [23], unnormalized aggregation can better preserves degree information and can outperform attention-based or normalized variants.

4. Graph Transformers

4.1. Graph Transformers As Learned Edge Weights

Li et al. [9] treats all node pairs as connected (Fig. Figure 1(c)), shifting from sparse adjacency to dense attention and inspiring numerous Graph Transformers(GTs) [27,28,29]. The propagation rule of GTs [30] is:

$${\mathit{H}}^{(l+1)}=\sigma \left(({\mathit{\alpha }}^{\left(l\right)}\odot \mathbf{1}){\mathit{H}}^{\left(l\right)}{\mathit{W}}^{\left(l\right)}\right),$$

(3)

where $\mathbf{1}\in {\mathbb{R}}^{N\times N}$ is the all-ones matrix. ${\mathit{H}}^{\left(l\right)}\in {\mathbb{R}}^{N\times D}$ denotes the node feature matrix at layer l, with ${\mathit{H}}^{\left(0\right)}=\mathit{X}$ [9] or ${\mathit{H}}^{\left(0\right)}$ formed by combining $\mathit{X}$ with additional features such as positional/structural encodings [31]. GTs learn global edge weights, unlike local methods, introducing two issues: (1) Loss of structural information, and (2) Limited scalability due to quadratic complexity. Given these challenges, community perspectives on GTs are divided: some remain optimistic, developing architectural variants to overcome current limitations, while others express concern over weak empirical gains, high computational costs, and excessive resource demands.

4.1.1. Optimism in Graph Transformers

GTs are often regarded as the next frontier in graph learning—aiming to overcome the limited expressiveness, over-smoothing, and oversquashing problems of MPNNs [12,32]. Inspired by the dominance of Transformers in language and vision, researchers have attempted to replicate this success for graph data, positioning GTs as successors to “traditional” GNNs [33,34]. Recent work mainly develops along two lines: improving structural encoding to recover information lost by removing the adjacency matrix, and enhancing scalability to overcome the $\mathcal{O}\left(N^2\right)$ cost of full attention (see Appendix E for details). This persistent optimism contrasts with the limited fundamental scrutiny of Graph Transformers and the actual effectiveness of attention in graph learning—precisely the gap this position paper seeks to address.

4.1.2. Pessimism in Graph Transformers

In contrast, a growing body of research highlights the limitations of Graph Transformers. They are often criticized for their high computational complexity [14], which restricts scalability to large graphs. Empirical studies have shown that well-tuned MPNNs can outperform GTs in both accuracy and efficiency [11,12,32], though such comparisons rarely question the design assumptions underlying attention on graphs. Furthermore, Xing et al. [13] identify an over-globalization problem: attention mechanisms often overemphasize distant nodes while underweighting nearby, more informative ones, leading to degraded representational quality. Combined with growing model complexity and preprocessing overhead, these issues cast serious doubt on whether architectural sophistication truly translates into better graph learning performance.

Together, these perspectives illustrate the ongoing tension in the GT research community between excitement for innovation and practical concerns.

4.2. Why GT Fails

Recent empirical studies [11,12,32] have conducted extensive evaluations showing that Graph Transformers often underperform strong GNN baselines. Building on these findings, this position paper does not reproduce such experiments but instead infers GT behavior from prior evidence on GATs and Inception-style models.

4.2.0.1. From Local to Global: Amplifying a Flawed Premise

Section 3 and Section 3.3 show that learned attention weights provide little benefit, and can even be detrimental, both in local Graph Attention Networks (GATs) and in multi-scale Inception models. Graph Transformers (GTs) extend this flawed premise to its extreme by replacing locality with full global attention.

In doing so, GTs remove valuable local inductive biases while retaining the same ineffective attention-weighting mechanisms. This combination aggravates the problem: their global receptive fields promote aggregation over irrelevant nodes, often washing out the informative local structures that are crucial for effective graph reasoning. The following section examines several factors contributing to these shortcomings.

4.2.1. Limitation of Gradient Descent

Graph Transformers (GTs) are designed to transcend fixed graph structures via global attention and dynamic connectivity [9]. However, gradient descent exhibits a fundamental limitation in realizing this objective.

Once global connections are initialized, gradient descent struggles to unlearn them by pruning irrelevant edges. Both empirical and theoretical analyses show that gradient-based dynamics tend to preserve connectivity and overfit to spurious edges, even in the large-data regime [26,35]. Our experiments corroborate this behavior: with proper normalization, random edge weights as small as 0.0001 perform equivalently to those as large as 10000, highlighting gradient descent’s inability to achieve exact sparsity. Although weights can take arbitrarily small positive or negative values during training, reaching an exact zero is exceedingly unlikely under standard gradient-based optimization.

This reveals a fundamental paradox: global attention, intended to enable dynamic structure learning, effectively degenerates into a fixed fully connected structure. When the ground-truth graph is sparse, Graph Transformers trained with gradient descent cannot recover it. Lacking strong local inductive biases, this over aggregation results in suboptimal generalization. Consequently, GTs become trapped in a static, fully connected regime rather than learning meaningful task-specific structures.

4.2.2. Language vs. Graph: Fundamental Mismatch

Structure Enhancement vs. Destruction:

Transformers enhance linear language sequences by inducing structure through attention, where heads capture syntax and coreference [36]. Graphs, however, already possess rich topology. Global attention disrupts this inherent structure [yingTransformersReallyPerform2021], effectively destroying it.

Semantic entanglement vs. local relevance:

Language tokens encode long-range dependencies (negation, coreference) where distant context critically shapes meaning [attention17]. Graph nodes, by contrast, especially for node classification, typically rely on local neighborhoods; distant nodes often introduce noise rather than signal.

Content-based classification: The presence or absence of a feature is often more informative than its exact magnitude. Local neighbors generally carry the signals defining a node’s class, whereas distant neighbors may belong to different classes. Aggregating information globally can introduce noise from these inter-class nodes, diluting discriminative features.

Structure-based classification: Global attention (GT) can disrupt scale-sensitive structural information, such as node degrees. Although incorporating PE/SE to input features initially preserves this information, a single global attention step mixes all nodes’ degrees, making it impossible to recover per-node degree information. In contrast, local message passing progressively captures richer structural statistics [21,26].

5. Alternative Views

This paper argues that attention mechanisms are largely unnecessary for graph node classification. At the same time, several commonly held positions in the research community support the opposite conclusion. In this section, we briefly outline these opposing views without endorsing them.

View 1: Attention improves performance and expressivity. A common position is that attention increases the capacity of graph models by allowing them to assign data-dependent weights [8] to neighbors instead of using fixed, normalization-based weights. From this perspective, attention is expected to provide stronger function approximation power and better empirical performance than simpler message-passing schemes.

View 2: Attention is needed for long-range interactions. Another widely held view is that global attention, as used in Graph Transformers, is essential for modeling long-range dependencies on graphs. In this line of reasoning, attention is considered necessary to propagate information between distant but semantically related nodes, overcoming the over-smoothing and oversquashing issues of traditional GNNs.

View 3: Attention’s cross-domain success justifies its use on graphs. Because attention mechanisms underpin state-of-the-art models in natural language processing, computer vision, and multimodal learning, many researchers expect similar benefits to carry over to graph-structured data. This expectation motivates continued development of attention-based graph architectures [15] and reinforces the assumption that attention should also be effective for node classification.

These views stand in contrast to the stance taken in this work. Our empirical analysis focuses specifically on graph node classification and leads to a different conclusion: in this setting, attention often does not deliver the improvements that these prevailing assumptions would predict.

6. Conclusions

This position paper systematically challenges the prevailing faith in attention mechanisms for graph node classification. Through three levels of analysis, we demonstrate that learned attention weights are largely dispensable. Graphs are fundamentally different from language in how and why attention mechanisms apply. For the specific task of node classification, we observe that citation networks built on sparse features rely primarily on local structural cues, where precise attention adds little value. In contrast, social networks emphasize degree and connectivity patterns, which global attention can easily distort. For global attention to function effectively in graphs, it would need the capacity to unlearn, that is, to reduce attention to exactly zero when unnecessary, a behavior that standard gradient-based optimization cannot reliably achieve.

While criticism of Graph Transformers is not new, the attention mechanism itself has escaped fundamental scrutiny. Our results indicate that attention is generally unnecessary for node classification and is only demonstrably effective in graphs whose node features are dense language word embeddings. This provides strong evidence against the prevailing assumption that attention is inherently beneficial across graph domains.

No Normalization

The simplest approach is to use the raw adjacency matrix without any normalization [37]: $f_1\left(A\right)=A$. In this case, the node feature update rule becomes:

$$h_i^{(l+1)}=\sigma \left(\sum _{j\in \mathcal{N}\left(i\right)}{h}_j^{\left(l\right)}{W}^{\left(l\right)}\right)$$

The aggregation directly sums neighboring features, leading to larger feature magnitudes for higher degree nodes. With homogeneous features $h_i^{\left(0\right)}=1$, node representations become proportional to degrees.

Row Normalization

Row normalization [16] scales each row of the adjacency matrix by the inverse of node degree: $f_2\left(A\right)=D^{-1}A$. The node feature update rule becomes:

$$h_i^{(l+1)}=\sigma \left({\displaystyle \frac{{\sum }_{j\in \mathcal{N}\left(i\right)}{h}_j^{\left(l\right)}{W}^{\left(l\right)}}{d_i}}\right)$$

For this formulation, the aggregated information represents the mean of neighboring features rather than their sum. Node degrees no longer directly influence feature magnitudes. With homogeneous features $h_i^{\left(0\right)}=1$, all nodes get identical representations.

Symmetric Normalization

Symmetric normalization [18] applies: $f_3\left(A\right)=D^{-1/2}A{D}^{-1/2}$. The node feature update rule becomes:

$$h_i^{(l+1)}=\sigma \left(\sum _{j\in \mathcal{N}\left(i\right)}{\displaystyle \frac{h_j^{\left(l\right)}{W}^{\left(l\right)}}{\sqrt{d_i{d}_j}}}\right)$$

The neighbor’s influence is determined by both degrees—if a neighbor’s degree is much larger than the center node’s, its feature weight becomes smaller than in row normalization.

Directed Normalization

For directed graphs, Rossi et al. [19] proposes: $f_4\left(A\right)=D_{in}^{-1/2}A{D}_{out}^{-1/2}$. The node feature update rule becomes:

$$h_i^{(l+1)}=\sigma \left(\sum _{j\in \mathcal{N}\left(i\right)}{\displaystyle \frac{h_j^{\left(l\right)}{W}^{\left(l\right)}}{\sqrt{d_i^{in}{d}_j^{out}}}}\right)$$

This distinguishes between in-degree and out-degree for more accurate normalization in directed graphs.

Softmax Normalization

Softmax normalization is introduced by GAT [8] to make coefficients easily comparable across different nodes, they normalize them across all choices of $\mathit{j}$ using the softmax function:

$${\alpha }_{ij}={softmax}_j\left(e_{ij}\right)={\displaystyle \frac{exp\left(e_{ij}\right)}{{\sum }_{k\in {\mathcal{N}}_i}exp\left(e_{ik}\right)}},$$

where ${\mathcal{N}}_i$ is node $\mathit{i}$’s 1-hop neighbors(including $\mathit{i}$), $e_{ij}$ is the original weights.

In summary, row normalization of adjacency matrices, when applied with uniform node features, results in loss of structural information since all nodes become indistinguishable. While using the unnormalized adjacency matrix preserves both degree information and feature distinctions, it can lead to numerical instability since the eigenvalues of $f\left(A\right)$ may grow or diminish exponentially, rather than being bounded within [-1, 1]. This instability can affect the training of graph neural networks.

Table A1. Dataset statistics. #Train is number of train nodes.

Dataset	#Nodes(#Train)	#Edges	#Feature	#Class
CoraML	2995(140)	8416	2879	7
CiteSeer	3312(120)	4715	3703	6
Telegram	245(145)	8912	1	4
WikiCS	11701(580)	297110	300	10
Coauthor-CS	18333(8793)	182121	6805	15
Coauthor-Phy.	34493(16555)	495924	8415	5
PubMed	19717(60)	88648	500	3
Amazon-Photo	7650(3669)	238162	745	8
Amazon-Comp.	13752(6595)	491722	767	10

Table A1. Dataset statistics. #Train is number of train nodes.

Dataset	#Nodes(#Train)	#Edges	#Feature	#Class
CoraML	2995(140)	8416	2879	7
CiteSeer	3312(120)	4715	3703	6
Telegram	245(145)	8912	1	4
WikiCS	11701(580)	297110	300	10
Coauthor-CS	18333(8793)	182121	6805	15
Coauthor-Phy.	34493(16555)	495924	8415	5
PubMed	19717(60)	88648	500	3
Amazon-Photo	7650(3669)	238162	745	8
Amazon-Comp.	13752(6595)	491722	767	10

Appendix B.1. Datasets

Our experiments are based on a diverse collection of directed and undirected graph benchmarks commonly used for node classification.

• Citation networks. CiteSeer, CoraML, PubMed and WikiCS are citation graphs.
  For Citeseer and Cora-ML, we use the splits specified in the DiGCN(ib) paper [24]. For the WikiCS dataset, the splits are described in PubMed is obtained from the Deep Graph Library (DGL) and treated as an undirected citation network. In all citation datasets, nodes represent papers and edges denote citation relationships. Node features are bag-of-words representations, and labels correspond to paper topics. For PubMed, we generate 10 random splits following the same protocol as CiteSeer and CoraML: 20 labeled nodes per class for training, 30 per class for validation, and the remaining nodes for testing.
• Social network. Telegram is a directed social network dataset from MagNet [38].
• Coauthor networks. Coauthor-CS and Coauthor-Physics (denoted as CS and Physics in the main paper) are co-authorship graphs derived from the Microsoft Academic Graph released for the KDD Cup 2016 challenge. Nodes represent authors, edges indicate co-authorship relations, node features aggregate keywords from each author’s papers, and labels correspond to the author’s primary field of study.
• Co-purchasing networks. Amazon-Computer and Amazon-Photo (denoted as Computer and Photo in the main paper) are co-purchase graphs introduced in Shchur et al. [10]. Nodes represent products, edges connect frequently co-purchased items, node features are bag-of-words representations of product reviews, and labels indicate product categories.

Among these datasets, CoraML, CiteSeer, WikiCS, and Telegram are originally directed graphs, whereas PubMed, Photo, Computers, Coauthor-CS, and Coauthor-Physics are originally undirected. All datasets are evaluated using cross-validation with 10 train/validation/test splits, except for WikiCS, which provides 20 splits as in the original source. We use the official splits when available; otherwise, we randomly generate 10 splits. For experiments involving GAT variants, we convert directed graphs into undirected ones by augmenting each edge with its reverse counterpart.

Appendix B.2. Code and Hyperparameters

GATConv Variants Implementation.

We implement three GATConv variants: learned attention (standard GAT), uniform weights, and random weights. Implementation is based on PyTorch Geometric GATConv [39]. Specific code for the GATConv variants and GAT model is in Supplementary Material.

All variants use the same architecture and hyperparameters (see Table A2 and Table A3) to ensure a fair comparison. Directed graphs can be converted to undirected by adding reverse edges. All models use a hidden dimension of 128. Training is performed for up to 1500 epochs with early stopping. If validation performance does not improve for 80 epochs, the learning rate is reduced by a factor of 0.5. Model variants differ in whether a ReLU activation function is applied after each GNN layer.

Our experimental design prioritizes (i) computational efficiency, (ii) fairness across model variants by using comparable hyperparameter search spaces and selecting the best-performing configuration for each model, and (iii) practical GPU constraints. In particular, for large graphs, using more than one attention head is often prohibitive in terms of memory, and thus we restrict the number of heads to one in such cases.

GAT Variants

Table A2 summarizes the hyperparameter configurations that were exhaustively explored for the experiments on GAT variants (Table 1).

Table A2. Hyperparameters used for the GAT variants evaluated in Table 1.

Dataset	Layer	Learning Rate	Heads
Coauthor-CS	2	0.005	1, 2, 8
Coauthor-Physics	2	0.005	1, 2, 8
PubMed	2	0.005	1, 2, 8
Amazon-Photo	2, 3	0.005	1, 2, 4
Amazon-Computer	2, 3	0.005	1, 2, 8
WikiCS	2	0.005	1, 2, 4
Telegram	1, 2, 3, 4, 5	0.01, 0.005	1, 16
ArXiv-Year [40]	3	0.005	1, 8
Roman-Empire [40]	2	0.005	1, 8
Chameleon [41]	2	0.005	1, 8
Squirrel [41]	2	0.005	1, 8

Table A2. Hyperparameters used for the GAT variants evaluated in Table 1.

Dataset	Layer	Learning Rate	Heads
Coauthor-CS	2	0.005	1, 2, 8
Coauthor-Physics	2	0.005	1, 2, 8
PubMed	2	0.005	1, 2, 8
Amazon-Photo	2, 3	0.005	1, 2, 4
Amazon-Computer	2, 3	0.005	1, 2, 8
WikiCS	2	0.005	1, 2, 4
Telegram	1, 2, 3, 4, 5	0.01, 0.005	1, 16
ArXiv-Year [40]	3	0.005	1, 8
Roman-Empire [40]	2	0.005	1, 8
Chameleon [41]	2	0.005	1, 8
Squirrel [41]	2	0.005	1, 8

DiGib Variants

Table A3 reports the hyperparameter configurations used for DiGib variants.

Table A3. Hyperparameters used for the DiGib variants evaluated in Table 2.

Dataset	Layer	Learning Rate	Heads
CoraML	3	0.01	1, 8, 16
CiteSeer	3	0.01	1, 8, 16
Telegram	1-5	0.005, 0.01	1, 8, 16
WikiCS	2, 3	0.005	1, 2
Amazon-Photo	2, 3	0.005	1
PubMed	2	0.005	1, 8
Coauthor-CS	2	0.005	1
Coauthor-Physics	2	0.005	1
Roman-Empire [40]	2	0.005	1, 8
Chameleon [41]	2	0.005	1, 8
Squirrel [41]	2	0.005	1, 8

Table A3. Hyperparameters used for the DiGib variants evaluated in Table 2.

Dataset	Layer	Learning Rate	Heads
CoraML	3	0.01	1, 8, 16
CiteSeer	3	0.01	1, 8, 16
Telegram	1-5	0.005, 0.01	1, 8, 16
WikiCS	2, 3	0.005	1, 2
Amazon-Photo	2, 3	0.005	1
PubMed	2	0.005	1, 8
Coauthor-CS	2	0.005	1
Coauthor-Physics	2	0.005	1
Roman-Empire [40]	2	0.005	1, 8
Chameleon [41]	2	0.005	1, 8
Squirrel [41]	2	0.005	1, 8

(1) Encoding strategies for recovering structural information.

By discarding the adjacency matrix $\mathit{A}$, GTs lose original graph connectivity. To compensate, models commonly employ positional and structural encodings (PE/SE) [27,31]. These encodings infuse node features with graph-aware signals such as Laplacian eigenvectors [42], degree centrality [31], or edge regularization [43], helping attention mechanisms infer relative positions and relationships. However, these strategies often introduce costly pre-processing [44], sometimes with $\mathcal{O}\left(N^3\right)$ complexity, and their ability to effectively or fully restore the lost structural information remains questionable.

(2) Improving scalability.

Full attention incurs $\mathcal{O}\left(N^2\right)$ time and memory complexity, limiting applicability to large graphs. To mitigate this, scalable architectures have been proposed, such as GraphMamba [45], NodeFormer [43], SGFormer [44], etc. For more details on Graph Transformers, see [15,46] or read a survey [30].

Table A6. Accuracy (mean±standard deviation) of GAT variants under different weight settings and normalization schemes on heterophilic datasets. yellow cells correspond to the original GAT configuration. Results in bold exceed the baseline mean, while underlined results exhibit strong distributional overlap with the baseline.

Datasets	Weight	Heads	Softmax	Dir	Sym	Row	None
Arxiv-Year Directed (Heterophilic)	Attention	1	44.7±0.2	52.7±0.7	30.1±4.1	40.1±1.1	30.1±1.7
	Attention	8	43.3±0.9	59.2±0.5	36.1±0.7	45.8±0.6	36.8±2.5
	Uniform	1	30.0±1.7	50.1±0.2	47.4±0.2	29.7±1.1	42.9±0.1
	Random	1	35.3±0.2	48.2±0.3	42.5±0.3	29.1±0.8	41.5±0.2
Roman-Empire Directed (Heterophilic)	Attention	1	46.4±3.7	55.4±2.4	50.5±3.1	43.8±3.0	55.0±2.4
	Attention	8	71.5±2.0	82.6±0.9	70.3±4.7	70.3±3.0	79.9±0.6
	Uniform	1	32.9±0.3	36.3±0.4	33.4±0.5	32.8±0.4	36.1±0.6
	Random	1	20.9±0.6	32.3±0.6	29.8±1.0	29.9±0.5	30.0±0.4
Chameleon Undirected (Heterophilic)	Attention	1	67.1±1.5	65.4±1.7	66.4±2.2	66.0±3.5	59.2±5.0
	Attention	8	68.8±2.1	66.8±3.2	68.8±3.1	67.5±1.6	59.4±2.8
	Uniform	1	67.9±2.6	67.5±2.5	67.6±2.4	68.0±2.2	66.4±2.2
	Random	1	40.3±2.7	65.2±2.5	66.8±2.0	66.9±1.9	64.1±2.3
	Random	8	56.5±2.6	67.2±2.6	67.7±2.3	68.0±2.2	65.3±2.2
Squirrel Undirected (Heterophilic)	Attention	1	56.1±2.1	55.8±0.9	57.2±2.1	55.1±2.3	39.9±4.5
	Attention	8	60.5±1.8	55.1±1.9	58.0±1.2	57.0±1.8	42.5±1.8
	Uniform	1	53.1±1.2	56.4±1.3	56.6±1.7	53.4±0.9	45.9±1.6
	Random	1	29.0±1.7	54.5±1.2	54.4±1.2	50.3±1.5	45.6±1.8
	Random	8	38.7±1.2	56.3±1.6	55.9±1.3	53.6±1.2	43.3±2.3

Table A6. Accuracy (mean±standard deviation) of GAT variants under different weight settings and normalization schemes on heterophilic datasets. yellow cells correspond to the original GAT configuration. Results in bold exceed the baseline mean, while underlined results exhibit strong distributional overlap with the baseline.

Datasets	Weight	Heads	Softmax	Dir	Sym	Row	None
Arxiv-Year Directed (Heterophilic)	Attention	1	44.7±0.2	52.7±0.7	30.1±4.1	40.1±1.1	30.1±1.7
	Attention	8	43.3±0.9	59.2±0.5	36.1±0.7	45.8±0.6	36.8±2.5
	Uniform	1	30.0±1.7	50.1±0.2	47.4±0.2	29.7±1.1	42.9±0.1
	Random	1	35.3±0.2	48.2±0.3	42.5±0.3	29.1±0.8	41.5±0.2
Roman-Empire Directed (Heterophilic)	Attention	1	46.4±3.7	55.4±2.4	50.5±3.1	43.8±3.0	55.0±2.4
	Attention	8	71.5±2.0	82.6±0.9	70.3±4.7	70.3±3.0	79.9±0.6
	Uniform	1	32.9±0.3	36.3±0.4	33.4±0.5	32.8±0.4	36.1±0.6
	Random	1	20.9±0.6	32.3±0.6	29.8±1.0	29.9±0.5	30.0±0.4
Chameleon Undirected (Heterophilic)	Attention	1	67.1±1.5	65.4±1.7	66.4±2.2	66.0±3.5	59.2±5.0
	Attention	8	68.8±2.1	66.8±3.2	68.8±3.1	67.5±1.6	59.4±2.8
	Uniform	1	67.9±2.6	67.5±2.5	67.6±2.4	68.0±2.2	66.4±2.2
	Random	1	40.3±2.7	65.2±2.5	66.8±2.0	66.9±1.9	64.1±2.3
	Random	8	56.5±2.6	67.2±2.6	67.7±2.3	68.0±2.2	65.3±2.2
Squirrel Undirected (Heterophilic)	Attention	1	56.1±2.1	55.8±0.9	57.2±2.1	55.1±2.3	39.9±4.5
	Attention	8	60.5±1.8	55.1±1.9	58.0±1.2	57.0±1.8	42.5±1.8
	Uniform	1	53.1±1.2	56.4±1.3	56.6±1.7	53.4±0.9	45.9±1.6
	Random	1	29.0±1.7	54.5±1.2	54.4±1.2	50.3±1.5	45.6±1.8
	Random	8	38.7±1.2	56.3±1.6	55.9±1.3	53.6±1.2	43.3±2.3

In addition, GTs have been extended to new domains, such as directed graphs [47] and few-shot learning [48]. The field continues to expand rapidly, with new models proposed each year [49,50]. Although many studies acknowledge limitations of existing GTs [15,32], attention is still widely treated as an inherently powerful mechanism. Consequently, researchers either design new GT variants [46] or propose alternative attention-based architectures [15].

Summary

Overall, despite rapid progress, existing Graph Transformer designs remain heavily focused on engineering solutions for structure and efficiency, while the fundamental role and necessity of attention for graph learning remain insufficiently understood.

Table A7. Accuracy (mean ± standard deviation) of DiGib [25] variants with different weight settings and normalization schemes evaluated on heterophilic datasets. yellow cells correspond to the best attention-weighted configuration. Results in bold exceed the baseline mean, while underlined results exhibit strong distributional overlap with the baseline.

Datasets	Weight	Heads	Dir	Sym	Row	Softmax	None
Roman-Empire Direct (Heterophilic)	Attention	1	80.7±0.4	80.3±0.6	78.7±0.3	80.3±0.4	79.4±0.5
	Attention	8	82.1±0.3	83.0±0.7	80.4±0.6	82.2±0.4	83.4±0.4
	Uniform	1	80.1±0.5	80.1±0.5	78.9±0.5	78.9±0.5	77.6±0.4
	Random	1	76.9±0.5	76.5±0.4	76.1±0.3	68.3±0.5	14.2±0.3
	Random	8	76.8±0.4	75.7±0.8	76.0±0.4	69.7±0.5	14.3±0.6
Chameleon Direct (Heterophilic)	Attention	1	58.3±1.2	59.3±1.7	62.0±2.1	63.7±1.3	27.6±2.6
	Attention	8	55.5±2.2	58.3±1.8	59.7±1.3	58.7±1.6	37.8±3.4
	Uniform	1	60.0±1.6	60.0±1.6	63.9±1.9	63.9±1.9	26.8±2.6
	Random	1	59.8±2.0	59.3±1.3	61.3±2.1	43.3±2.4	26.6±1.8
	Random	8	58.6±2.2	59.6±3.0	62.1±1.6	47.1±2.6	34.0±2.4
Squirrel Direct (Heterophilic)	Attention	1	37.5±1.8	38.5±0.9	39.5±1.5	40.6±1.7	28.0±3.2
	Attention	8	38.0±1.9	39.3±1.6	41.0±1.8	39.2±1.7	25.8±1.7
	Uniform	1	40.3±1.3	40.3±1.3	43.3±1.5	43.6±1.4	27.9±2.6
	Random	1	40.4±1.4	39.9±1.5	42.7±1.1	34.5±1.2	27.4±1.7
	Random	8	40.4±1.4	39.9±1.5	42.7±1.1	34.3±1.6	27.4±1.7

Table A7. Accuracy (mean ± standard deviation) of DiGib [25] variants with different weight settings and normalization schemes evaluated on heterophilic datasets. yellow cells correspond to the best attention-weighted configuration. Results in bold exceed the baseline mean, while underlined results exhibit strong distributional overlap with the baseline.

Datasets	Weight	Heads	Dir	Sym	Row	Softmax	None
Roman-Empire Direct (Heterophilic)	Attention	1	80.7±0.4	80.3±0.6	78.7±0.3	80.3±0.4	79.4±0.5
	Attention	8	82.1±0.3	83.0±0.7	80.4±0.6	82.2±0.4	83.4±0.4
	Uniform	1	80.1±0.5	80.1±0.5	78.9±0.5	78.9±0.5	77.6±0.4
	Random	1	76.9±0.5	76.5±0.4	76.1±0.3	68.3±0.5	14.2±0.3
	Random	8	76.8±0.4	75.7±0.8	76.0±0.4	69.7±0.5	14.3±0.6
Chameleon Direct (Heterophilic)	Attention	1	58.3±1.2	59.3±1.7	62.0±2.1	63.7±1.3	27.6±2.6
	Attention	8	55.5±2.2	58.3±1.8	59.7±1.3	58.7±1.6	37.8±3.4
	Uniform	1	60.0±1.6	60.0±1.6	63.9±1.9	63.9±1.9	26.8±2.6
	Random	1	59.8±2.0	59.3±1.3	61.3±2.1	43.3±2.4	26.6±1.8
	Random	8	58.6±2.2	59.6±3.0	62.1±1.6	47.1±2.6	34.0±2.4
Squirrel Direct (Heterophilic)	Attention	1	37.5±1.8	38.5±0.9	39.5±1.5	40.6±1.7	28.0±3.2
	Attention	8	38.0±1.9	39.3±1.6	41.0±1.8	39.2±1.7	25.8±1.7
	Uniform	1	40.3±1.3	40.3±1.3	43.3±1.5	43.6±1.4	27.9±2.6
	Random	1	40.4±1.4	39.9±1.5	42.7±1.1	34.5±1.2	27.4±1.7
	Random	8	40.4±1.4	39.9±1.5	42.7±1.1	34.3±1.6	27.4±1.7

Table A8. Accuracy (mean±standard deviation) of LargeScaleNet [23] variants under different weight settings and normalization schemes on heterophilic datasets. Hyperparameters are taken from the LargeScaleNet paper’s best-performing configuration, but only the first scale is used (no higher-order scales). yellow cells correspond to the best attention-weighted configuration. Results in bold exceed the baseline mean, while underlined results exhibit strong distributional overlap with the baseline.

Datasets	Weight	Dir	Sym	Row	None
Roman-Empire	Attention	93.47±0.24	93.01±0.54	92.60±0.60	93.60±0.32
	Uniform	93.35±0.55	93.25±0.34	92.27±0.29	93.59±0.29
Arxiv-year	Attention	65.56±0.12	64.43±0.31	59.74±0.70	65.18±0.26
	Uniform	66.02±0.28	63.48±0.51	41.52±3.00	56.46±2.44
Chameleon	Attention	74.74±0.94	77.11±1.71	78.07±1.00	66.01±2.67
	Uniform	79.21±1.01	79.63±0.92	79.50±1.16	58.68±5.05
Squirrel	Attention	69.33±2.09	69.76±1.62	72.88±2.01	63.53±2.65
	Uniform	74.63±1.60	75.05±1.98	73.38±1.74	74.53±2.10

Table A8. Accuracy (mean±standard deviation) of LargeScaleNet [23] variants under different weight settings and normalization schemes on heterophilic datasets. Hyperparameters are taken from the LargeScaleNet paper’s best-performing configuration, but only the first scale is used (no higher-order scales). yellow cells correspond to the best attention-weighted configuration. Results in bold exceed the baseline mean, while underlined results exhibit strong distributional overlap with the baseline.

Datasets	Weight	Dir	Sym	Row	None
Roman-Empire	Attention	93.47±0.24	93.01±0.54	92.60±0.60	93.60±0.32
	Uniform	93.35±0.55	93.25±0.34	92.27±0.29	93.59±0.29
Arxiv-year	Attention	65.56±0.12	64.43±0.31	59.74±0.70	65.18±0.26
	Uniform	66.02±0.28	63.48±0.51	41.52±3.00	56.46±2.44
Chameleon	Attention	74.74±0.94	77.11±1.71	78.07±1.00	66.01±2.67
	Uniform	79.21±1.01	79.63±0.92	79.50±1.16	58.68±5.05
Squirrel	Attention	69.33±2.09	69.76±1.62	72.88±2.01	63.53±2.65
	Uniform	74.63±1.60	75.05±1.98	73.38±1.74	74.53±2.10