Syntactic Dependency-Aware Neural Networks for Enhanced Entity Relation Analysis

1. Introduction

Identifying relations among textual entities is crucial for numerous language processing applications, such as biomedical data mining (Quirk and Poon 2017), database enrichment (Zhang et al. 2017), and interactive query systems (Yu et al. 2017). Consider, for instance, a scenario detailing a relationship among the entities L858E, EGFR, and gefitinib across two sentences.

Figure 1. An example for entity relation extraction.

Figure 1. An example for entity relation extraction.

Relation extraction techniques primarily divide into two categories: sequence-oriented and dependency-focused. The former relies solely on word sequences (Zeng et al. 2014; Wang et al. 2016), while the latter incorporates syntactic dependency trees (Bunescu and Mooney 2005; Peng et al. 2017). Dependency-focused methods surpass their sequence-oriented counterparts by capturing syntactic relations that are not immediately evident. Various pruning techniques have been suggested to refine dependency data, thereby enhancing performance. For instance, Xu et al. (2015); Fei et al. (2022); Xu et al. (2015) employ neural networks on the shortest dependency paths, while Miwa and Bansal (2016) concentrate on subtrees linked to the lowest common ancestor (LCA) of the entities. Zhang et al. (2018) apply graph convolutional networks (GCNs) (Kipf and Welling 2017) to pruned trees, including nodes within a certain distance from the dependency path in the LCA subtree.

However, these rule-based pruning approaches risk losing crucial data from the complete tree. For example, critical tokens like partial response might be overlooked if only pruned trees are considered. Ideally, models should adeptly include or exclude data from the full tree (Tang et al. 2016; Li et al. 2022; Fei et al. 2021; Dong et al. 2014; Fei et al. 2020). Our paper introduces the SDANNs, functioning directly on the full tree and employing a ’soft pruning’ approach. This method transforms the dependency tree into a fully-connected, edge-weighted graph, with self-attention mechanisms (Vaswani et al. 2017) determining the relevance of connections.

To effectively process these dense graphs, we integrate dense connections into the GCN model as per (Huang et al. 2017; Guo et al. 2019). Standard GCNs require multiple layers to capture multi-hop neighborhood information, yet deeper models often do not yield proportional benefits. With dense connections, our SDANN model can be deeply layered, effectively capturing both local and non-local dependency information.

Our experiments affirm the SDANN’s enhanced performance in various scenarios. In multi-sentence relation extraction, our model excels, surpassing current leading models in both ternary and binary relation accuracy by significant margins. On the comprehensive sentence-level TACRED dataset, the SDANN consistently outperforms existing models. In summary, our proposed SDANNs introduce an end-to-end ’soft pruning’ strategy for better graph representation learning, achieving state-of-the-art results without increased computational demands. The adjacency matrix size remains constant compared to the original tree, ensuring efficiency in parallel processing over dependency trees, unlike tree-structured models like Tree-LSTM (Tai et al. 2015).

2. Related Work

The foundation of our research is rooted in the evolving landscape of relation extraction models and the advancements in graph convolutional networks. In the early stages, relation extraction research predominantly hinged on statistical methodologies. Pioneers in this field explored tree-based (Zelenko et al. 2002) and dependency path-based kernels (Bunescu and Mooney 2005) to delve into the intricacies of relation extraction. Groundbreaking work by McDonald et al. (2005) involved constructing maximal cliques of entities, providing a novel perspective on predicting relations. Adding another dimension to this approach, Mintz et al. (2009) amalgamated syntactic features into statistical classifiers, enriching the analytical process.

With the advent of neural networks, the landscape shifted towards sequence-based models. These models harnessed the power of various neural network architectures for relation extraction. Convolutional neural networks (CNNs) were employed in groundbreaking studies (Zeng et al. 2014; Wang et al. 2016; Nguyen and Grishman 2015), while recurrent neural networks (RNNs) found their use in attention-based learning (Zhang et al. 2017; Zhou et al. 2016; Fei et al. 2023). The ingenuity of combining CNNs and RNNs was demonstrated by Vu et al. (2016), and the emergence of transformers (Verga et al. 2018) marked a significant evolution in this domain.

Dependency-based models also evolved, integrating structural information into neural frameworks. Peng et al. (2017) innovated by splitting the dependency graph into two Directed Acyclic Graphs (DAGs) and extending the tree LSTM model (Tai et al. 2015) for complex n-ary relation extraction. Aligning closely with our work, Song et al. (2018a) employed graph recurrent networks (Song et al. 2018b) to encode entire dependency graphs. This approach contrasts with ours similarly to the differences between CNNs and RNNs. Various pruning strategies were also proposed to distill dependency information for enhanced model performance. Techniques ranged from encoding the shortest dependency path (Xu et al. 2015; Fei et al. 2023), applying LSTM models to the LCA subtree of entities (Miwa and Bansal 2016; Fei et al. 2020), combining dependency paths and subtrees (Liu et al. 2015), to adopting path-centric pruning strategies (Zhang et al. 2018). Our model diverges from these by learning to variably weigh each edge in an end-to-end manner, rather than removing edges in preprocessing.

The initial steps to adapt neural networks for arbitrary structured graphs were made by Gori et al. (2005); Bruna (2014). These foundational efforts paved the way for more computationally efficient adaptations through local spectral convolution techniques (Henaff et al. 2015; Defferrard et al. 2016). Our methodology closely aligns with the GCNs (Kipf and Welling 2017; Fei et al. 2020), which focus on filters operating within a first-order neighborhood around each node.

In a more recent and innovative stride, Velickovic et al. (2018) introduced graph attention networks (GATs), leveraging masked self-attentional layers (Vaswani et al. 2017) to effectively summarize neighborhood states. While sharing some conceptual similarities with our work, their motivations and network structures exhibit significant differences. In GATs, each node’s focus is confined to its immediate neighbors, whereas our AGGCNs assess the relatedness across all nodes. This distinct approach allows for the construction of fully connected graphs in AGGCNs, enabling the capture of long-range semantic interactions, a feature not present in the static network topology of GATs.

3. Methodology

This section delineates the core elements constituting our Syntactic Dependency Graph Convolutional Network (SDANN) model.

3.1. Graph-Based Neural Networks

Graph-based Neural Networks (GNNs), a variant of neural networks tailored for graph data, operate directly on graphs (Kipf and Welling 2017). We mathematically demonstrate how GNNs function on a graph with n nodes, represented by an $n\times n$ adjacency matrix $\mathbf{A}$. Marcheggiani and Titov (2017) augmented GNNs to process dependency trees by integrating edge directionality. They introduce self-loops for each node, and the adjacency matrix reflects both directions of a dependency arc, i.e., ${\mathbf{A}}_{ij}=1$ and ${\mathbf{A}}_{ji}=1$ if there’s an edge from node i to j, and zero otherwise. The convolution operation for node i at layer l, with input feature ${\mathbf{h}}^{(l-1)}$ and output ${\mathbf{h}}_i^{\left(l\right)}$, is defined as:

$${\mathbf{h}}_i^{\left(l\right)}=\rho \left(\sum _{j=1}^n{\mathbf{A}}_{ij}{\mathbf{W}}^{\left(l\right)}{\mathbf{h}}_j^{(l-1)}+{\mathbf{b}}^{\left(l\right)}\right)$$

(1)

where ${\mathbf{W}}^{\left(l\right)}$ and ${\mathbf{b}}^{\left(l\right)}$ denote the weight matrix and bias vector, respectively, and $\rho$ is an activation function such as RELU. The initial input ${\mathbf{h}}_i^{\left(0\right)}$ is ${\mathbf{x}}_i$, with ${\mathbf{x}}_i\in {\mathbb{R}}^d$ and d being the input feature dimension.

3.2. Syntactic Attention Layer

The SDANN comprises M identical blocks, each containing three layers: the syntactic attention layer, densely connected layer, and linear combination layer. The syntactic attention layer is the innovative part of SDANN.

As discussed in Section 1, typical pruning methods are preset and transform the full tree into a subtree, forming the basis of the adjacency matrix. This can be likened to hard attention (Xu et al. 2015), where non-subtree edges are neglected. Our model employs a ’soft pruning’ method in the syntactic attention layer, assigning weights to all edges, learned end-to-end.

In this layer, the original tree is converted into a fully connected edge-weighted graph by constructing a syntactic attention adjacency matrix $\tilde{\mathbf{A}}$, where each entry ${\tilde{\mathbf{A}}}_{ij}$ represents the weight from node i to node j. Figure ?? illustrates this with ${\tilde{\mathbf{A}}}^{\left(\mathbf{1}\right)}$ corresponding to graph $G^{\left(1\right)}$. We use a self-attention mechanism (Cheng et al. 2016), capturing interactions within a sequence, to construct $\tilde{\mathbf{A}}$. For computational layers, $\tilde{\mathbf{A}}$ replaces the original $\mathbf{A}$, without additional computational cost. This layer leverages attention to infer node relations, particularly for indirect paths, enabling the model to capture nuanced connections through differentiable functions.

Multi-head attention (Vaswani et al. 2017) is used for calculating $\tilde{\mathbf{A}}$, enabling simultaneous attention to different representation subspaces. The calculation involves a query and key-value pairs, with output as a weighted sum of values, based on the query and corresponding key.

$${\tilde{\mathbf{A}}}^{\left(\mathbf{t}\right)}=softmax\left(\frac{Q{\mathbf{W}}_i^Q\times {\left(K{\mathbf{W}}_i^K\right)}^T}{\sqrt{d}}\right)$$

(2)

Here, Q and K are equal to ${\mathbf{h}}^{(l-1)}$, the collective representation at layer $l-1$. The projections are parameter matrices ${\mathbf{W}}_i^Q,{\mathbf{W}}_i^K\in {\mathbb{R}}^{d\times d}$.

${\tilde{\mathbf{A}}}^{\left(\mathbf{t}\right)}$ is the t-th syntactic attention adjacency matrix corresponding to the t-th head, with up to N matrices constructed, where N is a hyper-parameter.

3.3. Linear Layer

In contrast to traditional pruning, resulting in smaller structures, our syntactic attention layer outputs a larger fully connected graph. Adapting from (Guo et al. 2019), we integrate dense connections (Huang et al. 2017) into SDANN to encapsulate more structural information on large graphs. These connections enable a deeper model, capturing rich local and non-local information for an improved graph representation.

Dense connectivity introduces direct connections from any layer to all preceding layers. We define ${\mathbf{g}}_j^{\left(l\right)}$ as the concatenation of the initial node representation and the representations from layers 1 to $l-1$:

$${\mathbf{g}}_j^{\left(l\right)}=[{\mathbf{x}}_j;{\mathbf{h}}_j^{\left(1\right)};...;{\mathbf{h}}_j^{(l-1)}].$$

(3)

Each densely connected layer has L sub-layers, with dimensions $d_{hidden}=d/L$. For instance, a layer with 3 sub-layers and 300 input dimensions will have $d_{hidden}=100$, and the output dimension remains 300 (3 × 100). This structure, akin to DenseNets (Huang et al. 2017), improves parameter efficiency.

Given N different syntactic attention adjacency matrices, N separate densely connected layers are needed. The computation for each layer, corresponding to the t-th matrix ${\tilde{\mathbf{A}}}^{\left(\mathbf{t}\right)}$, is modified as follows:

$${\mathbf{h}}_{t_i}^{\left(l\right)}=\rho \left(\sum _{j=1}^n{\tilde{\mathbf{A}}}_{ij}^{\left(t\right)}{\mathbf{W}}_t^{\left(l\right)}{\mathbf{g}}_j^{\left(l\right)}+{\mathbf{b}}_t^{\left(l\right)}\right)$$

(4)

where $t=1,...,N$, selecting the appropriate weight matrix and bias term for each ${\tilde{\mathbf{A}}}^{\left(\mathbf{t}\right)}$. The column dimension of ${\mathbf{W}}_t^{\left(l\right)}$ increases by $d_{hidden}$ per sub-layer.

3.4. Linear Combination Layer

SDANN includes a linear combination layer to integrate representations from the N densely connected layers. The output is defined as:

$${\mathbf{h}}_{comb}={\mathbf{W}}_{comb}{\mathbf{h}}_{out}+{\mathbf{b}}_{comb}$$

(5)

where ${\mathbf{h}}_{out}$ is the concatenation of outputs from N layers, and ${\mathbf{W}}_{comb}\in {\mathbb{R}}^{(d\times N)\times d}$ and ${\mathbf{b}}_{comb}$ are the weight matrix and bias vector for the linear transformation.

3.5. SDANNs for Relation Extraction

Applying SDANN over the dependency tree yields hidden representations for all tokens. The goal of relation extraction is to predict relations among entities using these representations. Following (Zhang et al. 2018), we concatenate sentence and entity representations for the final classification. Sentence representation $h_{sent}$ is obtained by:

$$h_{sent}=f\left({\mathbf{h}}_{\mathbf{mask}}\right)=f\left(\mathrm{SDANN}\left(\mathbf{x}\right)\right)$$

(6)

where ${\mathbf{h}}_{\mathbf{mask}}$ represents selected non-entity token representations, and f is a max pooling function mapping n output vectors to one sentence vector. Entity representations are similarly obtained and concatenated with $h_{sent}$. A feed-forward neural network (FFNN) processes these representations for final prediction:

$$h_{final}=\mathrm{FFNN}\left([h_{sent};h_{e_1};...;h_{e_i}]\right)$$

(7)

$h_{final}$ is then used in a logistic regression classifier for prediction.

4. Experiments

4.1. Experimental Framework

We assess the SDANN model’s capabilities in two key areas: cross-sentence n-ary relation extraction and sentence-level relation extraction.

In cross-sentence n-ary relation extraction, we employ the dataset from (Peng et al. 2017), featuring 6,987 ternary and 6,087 binary relation instances from PubMed. Instances span multiple sentences and are categorized into five labels: resistance or nonresponse”, sensitivity”, response”, resistance”, and None”. We bifurcate our analysis into binary-class n-ary relation extraction and multi-class n-ary relation extraction, with the former consolidating four relation types into Yes” and assigning None” as No”, following (Peng et al. 2017).

For sentence-level relation extraction, the TACRED dataset  (Zhang et al. 2017) and Semeval-10 Task 8  (Hendrickx et al. 2010) are utilized as per the methodologies in  (Zhang et al. 2018). TACRED, with over 106K instances, enumerates 41 relation types plus a no relation” category, while Semeval-10 Task 8 offers 10,717 instances across 9 relations plus an other” category.

Hyper-parameter optimization is guided by development set outcomes. For cross-sentence tasks, we use the data split from (Song et al. 2018a), and for sentence-level tasks, we follow the development set guidelines from (Zhang et al. 2018). The number of attention heads N, block numbers M, and sub-layer counts L in densely connected layers are selected from predefined sets. Optimal combinations are identified as (N=2, M=2, $L_1$=2, $L_2$=4, $d_{hidden}$=340) for cross-sentence tasks and (N=3, M=2, $L_1$=2, $L_2$=4, $d_{hidden}$=300) for sentence-level tasks. GloVe vectors (Pennington et al. 2014) serve as initial word embeddings.

Model evaluations align with the metrics used in prior studies (Zhang et al. 2018; Song et al. 2018a). For cross-sentence tasks, test accuracy is reported as an average over five validation folds (Song et al. 2018a). In sentence-level tasks, we report micro-averaged F1 scores for TACRED and macro-averaged F1 scores for SemEval.

4.2. Results on Cross-Sentence n-ary Relation Extraction

In evaluating the SDANN model for cross-sentence n-ary relation extraction, we benchmark against three categories of models: 1) Feature-based classifiers (Quirk and Poon 2017) utilizing shortest dependency paths, 2) Graph-structured LSTM frameworks like Graph LSTM (Peng et al. 2017), Bidirectional DAG LSTM (Bidir DAG LSTM) (Song et al. 2018a), and Graph State LSTM (GS GLSTM) (Song et al. 2018a), which encode graphs from sentences with dependency edges, and 3) Graph Convolutional Networks (GCN) with pruned trees(Zhang et al. 2018). Additionally, the tree-structured LSTM method (SPTree)(Miwa and Bansal 2016) is included for binary relation extraction of drug-mutations, as per(Song et al. 2018a). Results are presented in Table 1.

Focusing on binary-class n-ary relation extraction, the SDANN model demonstrates superior performance, achieving 87.1% and 87.0% accuracies for single sentence ($\mathtt{Single}$) and all instances ($\mathtt{Cross}$) settings in ternary relation extraction. This outperforms all baselines, including surpassing the GS GLSTM model by significant margins. In binary relation extraction, SDANN continues to excel, consistently outshining both GS GLSTM and GCN models.

The SDANN model’s effectiveness is attributed to its advanced graph convolution capabilities, which surpass traditional full tree-based methods like GS GLSTM. This improvement is likely due to the synergy between the densely connected layers and attention guided layers in SDANN, which enhance long-distance dependency learning without pruning, and refine the information extraction process.

For the multi-class classification task, our SDANN model maintains its lead, outperforming GS GLSTM and all GCN models. This further underscores SDANN’s proficiency in handling complex relation extraction tasks.

4.3. Results on Sentence-level Relation Extraction

Turning to the TACRED dataset for sentence-level relation extraction, we compare SDANN against both dependency-based and sequence-based models. Dependency-based contenders include logistic regression classifiers (LR) (Zhang et al. 2017), Shortest Path LSTM (SDP-LSTM)(Xu et al. 2015), Tree-LSTM (Tai et al. 2015), and GCN variants (Zhang et al. 2018). The sequence-based category is represented by Position Aware LSTM (PA-LSTM)(Zhang et al. 2017). The results are detailed in Table 2.

The SDANN model showcases its efficacy by surpassing the GCN in F1 score. The integration of contextualized information via a bi-directional LSTM network, resulting in the Contextualized SDANN (C-SDANN), further elevates its performance, outdoing the C-GCN model. This enhancement signifies the importance of contextual data in relation extraction tasks.

4.4. Results on Sentence-level Relation Extraction

In the realm of sentence-level relation extraction, our SDANN model was scrutinized against two distinct categories of models on the TACRED dataset. The first category included dependency-based models such as the Logistic Regression Classifier (LR) (Zhang et al. 2017), Shortest Path LSTM (SDP-LSTM)(Xu et al. 2015), Tree-structured LSTM (Tree-LSTM)(Tai et al. 2015), and Graph Convolutional Networks (GCN) including Contextualized GCN (C-GCN) (Zhang et al. 2018). The second category was represented by sequence-based models, notably the Position Aware LSTM (PA-LSTM)(Zhang et al. 2017). The results of this comparative analysis are detailed in Table 2.

The SDANN model, in its C-SDANN form (an extension with a bi-directional LSTM for contextual representation), demonstrated significant efficacy, achieving a noteworthy F1 score of 69.0, surpassing the state-of-the-art C-GCN model. This performance indicates the effectiveness of integrating contextual information in the SDANN framework for nuanced relation extraction tasks.

Further evaluation on the SemEval dataset, under the same experimental conditions as (Zhang et al. 2018), revealed consistent superiority of our C-SDANN model. Despite the smaller size of the SemEval dataset compared to TACRED, C-SDANN achieved an F1 score of 85.7, clearly outperforming the C-GCN model. These results, presented in Table 3, underscore the model’s robust generalizability across different dataset scales and complexities.

4.5. Further Results

Ablation Study.

In our ablation study of the SDANN model, specifically its contextualized variant, we explored the impact of removing key components like attention-guided and densely connected layers. The results, presented in Table 4, illustrate that both types of layers significantly contribute to the model’s performance. The attention-guided layer, in particular, shows a more pronounced effect, indicating its crucial role in enhancing the model’s capabilities.

Performance with Pruned Trees.

The effectiveness of SDANN under varying levels of tree pruning is detailed in Table 5. Notably, SDANN achieves its best performance with the full tree structure, surpassing the versions with pruned trees. This underscores the advantage of the model’s ability to utilize comprehensive syntactic information, as opposed to relying on pruning strategies.

Performance against Sentence Length.

Our analysis across different sentence lengths demonstrated that the full tree SDANN model consistently outperforms its pruned counterparts and the C-GCN model. The superiority of the full tree structure becomes more evident as sentence length increases, highlighting the model’s proficiency in handling more complex dependency structures.

Performance against Training Data Size.

Under various training data size scenarios, the SDANN model consistently surpassed the C-GCN model, even with lesser training data. This efficiency becomes more apparent with increasing amounts of data, showcasing SDANN’s effectiveness in resource utilization and learning from extensive data sets.

5. Conclusion and Future Directions

In this paper, we present a groundbreaking model, Syntactic Dependency-Aware Neural Networks (SDANN), which has demonstrated remarkable success in diverse relation extraction tasks, surpassing previous state-of-the-art models. The SDANN framework is unique in its methodology; it processes entire syntactic dependency trees, effectively sifting through and harnessing valuable information in an end-to-end manner. This contrasts sharply with prior methods that often relied on partial tree structures. Looking ahead, there are several promising directions for further exploration with SDANN. A particularly intriguing avenue is to explore how this innovative framework can be harnessed to enhance graph representation learning in various graph-related tasks, as discussed in (Bastings et al. 2017). The potential of SDANN in such applications is vast, given its ability to capture complex relational data from full syntactic structures, which opens up new possibilities in graph-based learning paradigms.