Addressing Long-Distance Dependencies in AMR Parsing with Hierarchical Clause Annotation

1. Introduction

Most natural language processing (NLP) tasks operate an input sentence as a word sequence with token-level embeddings and features, suffering long-distance dependencies (LDDs) when encountering long complex sentences such as dependency parsing [1], constituency parsing [2], semantic role labeling (SRL) [3], machine translation [4], discourse parsing [5] and text summarization [6]. In previous works, the length of a sentence is blamed for LDDs superficially, and several universal methods are proposed to cure this issue, e.g., hierarchical recurrent neural networks [7], long short-term memory (LSTM) [8], attention mechanism [9], Transformer [10], implicit graph neural networks [11], and such.

For example, abstract meaning representation (AMR) parsing [12], translating a sentence to a directed acyclic semantic graph with relations among abstract concepts, has witnessed some efforts for LDDs in different approaches. In transition-based strategies, Peng et al. [13] propose a cache system to predict arcs between long-distant words. In graph-based methods, Cai and Lam [14] present a graph↔sequence iterative inference to overcome inherent defects of the one-pass prediction process in parsing long sentences. In seq2seq-based approaches, Bevilacqua et al. [15] employ the Transformers-based pre-trained language model, BART [16], to address LDDs in long sentences. Among these categories, seq2seq-based approaches become mainstream and recent parsers [17,18,19,20] employ the seq2seq architecture with the popular codebase, SPRING [15], achieving better performances. Notably, HGAN [20] integrates token-level features, syntactic dependencies (SDP), and SRL with heterogeneous graph neural networks and becomes the state-of-the-art (SOTA) in the settings of removing extra silver training data, graph-categorization and ensemble methods.

However, these AMR parsers still suffer performance degradation when encountering long sentences with deeper AMR graphs [18,20] that introduces the most LDD cases. We argue that the complexity of the clausal structure inside a sentence is the essence of LDDs, where clauses are core units of grammar and center on a verb that determines the occurrences of other constituents [21]. Our intuition is that non-verb words in a clause typically can not depend on words outside, while dependencies between verbs correspond to the inter-clause relations, resulting in LDDs across clauses [22].

To prove our claim, we demonstrate the AMR graph of a sentence from the AMR 2.0 dataset 1 and distinguish the AMR relation distances in terms of different segment levels (clause/phrase/token) in Figure 1. Every AMR relation is represented as a dependent edge between two abstract AMR nodes that align to one or more input tokens. The dependency distances of inter-token relations are subtractive results from the indices of tokens aligned to the source and target nodes, while those of inter-phrase and inter-clause relations are calculated by indices of the headwords in phrases and the verbs 2 in clauses, respectively. As can be observed:

• Dependency distances of inter-clause relations are typically much longer than those of inter-phrase and inter-token relations, leading to the most LDD cases. E.g., the AMR relation, $anxious\stackrel{:ARG0-of}{\to }go-01$, occurring in the clause “I get very anxious” and its relative clause “which does sort of go away ...”, has a dependency distance of 6 (subtracting the 9^th token “anxious” from the 15^th token “go”).
• Reentrant AMR nodes abstracted from pronouns also bring the long distant AMR relations. E.g., the AMR relation, $wait-01\stackrel{:ARG0}{\to }I$, has a dependency distance of 33 (subtracting the 1^st token “I” from the 34^th token “wait”).

Based on the findings above, we are inspired to utilize the clausal features of a sentence to cure LDDs. Rhetorical structure theory (RST) [23] provides a general way to describe the coherence relations among clauses and some phrases, i.e., elementary discourse units, and postulates a hierarchical discourse structure called discourse tree. Except for RST, a novel clausal feature, hierarchical clause annotation (HCA) [24], also captures a tree structure of a complex sentence, where the leaves are segmented clauses, and the edges are the inter-clause relations.

Due to the better parsing performances of the clausal structure [24], we select and integrate the HCA trees of complex sentences to cure LDDs in AMR parsing. Specifically, we propose two HCA-based approaches, HCA-based self-attention (HCA-SA) and HCA-based curriculum learning (HCA-CL), to integrate the HCA trees as clausal features in the popular AMR parsing codebase, SPRING [15]. In HCA-SA, we convert an HCA tree into a clause adjacency matrix and a token visibility matrix to restrict the attention scores between tokens from unrelated clauses and increase those from related clauses in masked-self-attention encoder layers. In HCA-CL, we employ curriculum learning with two training curricula, Clause-Number and Tree-Depth, with the intuition that “the more number of clauses or the deeper clausal tree in a sentence, the more difficult it is to learn”.

We conduct extensive experiments on two in-distribution (ID) AMR datasets (i.e., AMR 2.0 and AMR 3.0 3) and three out-of-distribution (OOD) ones (i.e., TLP, New3, and Bio) to evaluate our two HCA-based approaches. In ID datasets, our parser achieves 0.7 Smatch F1 score improvements against the baseline model, SPRING, on both AMR 2.0 and AMR 3.0., and outperforms the SOTA parser, HGAN, by 0.5 and 0.6 F1 scores of the fine-grained metric SRL in the two datasets. Notably, as the clause number of the sentence increases, our parser outperforms SPRING by a large margin and achieves better Smatch F1 scores than HGAN, indicating the ability to cure LDDs. In OOD datasets, the performance boosts achieved by our HCA-based approaches are more evident on complicated corpora like New3 and Bio, where sentences consist of more clauses and longer clauses. Our code is publicly available at https://github.com/MetroVancloud/HCA-AMRparsing (accessed on 3 August 2023).

The rest of this paper is organized as follows: The related works are summarized in Section 2, and the proposed approaches are detailed in Section 3. Then, the experiments of AMR parsing are presented in Section 4. Next, the discussion of the experimental results is presented in Section 5. Finally, our work is concluded in Section 6.

2. Related Work

In this section, we first introduce the open problem LDDs and some universal methods. Then, we summarize the four main categories of parsers and the LDD cases in AMR parsing. Finally, we introduce the novel clausal feature, HCA.

2.1. Long-Distance Dependencies

LDDs, first proposed by Hockett [25], describe an interaction between two (or more) elements in a sequence separated by an arbitrary number of positions. LDDs are related to the rate of decay of statistical dependence of two points with increasing time intervals or spatial distances between them [26]. In recent linguistic research of LDDs, Liu et al. [22] propose the verb-frame frequency account to robustly predict acceptability ratings in sentences with LDDs, indicating the affinities between the number of verbs and LDDs in sentences.

In recent advances in NLP tasks, hierarchical recurrent neural networks [7], LSTM [8], attention mechanism [9], Transformer [10], implicit graph neural networks [11] are proposed to cure LDDs. These universal neural models all represent the input sequence with token-level embeddings from pretrained language models in most NLP tasks.

2.2. AMR Parsing

AMR parsing is a challenging semantic parsing task since AMR is a deep semantic representation consisting of many special annotations (e.g., abstract concept nodes, named entities, co-reference, and such.) [12]. The aim of AMR parsing is translating a sentence to a directed acyclic semantic graph with relations among abstract conceptsPrevious methods, where two main characteristics are:

• Abstraction: assigns the same AMR to sentences with the same basic meaning and also brings a challenge of alignments between input tokens and output AMR nodes [12], e.g., the token “can” and its corresponding AMR node $possible-01$ in Figure 1.
• Reentrancy: introduces the presence of nodes with multiple parents and represent sentences as graphs rather than trees [27], causing some LDD cases, e.g., the AMR node “I” in Figure 1.

Existing AMR parsers can be summarized into four categories:

• Graph-based: Directly predict nodes and edges in either two-stage procedures [28,29,30] or incremental one-stage [14,31] procedures. The SOTA graph-based model, AMR-gs [14], enhances the incremental graph construction with an AMR graph↔sequence (AMR-gs) iterative inference mechanism in one-stage procedures.
• Seq2seq-based: Model the task as transduction of the sentence into a linearization of the AMR graph [15,17,18,19,20,32,33,34,35]. SPRING [15] is a popular seq2seq-based codebase that employs transfer learning by exploiting pretrained encoder-decoder model, BART [16], to generate a linearized graph incrementally with a single auto-regressive pass of a seq2seq decoder. The subsequent models, ANCES [17], HCL [18], ATP [19], and HGAN [20] all follow the architecture of SPRING, where HGAN integrates SDP and SRL features with heterogeneous graph neural networks and achieves the best performances in the settings of removing extra silver training data, graph re-categorization, and ensemble methods.
• Transition-based: Predict a sequence of actions that generate the graph while processing tokens left-to-right through the sentence [36,37,38,39,40,41,42]. The SOTA transition-based model, StructBART [42], explores the integration of general pre-trained sequence-to-sequence language models and a structure-aware transition-based approach.
• Grammar-based: Peng et al. [43] introduce a synchronous hyperedge replacement grammar solution. Pust et al. [44] regard the task as a machine translation problem, while Artzi et al. [45] adapt combinatory categorical grammar. Groschwitz et al. [46] and Lindemann et al. [47] view AMR graphs as the structural AM algebra.

Despite different architectures, most AMR parsers employ word embeddings from pretrained language models and utilize token-level features like part-of-speech (POS), SDP, and SRL. However, these parsers still suffer performance degradation [18,20] when parsing complex sentences with LDDs due to the difficulties of aligning input tokens with output AMR nodes in such a long sequence.

2.3. Hierarchical Clause Annotation

RST [23] provides a general way to describe the coherence relations among parts in a text and postulates a hierarchical discourse structure called a discourse tree. The leaves of a discourse tree can be a clause or a phrase without strict definitions, known as elementary discourse units. However, the performances of RST parsing tasks are unsatisfactory due to the loose definitions of elementary discourse units and abundant types of discourse relations[48,49].

Except for RST, Fan et al. [24] also propose a novel clausal feature, HCA, which represents a complex sentence as a tree consisting of clause nodes and inter-clause relation edges. The HCA framework is based on the English grammar [21], where clauses are elementary grammar units that center around a verb, and inter-clause relations can be classified into two categories:

• Coordination: is an equal relation shared by clauses with the same syntactic status, including And, Or, and But relations
• Subordination: occur in a matrix and a subordinate clause, including Subjective, Objective, Predicative, Appositive, Relative, and nine sublevel Adverbial relations.

Inter-clause relations have appropriate alignments with AMR relations, where nominal clause relations correspond to the frame arguments (e.g., Subjective vs. :ARG0) and adverbial clause relations are mapped to general semantic relations (e.g., Adverbial_of_Condition vs. :conditon). Figure 2 demonstrates the segmented clauses and the HCA tree of the same exemplified sentence in Figure 1.

Based on well-defined clauses and inter-clause relations, Fan et al. provide a manual-annotated HCA corpus for AMR 2.0 and high-performance neural models to generate silver HCA trees for more complex sentences. Therefore, we select and utilize the HCA trees as clausal features to address LDDs in AMR parsing.

3. Model

In this paper, we propose two HCA-based approaches, HCA-SA (Section 3.1) and HCA-CL (Section 3.2), to integrate HCA trees in the popular AMR parser codebase, SPRING for addressing LDDs in AMR parsing.

3.1. HCA-based Self-Attention

Existing AMR parsers (e.g., SPRING) employ Transformer [10] as an encoder to obtain the input sentence representation. However, in the standard self-attention mechanism adopted by Transformer, every token needs to attend to all other tokens, and the learned attention matrix A is often very sparse across most data points [50]. Inspired by the work of Liu et al. [51], we propose the HCA-SA approach, which utilizes hierarchical clause annotations as a structural bias to restrict the attention scope and attention scores.

3.1.1. Token Visibility Matrix

For the exemplified sentence in Figure 2, its HCA tree can be transferred into a clause adjacency matrix in Figure 3a by checking if two clauses have an inter-clause edge (not meaning that they are adjacent in the source sentence), where adjacent clauses have specific correlation (pink) and non-adjacent ones share no semantics (white). Especially each clause has the strongest correlation with itself (red).

Furthermore, we transform the clause adjacency matrix into token visibility matrices by splitting every clause into tokens. As shown in Figure 3b,c, the visibilities between tokens can be summarized into the following cases:

• Full Visibility: tokens in the same clause are fully visible mutually
• Patial Visibility: tokens from two adjacent clauses $C_1$ and $C_2$ are sharing partial visibility
• None Visibility: tokens from non-adjacent clauses $C_2$ and $C_5$ are invisible to each other
• Global Visibility: tokens with a pronoun POS (e.g., “I” in $C_5$) are globally visible for linguistic phenomena of co-reference
• Additional Visibility: tokens that are clausal keywords (i.e., coordinators, subordinators, and antecedents) share additional visibilities with the tokens in adjacent clauses (e.g., “if” in $C_1$ to tokens in $C_2$)

Therefore, we introduce two token visibility matrices $M^{\mathrm{vis}}$ and $M^{\mathrm{deg}}$, where the former signals if two tokens are visible mutually, and the latter measures the visibility degree between them:

$$\begin{array}{cc}\hfill M_{i,j}^{\mathrm{vis}}& =\left\{\begin{array}{cc}-\infty \hfill & w_i\otimes w_j\hfill \\ 0,\hfill & otherwise\hfill \end{array}\right.\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill M_{i,j}^{\mathrm{deg}}& =\left\{\begin{array}{cc}0\hfill & w_i\otimes w_j\hfill \\ 1,\hfill & w_i\odot w_j\hfill \\ {\lambda }_0,\hfill & w_i\ominus w_j\hfill \\ {\lambda }_1,\hfill & w_i\ominus w_j\wedge Key(w_i/w_j)\hfill \\ \mu ,\hfill & PRN(w_i/w_j)\hfill \end{array}\right.\hfill \end{array}$$

(2)

where ⊗, ⊙, and ⊖ mean that the corresponding two tokens, $w_i$ and $w_j$, are positioned in two nonajacent (None Visibility), same (Full Visibility), and adjacent (Partial Visibility) clauses, respectively. $Key(w_i/w_j)$ indicates that at least one of $w_i$ and $w_j$ is a clausal keyword token, while $PRN(w_i/w_j)$ denotes the existences of at least one pronoun in $w_i$ and $w_j$. Values of the hyperparameters ${\lambda }_0$, ${\lambda }_1$ and $\mu$ are in $(0,1)$, and ${\lambda }_1>{\lambda }_0$.

3.1.2. Masked-Self-Attention

To some degree, the token visibility matrices $M^{\mathrm{vis}}$ and $M^{\mathrm{deg}}$ contain the structural information of the HCA tree. For vanilla Transformers employed in existing AMR parsers, stacked self-attention layers inside can not receive $M^{\mathrm{vis}}$ and $M^{\mathrm{deg}}$ as inputs, so we modify it into Masked-Self-Attention, which can restrict the attention scope and attention scores according to $M^{\mathrm{vis}}$ and $M^{\mathrm{deg}}$. Formally, the masked attention scores $S^{\mathrm{mask}}$ and the masked attention matrix $A^{\mathrm{mask}}$ are defined as:

$$\begin{array}{cc}\hfill S^{\mathrm{mask}}& =\mathrm{Softmax}(\frac{Q{K}^{\top }}{\sqrt{d}}·M^{\mathrm{deg}}+M^{\mathrm{vis}})\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill A^{\mathrm{mask}}& =S^{\mathrm{mask}}V\hfill \end{array}$$

(4)

where self-attention inputs $Q,K,V\in {\mathbb{R}}^{N\times d}$, N is the length of input sentences, and scaling factor d is the dimension of the model. Intuitively, if token $w_i$ is invisible to $w_j$, the attention scores $S_{i,j}^{\mathrm{mask}}$ will be masked to 0 due to the value $-\infty$ of $M_{i,j}^{\mathrm{vis}}$ and the value 0 of $M_{i,j}^{\mathrm{deg}}$. Otherwise, $S_{i,j}^{\mathrm{mask}}$ will be scaled according to $M_{i,j}^{\mathrm{deg}}$ in different cases.

3.1.3. Clause-Relation-Binded Attention Head

In every stacked self-attention layer of Transformer, multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions [10]. It also provides us the possibility of integrating different inter-clause relations in different attention heads. Instead of masking the attention matrix with non-labeled HCA trees, we propose a clause-relation-binded attention head setting, where every head attends more to a specific inter-clause relation.

In this setting, we increase the visibilities between tokens in two adjacent clauses with interrelation ${\mathrm{rel}}_i$ in the attention matrix ${\mathrm{A}}_{{\mathrm{rel}}_i}^{\mathrm{mask}}$ of the binded head, i.e., increasing ${\lambda }_0$ to 1 in Equation (1). Therefore, the final attention matrix $A^{\mathrm{MultiHead}}$ of each stacked self-attention layer is defined as:

$$A^{\mathrm{MultiHead}}=\mathrm{Concat}({\mathrm{A}}_{{\mathrm{rel}}_1}^{\mathrm{mask}},\cdots ,{\mathrm{A}}_{{\mathrm{rel}}_i}^{\mathrm{mask}},\cdots ,{\mathrm{A}}_{{\mathrm{rel}}_h}^{\mathrm{mask}})W^O$$

(5)

where the parameter matrix $W^O\in {\mathbb{R}}^{hN\times d}$ and h is the attention head number mapping to 16 inter-clause relations in HCA [24].

3.2. HCA-Based Curriculum Learning

Inspired by the idea of curriculum learning [52] that humans handle difficult tasks from easy examples to hard ones, we propose an HCA-CL approach for training an AMR parser, where the clause number and the tree depth of a sentence’s HCA are the measurements of learning difficulty. Referring to the previous work of Wang et al. [18], we set two learning curricula, Clause-Number and Tree-Depth, in our HCA-CL approach demonstrated in Figure 4.

3.2.1. Clause-Number Curriculum

In Clause-Number (CN) curriculum, sentences with more clauses that involve more inter-clause relations and longer dependency distances (demonstrated in Figure 1) are considered to be harder to learn. With this intuition, all training sentences are divided into N buckets according to their clause number $\{C_i:i=1,\cdots ,N\}$, where $C_i$ contains sentences with the clause number i. In the training epoch of the Clause-Number curriculum, there are N training episodes with $T_{cn}$ steps individually. In each step of the i-th episode, the training scheduler samples a batch of examples from buckets $\{I_j:j\le i\}$ to train the model.

3.2.2. Tree-Depth Curriculum

In Tree-Depth (TD) curriculum, sentences with deeper HCA trees that correspond to deeper hierarchical AMR graphs are considered to be harder to learn. With this intuition, all training sentences are divided into M buckets according to their clause number $\{D_i:i=1,\cdots ,M\}$, where $D_i$ contains sentences with the clause number i. In the training epoch of the Clause-Number curriculum, there are M training episodes with $T_{td}$ steps individually. In each step of the i-th episode, the training scheduler samples a batch of examples from buckets $\{I_j:j\le i\}$ to train the model.

4. Experiments

In this section, we describe the details of datasets, environments, model hyperparameters, evaluation metrics, compared models, and parsing results in the experiments.

4.1. Datasets

For the benchmark datasets, we choose two standard AMR datasets, AMR 2.0 and AMR 3.0, as the ID settings and three test sets, TLP, New3, and Bio, as the OOD settings.

For the HCA tree of each sentence, we use the manual-annotated HCA trees for AMR 2.0 provided by [24], and auto-annotated HCA trees for the rest datasets, which are all generated by the HCA Segmenter and the HCA Parser proposed by [24].

4.1.1. In-Distribution Datasets

We first train and evaluate our HCA-based parser on two standard AMR parsing evaluation benchmarks:

• AMR 2.0: includes 39,260 sentence-AMR pairs in which source sentences are collected for the DARPA BOLT AND DEFT programs, transcripts and English translations of Mandarin Chinese broadcast news programming from China Central TV, Wall Street Journal text, translated Xinhua news texts, various newswire data from NIST OpenMT evaluations and weblog data used in the DARPA GALE program.
• AMR 3.0: is a superset of AMR 2.0 and enriches the data instances to 59,255. New source data added to AMR 3.0 includes sentences from Aesop’s Fables, parallel text and the situation frame data set developed by LDC for the DARPA LORELEI program, and lead sentences from Wikipedia articles about named entities.

The training, development, and test sets in both datasets are a random split, and therefore we take them as ID datasets like previous works [15,17,18,20,42].

4.1.2. Out-of-Distribution Datasets

To further estimate the effects of our HCA-based approaches on open-world data that comes from a different distribution, we follow the OOD settings introduced by [15], and predict on three OOD test sets with the parser trained on the AMR 2.0 training set:

• New3 4: a set of 527 instances from AMR 3.0, whose source was the LORELEI DARPA project – not included in the AMR 2.0 training set – consisting of excerpts from newswire and online forums
• TLP 5: the full AMR-tagged children’s novel, The Little Prince (version 3.0), consisting of 1,562 pairs
• Bio 6: the test set of the Bio-AMR corpus, consisting of 500 instances, featuring biomedical texts [53]

4.1.3. Hierarchical Clause Annotations

For the hierarchical clausal features utilized in our HCA-based approaches, we use the manual-annotated HCA corpus for AMR 2.0 provided in [24]. Moreover, we employ the HCA-segmenter and the HCA-parser proposed in [24] to generate silver HCA trees for AMR 3.0 and three OOD test sets. Detailed statistics of the evaluation datasets in this paper are listed in Table 1.

4.2. Baseline and Compared Models

We compare our HCA-based AMR parser with several recent parsers:

• AMR-gs (2020) [14], a graph-based parser that enhances incremental graph construction with an AMR graph↔sequence (AMR-gs) iterative inference mechanism in one-stage procedures.
• APT (2021) [41], a transition-based parser that employs an action-pointer Transformer (APT) to decouple source tokens from node representations and address alignments.
• StructBART (2021) [42], a transition-based parser that integrates the pre-trained language model, BART, for structured fine-tuning.
• SPRING (2021) [15], a fine-tuned BART model that predicts a linearized AMR graph.
• HCL (2022) [18], a hierarchical curriculum learning (HCL) framework that helps the seq2seq model adapts to the AMR hierarchy.
• ANCES (2022) [17], a seq2seq-based parser that adds the important ancestor (ANCES) information into the Transformer decoder.
• HGAN (2022) [20], a seq2seq-based parser that applies a heterogeneous graph attention network (HGAN) to argument word representations with syntactic dependencies and semantic role labelings of input sentences. It is also the current SOTA parser in the settings of removing graph re-categorization, extra silver training data, and ensemble methods.

Since SPRING provides a clear and efficient seq2seq-based architecture based on a vanilla BART, recent seq2seq-based models, HCA, ANCES, and HGAN, all select it as the codebase. Therefore, we also choose SRPING as the baseline model to apply our HCA-based approaches. Besides, we do not take the competitive AMR parser, ATP [19], into our compared models since it employs syntactic dependency parsing and semantic role labeling as intermediate tasks to introduce extra silver training data.

4.3. Hyper-Parameters

For the hyper-parameters of our HCA-based approaches, we list their layer, name, and value in Table 2. All models are trained until reaching the maximum epochs, and then select the best model checkpoint on the development set.

4.4. Evaluation Metrics

Following previous AMR parsing works, we use the Smatch scores [54] and the fine-grained metrics [55] to evaluate the performances. Specifically, the fine-grained AMR metrics are:

Unlabeled (Unlab.): Smatch computed on the predicted graphs after removing all edge labels.

No WSD. (NoWSD): Smatch while ignoring Propbank senses (e.g., “go-01” vs “go-02”).

Named entity (NER): F-score on the named entity recognition (:name roles).

Wikification (Wiki.): F-score on the wikification (:wiki roles).

Negation (Neg.): F-score on the negation detection (:polarity roles).

Concepts (Conc.): F-score on the concept identification task.

Reentrancy (Reent.): Smatch computed on reentrant edges only, e.g., the edges of node “I” in Figure 1.

(8)

Semantic role labelings (SRL): Smatch computed on :ARGi roles only.

As suggested in [18], Unlab., Reent, and SRL are as structure-dependent metrics, since

• Unlab. does not consider any edge labels and only considers the graph structure.
• Reent. is a typical structure feature for the AMR graph. Without reentrant edges, the AMR graph is reduced to a tree.
• SRL denotes the core-semantic relation of the AMR, which determines the core structure of the AMR.

Conversely, all other metrics are classified as structure-independent metrics.

4.5. Experimental Environments

Table 3 lists the information on the main hardware and software used in our experimental environments. Note that the model on AMR 2.0 is trained in a total of 30 epochs for 16 hours, while the model trained on AMR 3.0 is finished in a total of 30 epochs for 28 hours with the experimental environments.

4.6. Experimental Results

We now report the AMR parsing performances of our HCA-based parser and other compared parsers on ID datasets and OOD datasets, respectively.

4.6.1. Results in ID datasets

As demonstrated in Table 4, we report AMR parsing performances of the baseline model (SPRING), other compared parsers, and the modified SRPING that applies our HCA-based Self-Attention (HCA-SA) and Curriculum Learning (HCA-CL) approaches on ID datasets, AMR 2.0 and AMR 3.0. All the results of our HCA-based model have averaged scores of five experimental trials, and we compute the significance of performance differences using the non-parametric approximate randomization test [56]. From the results, we have the following observations:

• Equipped with our HCA-SA and HCA-CL approaches, the baseline model SPRING achieves 0.7 Smatch F1 score improvements on both AMR 2.0 and AMR 3.0. The improvements are significant with $p<0.005$ and $p<0.001$, respectively.
• In AMR 2.0, our HCA-based model outperforms all compared models except ANCES and the HGAN version that introduces both DP and SRL features.
• In AMR 3.0, consisting of more sentences with HCA trees, the performance gap between our HCA-based parser and the SOTA (HGAN with DP and SRL) is only 0.2 Smatch F1 scores.

To better analyze how the performance improvements of the baseline model are achieved when applying our HCA-based approaches, we also report structure-dependent fine-grained results in Table 4. As claimed in Section 1, inter-clause relations in the HCA can bring LDD issues, which are typically related to AMR concept nodes aligned with verb phrases, and reflected in structure-dependent metrics. As can be observed:

• Our HCA-based model outperforms the baseline model in nearly all fine-grained metrics, especially in structure-dependent metrics with 1.1, 1.8, and 3.9 F1 scores improvements in Unlab., Reent., and SRL, respectively.
• In the $SRL$, $Conc.$, and $Neg$ metrics, our HCA-based model achieves the best performances against all compared models.

4.6.2. Results in OOD Datasets

As demonstrated in Table 5, we report the parsing performances of our HCA-based model and compared models on three OOD datasets. As can be seen,

• Our HCA-based model outperforms the baseline model SPRING with 2.5, 0.7, and 3.1 Smatch F1 score improvements in New3, TLP, and Bio test sets, respectively.
• In the New3 and Bio datasets that contain long sentences of newswire and biomedical texts and have more HCA trees, our HCA-based model outperforms all compared models.
• In the TLP dataset that contains many simple sentences of a children’s story and fewer HCA trees, our HCA-based does not perform as well as HCL and HGAN.

5. Discussion

As shown in the previous section, our HCA-based model achieves prominent improvements against the baseline model, SPRING, and outperforms other compared models, including the SOTA model, HGAN in some fine-grained metrics in ID and ODD datasets. In this section, we further discuss the paper’s main issue of whether our HCA-based approaches have effects in curing LDDs. Additionally, the ablation studies and the case studies are also provided.

5.1. Effects on Long-Distance Dependencies in ID datasets

As claimed in Section 1, most LDD cases occur in sentences with complex hierarchical clause structures. In Figure 5, we demonstrate the parsing performances trends of the baseline model SPRING, the SOTA parser HGAN7, and our HCA-based model over the number of tokens and clauses in sentences from AMR 2.0. As can be observed:

• When the number of tokens (denoted as #Token for simplicity ) >20 in a sentence, the performance boosts of our HCA-based model against the baseline SPRING gradually become significant.
• For the case #Token>50 that indicates sentences with many clauses and inter-clause relations, our HCA-based model outperforms both SPRING and HGAN.
• When compared on performances trends over #Clause, the performance lead of our HCA-based model against SPRING becomes much more evident as #Clause increases.

To sum up, our HCA-based approaches show significant effectiveness on long sentences with complex clausal structures that introduce most LDD cases.

5.2. Effects on Long-Distance Dependencies in OOD datasets

As the performance improvements achieved by our HCA-based approaches are much more prominent on OOD datasets than on ID datasets, we have further exploration of the OOD datasets with different characteristics.

Figure 6 demonstrates the two main statistics of three OOD datasets, i.e., the average number of clauses per sentence (denoted as $\overline{\#C/S}$) and the average number of tokens per clause ($\overline{\#T/C}$). These two statistics of datasets both characterize the complexity of the clausal structure inside a sentence, where

• $\overline{\#C/S}$ shows the number of complex sentences with more than one clauses
• $\overline{\#T/C}$ depicts the latent dependency distance between two tokens from different clauses

We also present the performance boosts of our HCA-based parser against SPRING in Figure 6. As can be observed, the higher values of $\overline{\#C/S}$ and $\overline{\#C/S}$ in an OOD dataset, the higher Smatch improvements are achieved by our HCA-based approaches. Specifically, New3 and Bio seem to cover more complex texts from newswire and biomedical articles, while TLP contains simpler sentences that are easy for children to read. Therefore, our AMR parser performs much better on complex sentences from Bio and New3, indicating the effectiveness of our HCA-based approaches on LDDs.

5.3. Ablation Study

In the HCA-SA approach, two token visibility matrices derived from HCA trees are introduced to mask certain attention heads. Additionally, we propose a clause-relation-binded attention head setting to integrate inter-clause relations in the encoder. Therefore, we conduct ablation studies by introducing random token visibility matrices (denoted as “w/o VisMask”) and removing the clause-relation-binded attention setting (denoted as “w/o ClauRel”). Note that “w/o VisMask” contains the case of “w/o ClauRel” because the clause-relation-binded attention setting is based on the masked-self-attention mechanism.

In the HCA-CL approach, extra training epochs for Clause-Number and Tree-Depth curricula serve as a warm-up stage for the subsequent training process. To eliminate the effect of the extra epochs, we also add the same number of training epochs in the ablation study of our HCA-CL approach.

As shown in Table 6:

• In HCA-SA, the clause-relation-binded attention setting (demoted as “ClauRel”) contributes most in the SRL metric due to the mappings between inter-clause relations (e.g., Subjective and Objective) and SRL-type AMR relations (e.g., :ARG0 and :ARG1).
• In HCA-SA, the masked-self-attention mechanism (demoted as “VisMask”) achieves significant improvements in the Reent. metric by increasing the visibility of pronoun tokens to all tokens.
• In HCA-CL, the Tree-Depth curriculum (demoted as “TD”) has no effects on the parsing performances. We conjecture that sentences with much deeper clausal structures are rare, and the number of split buckets for the depth of clausal trees is not big enough to distinguish the training sentences.

5.4. Case Study

To further demonstrate the effectiveness of our HCA-based approaches on LDDs in AMR parsing, we compare the output AMR graphs of the same sentence exemplified in Figure 1, parsed by the baseline model SPRING and the modified SPRING that applies our HCA-SA and HCA-CL approaches (denoted as Ours), respectively in Figure 7.

For SPRING, it mislabels node “go-02” in subgraph $G_3$ as the :ARG1 role of node “contrast-01”. Then it fails to realize that it is “anxious” in $G_2$ that takes the :ARG1 role of “go-02” in $G_3$. Additionally, the causality between $G_4$ and $G_5$ is not interpreted correctly due to the absence of node “cause-01” and its arguments.

In contrast, when integrating the HCA, Ours seems to understand the inter-clause relations better. Although “possible-01” in subgraph $G_5$ is mislabeled as the :ARG2 role of node “contrast-01”, it succeeds in avoiding errors made by SPRING. Another mistake in Ours is that the relation :quant between “much” and “anxiety” is reversed and replaced by :domain, which impacts little on Smatch F1 scores. The vast performance gap between SPRING and our HCA-based SPRING in Smatch F1 scores% (66.8 vs. 88.7) also proves the effectiveness of the HCA on LDDs in AMR parsing.

6. Conclusion

We propose two HCA-based approaches, HCA-SA and HCA-CL, to integrate HCA trees of complex sentences for addressing LDDs in AMR parsing. Taking AMR parsing as an example, we apply our HCA-based framework to a popular AMR parser SPRING to integrate HCA features in the encoder. In the evaluations on ID datasets, our parser achieves prominent and explainable improvements against the baseline model, SPRING, and outperforms the SOTA parser, HGAN, in some fine-grained metrics. Notably, as the clause number of the sentence increases, our parser outperforms SPRING by a large margin and achieves better Smatch F1 scores than HGAN, indicating the ability to cure LDDs. In the evaluations on OOD datasets, the performance boosts achieved by our HCA-based approaches are more evident on complicated corpora like New3 and Bio, where sentences consist of more clauses and longer clauses.