Hierarchical Clause Annotation: Building a Clause-Level Corpus for Semantic Parsing with Complex Sentences

1. Introduction

Most natural language processing (NLP) tasks suffer performance degradation when encountering long complex sentences, such as abstract meaning representation (AMR) parsing [1], semantic dependency parsing [2], constituency parsing [3], semantic role labeling [4], machine translation [5], and text summarization [6]. An intuition to address this issue is first to decompose complex sentences and then re-link simple ones, which shares similar ideas with tasks like RST-style discourse parsing (hereafter RST parsing) [7], split-and-rephrase (SPRP) [8], text simplification (TS) [9], simple-sentence-decomposition (SSD1) [10], etc.

However, previous works with such intuitions to process complex sentences are not practical for semantic parsing tasks, such as AMR [11] and semantic dependency parsing [12]. RST parsing, aiming to extract the rhetorical relations among elementary discourse units (EDUs) [13] at a document level, is still an open problem, where the state-of-the-art model only achieves 55.4 and 80.4 Parseval-Full scores for multi- and intra-sentential parsing. Besides, the blurry definitions of EDUs and the misalignments between rhetorical relations and semantic relations make RST parsing unsuitable for semantic parsing. The SPRP task keeps a big splitting granularity, where outputs may still be complex sentences. The TS and SSD tasks, which decompose complex sentences into simple ones, can not preserve the original semantics for rephrasing for simpler syntax (TS) or dropping discourse connectives (SSD).

In this paper, we propose a novel task, hierarchical clause annotation (HCA), based on the linguistic research of clause hierarchy [14], where clauses are fundamental text units centering on a verb phrase, and sentences with multiple clauses form a complex hierarchy. Our HCA is a more lightweight task at the sentence level, has explicit definitions of clauses and appropriate mappings between inter-clause and semantic relations (vs. RST parsing), and aims to annotate complex sentences into a clause hierarchy (vs. SPRP) without changing or dropping any semantics (vs. TS and SSD).

To show the potentialities of HCA to facilitate semantic parsing with complex sentences, we demonstrate the HCA tree, AMR graph, and semantic dependency graph (SDG) of a complex sentence from the AMR 2.0 dataset2 in Figure 1:

$$\begin{array}{cc}\hfill & {[IfIdonotcheck,]}_{C_1}{[Igetveryanxious,]}_{C_2}[whichdoessortofgoawayafter15-30\hfill \\ \hfill & {mins,]}_{C_3}{\left[butoftentheanxietyissomuch\right]}_{C_4}{[thatIcannotwaitthatlong.]}_{C_5}\hfill \end{array}$$

The above sentence is segmented into five clauses $C_i$, where

• $C_2$ and $C_4$ are coordinate and contrastive,
• $C_1$ and $C_3$ are conditional adverbial and relative clauses of $C_2$, respectively,
• $C_5$ is a resultative adverbial clause of $C_4$.

In the HCA tree, the coordinate relation $But$ and clauses $C_i$ are nodes, and subordinate relations are directed edges. As demonstrated in Figure 1, the HCA tree shares the same hierarchy with two semantic parsing representations, indicating the possibility of incorporating HCA’s structural information into semantic parsing with complex sentences.

Our main contributions are as follows:

• We propose a novel framework, hierarchical clause annotation (HCA), to segment complex sentences into clauses and capture their interrelations based on the linguistic research of clause hierarchy, aiming to provide clause-level structural features to facilitate semantic parsing tasks.
• We elaborate on our experience developing a large HCA corpus – including determining an annotation framework, creating a silver-to-gold manual annotation tool, and ensuring annotating quality. The resulting HCA corpus contains $19,376$ English sentences from AMR 2.0, each including at least two clauses.
• We decompose HCA into two subtasks, i.e., clause segmentation and parsing, and adapt discourse segmentation and parsing models for the HCA subtasks. Experimental results show that the adapted models achieve satisfactory performances in providing reliable silver HCA data.

2. Related Work

2.1. RST Parsing

In a document, the clauses, sentences, and paragraphs are logically connected together to form a coherent discourse. Rhetorical Structure Theory (RST) [13] provides a general way to describe the relations among parts in a text and postulates a hierarchical discourse structure called discourse tree (DT). The leaves of a DT can be a clause or a phrase without strict definitions, known as elementary discourse units (EDUs). Adjacent EDUs and higher-order spans are non-overlapping and connected hierarchically through coherence relations. Thus, coherence in discourse can be analyzed in terms of how a nucleus interacts with its surrounding satellites to communicate the relationships between main ideas and related ideas.

The RST parsing task generally requires breaking the text into EDUs (i.e., the discourse segmentation task) and linking the EDUs into a DT (i.e., the discourse parsing task). For discourse segmentation, Gessler et al. [15] proposed a Transformer-based neural classifier that enhances contextualized word embeddings with hand-crafted features and achieved the current state-of-the-art performance in DISRPT 2021 Shared Task on Discourse Unit Segmentation3. For discourse parsing, Kobayashi et al. [16] explored a strong baseline by integrating previous simple parsing strategies, top-down and bottom-up, with various transformer-based pretrained language models (PLMs).

Table 1 demonstrates the comparison between our HCA and RST parsing with exemplified sentences from RST Discourse Tagging Reference Manual4. Although both tasks aim to extract a tree structure from input texts, their definitions of elementary units and target interrelations vary, leading to the following differences.

• The elementary units of HCA are clauses, while that of RST parsing are EDUs, including clauses and phrases. The blurry definitions of an EDU may cause obstacles in RST parsing. For example, phrases“as a result of margin calls” in (1) and “Despite some their considerable incomes and assets” in (2) are segmented as EDUs but not clauses due to the absence of a verb. Moreover, although the clause “that they have made it” functions as a predicative of the verb “feel”, it can not be annotated as an EDU.
• The rhetorical relations in RST parsing characterize the coherence among EDUs, and some can not map to semantic relations. For example, the semantic relation between “feel” and “that they have made it” in (2) is not captured in RST parsing.

In summary, the blurry definitions of EDUs and the misalignments between rhetorical and semantic relations make RST parsing unsuitable for semantic parsing compared with our HCA.

2.2. Other Similar Tasks

Some similar tasks share the idea of decomposing complex sentences into simpler parts without capturing their interrelations.

2.2.1. Clause Identification

The CoNLL-2001 shared task, clause identification, [17] proposes a dataset with the gold standard clause provided by the Penn Treebank II [18], where clause-level tags (i.e., S, SBAR, SBARQ, INV, and SQ) indicate target clauses and clausal conjunctions. The clauses identified in the shared task comprise tensed clauses, non-tensed verb phrases, coordinators, and subordinators.

2.2.2. Split-and-Rephrase

The split-and-rephrase (SPRP) task [8] aims to split a complex input sentence into shorter sentences while preserving meaning. In that task, the emphasis is on sentence splitting and rephrasing. There is no deletion and no lexical or phrasal simplification, but the systems must learn to split complex sentences into shorter ones and to make the syntactic transformations required by the split(e.g., turn a relative clause into a main clause).

2.2.3. Text Simplification

The text simplification (TS) task [19] is the process of reducing the linguistic complexity of a text to improve its understandability and readability while maintaining its original information content and meaning. Typically, it rephrases complex sentences with simpler vocabulary and syntax and ignores trivial clauses from the source.

2.2.4. Simple-Sentence-Decomposition

The simple-sentence-decomposition (SSD) task [10] converts complex sentences into a covering set of simple sentences derived from the tensed clauses in the source sentence, where shared nouns or pronouns are copied, and discourse connectives (e.g., and, but, although, etc) are dropped.

2.2.5. Summary

Table 2 compares our HCA task and similar tasks above with the exemplified sentence in Section 1.

• For clause identification, coordinator “but” and subordinators “If”, “which”, and “that” are segmented out. Besides, non-tensed verb phrases that function as a subject, object, or postmodifier are also target clauses in the task. These cases are out of the definition of annotated clauses in our HCA framework, as redundant hierarchies occur in capturing inter-clause relations.
• For split-and-rephrase, the granularity of decomposing is larger than clauses due to the consideration of preserving the original meaning of the input sentence. The output (1) and (3) are still complex sentences with two clauses. Additionally, the output (2) is segmented from the relative clause that modifies the “anxious” in the matrix clause, leading to a syntax transformation.
• For text simplification, dropping the subordinator “If” and the coordinator “but” leads to the uncertainties of discourse relations between output sentences. Moreover, as replacements for simpler syntax in (3) and (4) bring misalignments between substitute and substituted tokens, text simplification is unsuitable to serve as a preprocess for semantic dependency parsing, which is a token-level task.
• For simple-sentence-decomposition, it also drops clausal connectives like text simplification, leading to the uncertainties of some discourse relations captured by semantic parsing.

2.3. Clause Hierarchy

Clause Hierarchy can be described as a cline along which clauses distribute according to their different levels of grammatical integration [14,20,21,22,23,24]. These works propose that clause combinations in many languages can be described as a set of tighter or looser clauses. A tight clause means that a clausal constituent has, in comparison to a loose clause, more dependence on the clause with which it combines, typically a main clause. Table 3 shows three main versions of Clause Hierarchy and ordered linguistic phenomena according to their clause integration tightness degree.

Matthiessen [21] presented a type of clause hierarchy that extends from syntactic clause combination to cohesion and coherence at the discourse and text level. He considers clause combination to range from tight syntactic “embedding” (e.g., infinitival clauses as a complement to a main verb) to the looser relations of “hypotaxis” (e. g. a finite adverbial) to “parataxis” (e.g., coordination).

Hopper and Traugott [22] also offered a model that shares much with Matthiessen’s, where “parataxis” is the syntactic independence of clauses and “hypotaxis” is a more integrated clause that is syntactically dependent within another clause’s predicate. Tighter still is “subordination”, which, like “embedding” in Matthiessen’s type, covers all clauses which function as a constituent essential to grammaticality, e.g., verbal arguments.

Payne’s clause hierarchy [14] extended from “compound verbs” (tightest) through to separate “sentences” (loosest), where clause combination becomes more or less a single verbal element in his type. Different from the above two types, Payne argues that compound verbs (e.g., go get the book), though uncommon in English, are considered the tightest clause combination as they have two verbal elements placed adjacently in a verb phrase, one of which lacks full finiteness.

In summary, the linguistic researches of Clause Hierarchy provide solid theoretical support for our HCA framework.

3. Hierarchical Clause Annotation Framework

To present an framework for annotation the clause hierarchy of complex sentences, we reference and modify Payne’s version [14] of Clause Hierarchy due to his pellucid and comprehensive definitions of clause combination cases. As demonstrated in Figure 2, we do not consider compound verbs as a clause combination, as these cases are uncommon and produce one-verb clauses after annotation.

With the above version of Clause Hierarchy, we synthesize the HCA framework and build a dataset under the guidance of the framework. The annotation work consists of a preprocessing stage with silver annotations transformed from existing schemas (constituent parsing and syntactic dependency parsing) and a manual proofreading phase with gold annotations on an elaborate browser-based annotator.

3.1. Concept

We list major concepts in the HCA framework.

3.1.1. Sentence and Clause

Sentences, typically starting with a capitalized word and ending with a complete stop, are principally units of written grammar and annotation inputs in HCA. A sentence must consist of at least one clause.

Clauses, considered core units of grammar, center around a verb phrase that largely determines what else must or may occur [25]. Clauses can be categorized by the inner verb type:

• Finite: clauses that contain tensed verbs;
• Non-finite : clauses that only contain non-tensed verbs like ing-participles, ed-participles, and to-infinitives.

In the HCA framework, finite clauses should be annotated, while non-finite clauses that are separated by a comma are also segmented out.

3.1.2. Clause Combination

The main ways in which clauses combine to form sentences are by joining clauses of equal syntactic status (coordination) and subordinate relation (subordination).

(1)

Coordination and Coordinator

Coordination is an interrelation between clauses that share the same syntactic status and are typically connected by a coordinator such as and, or, but, etc. In addition, coordinators can be correlative structures (e.g., either...or... and not only...but also...) or just substituted by comma punctuation.

(2)

Subordination, Subordinator, and Antecedent

Subordination occurs in a subordinate clause and a matrix clause that is superordinate to the subordinate clause. Subordination can be cataloged as follows:

• Nominative: Function as clausal arguments or noun phrases in the matrix clause and can be subdivided into Subjective, Objective, Predicative and Appositive.
• Relative: Define or describe a preceding noun head in the matrix clause.
• Adverbial: Function as a Condition, Concession, Reason, and such for the matrix clause.

Subordinators are the words that introduce a subordinate clause and indicate a semantic relation between the subordinate clause and its matrix clause, including subordinate conjunctions, relative pronouns, and relative adverbs. Simple subordinators contain a single word, e.g., that, wh-words, if, etc., while complex ones consist of more than one word, e.g., as if, so that, even though, etc.

Antecedents are nouns or pronouns modified before relative clauses and nouns explained before appositive clauses.

To better explain these HCA definitions, we demonstrate some example sentences which are segmented into multiple clauses in Table 4.

3.2. HCA Representation

As illustrated in Figure 3, we model the two basic hierarchical schemas with concepts defined in Section 3.1, characterizing inter-clause relations with the same nucleus-satellite pattern in RST. To be specific, coordination is a multinuclear relation that involves two or more clauses (denoted as nucleus node $C_i$) dominated by the coordination node $co$, while subordination is a mononuclear relation (denoted as a directed edge $sub$) pointing from the matrix clause $C_1$ (nucleus) to its subordinate clause $C_2$ (satellite).

As a sentence consists of more clauses, its HCA representation can be a tree structure, where each node is a clause or an inter-clause coordination, and each directed edge is an inter-clause subordination. A three-layer HCA tree of a complex sentence involving five clauses and four interrelations is demonstrated in Figure 1a.

3.3. HCA corpus

With the annotation framework discussed above, we aim to build an HCA corpus for further research on the possibilities of applying clausal structure features to semantic parsing tasks. We choose the AMR 2.0 dataset as our corpus base, whose $39,260$ sentences are collected from the DARPA BOLT and DEFT programs, various newswire data, and weblog data.

The annotation work is conducted in two phases. First, two existing syntactic features, i.e., constituent and syntactic dependency parse trees, are employed to produce silver HCA annotations with transformation rules. Second, human annotators with prior English grammar research experience and extensive hands-on annotation training, review and modify silver annotations on a browser-based annotation tool.

3.3.1. Silver Data from Existing Schemas

Previous researchers[26,27,28,29] utilize constituent-based and syntactic dependency parse trees to extract clauses from sentences with some manual rules. Following the experience from these works, we employ Stanza [30] as our constituent parser and syntactic dependency parser to obtain silver HCA data.

• Consituency Parse Tree

The constituency parse tree (CPT) represents the syntactic structure of a sentence using a tree, where the nodes are sub-phrases that belong to a specific category in the grammar, and the edges are unlabeled. The transformation from CPT to the silver HCA data consists of three phases:

• Traverse non-leaf nodes in the CPT and find the clause-type nodes: S, SBAR, SBARQ, INV, and SQ.
• Identify the tokens dominated by a clause-type node as a clause.
• When a clause-type node dominates another one, an inter-clause relation between them is determined without an exact relation type.

As demonstrated in Figure 4, the first two clauses of the sentence exemplified in Section 1 are identified through their constituent parse tree. The SBAR node and its child node S are combined as a single clause, as no VP is dominated by the other child constituent IN of SBAR. Moreover, the S node on the top dominates the SBAR node, indicating subordination between the two clauses in dashed boxes.

• Syntactic Dependency Parse Tree

The syntactic dependency parse tree (SDPT) consists of a set of directed syntactic relations between the words in the sentence whose root is either a non-copular verb or the subject complement of a copular verb. The transformation from SDPT to silver HCA consists of three phases:

• Use a mapping of dependency relations to clause constituents: subjects (S) and the governor, i.e., a non-copular verb(V), via relation nsubj and such; objects (O) and complements (C) in V’s dependants via relations dobj, iobj, xcomp, ccomp, and such; adverbials (A) in V’s dependents via relations advmod, advcl, prep_in, and such.
• When detecting a verb5 in the sentence, a corresponding clause, consisting of the verb and its dependant constituents, can be identified.
• If a clause governs another clause via a dependency relation, the interrelation between them can be determined by the relation label:
  • Coordination: conj:and, conj:or, conj:but
  • Subjective: nsubj
  • Objective: dobj, iobj
  • Predicative: xcomp, ccmop
  • Appositive: appos
  • Relative: ccomp, acl:relcl, rcmod
  • Adverbial: advcl

As demonstrated in Figure 5, the first two clauses of the sentence exemplified in Section 1 are identified through their syntactic dependency tree. Moreover, the inter-clause relation can be inferred as Adverbial: Conditional with the dependency relation advcl and the subordinator “If”.

3.3.2. Gold Data from Manual Annotator

As discussed above, the syntactic structures of CPT and SDPT can be transformed into clauses and inter-clause relations. However, these silver annotations are still unable to fulfill the need to build an HCA corpus for the following reasons:

• Specific inter-clause relations can not be obtained via the two syntactic structures, where CPT can only provide the existence of a relation without a label, and the dependency relations in SDPT have multiple mappings (e.g., ccomp to Predcative or Relative) or no mapping (e.g., advcl to no exact Adverbial subtype like Conditional).
• Pre-set transformation rules identify more clauses out of the HCA definitions. For example, the extracted non-finite clauses (e.g., to-infinitives) embedded in its matrix clause are too short and lead to hierarchical redundancies in the HCA tree.
• The performances of two syntactic parsers degrade when encountering long and complex sentences which are the core concerns of our HCA corpus.

Therefore, we recruit a group of human annotators with prior English grammar research experience to proofread these silver HCA on a browsed-based software ClausAnn created for the annotation work. The Java Web application ClausAnn provides convenient operations and efficient keyboard shortcuts for annotators, and we open source it on our GitHub repository6. A typical annotation trial on ClausAnn consists of the following steps:

• Review annotations from CPT, SDPT, or other annotators by switching the name tags in Figure 6a.
• Choose an existing annotation to proofread or just start from the original sentence.
• Segment a text span into two by double-clicking the blank space of a split point, and select the relation between them in Figure 6b.

  (a)

  If the two spans are coordinated, select a specific coordination and label coordinators that indicate the interrelation in Figure 6c.

  (b)

  If the two spans are subordinated, designate the superordinate one, select a specific subordination, and label subordinators that indicate the interrelation in Figure 6d.

• Remerge two text spans into one by clicking the two spans successively.
• Repeat step c and d until all text spans are segmented into a set of clauses constructed in a right HCA tree.

3.3.3. Quality Assurance

There are mainly two steps taken jointly to ensure the quality of the final HCA corpus, i.e., multi-round annotation and consistency measurement.

The total annotation work consists of $39,260$ sentences, and three rounds of annotation are arranged by 5%, 5%, and 90% of total sentences and conducted with a progressive and negotiable strategy. Before the first round, every annotator thoroughly understands the HCA framework and uses the tool ClausAnn proficiently after adequate hands-on training. During the first two rounds, the lead annotator, who majors in English grammar, organizes a discussion on complex or abnormal cases with other annotators.

For consistency measurement, we tracked inter-annotator agreement (IAA) after each round of the annotation work. As discussed in Section 2.1, the HCA and RST parsing tasks aim to extract the clause/EDU hierarchy from texts. Thus, the evaluation metrics of discourse segmentation [15] and discours parsing [16] subtasks in RST parsing are adopted as IAA metrics in evaluating the annotation quality of the HCA corpus:

• P/R/$F_1$ on clauses: precision, recall, and $F_1$ score on the segmented clauses, where a positive match means that both segmented clauses from two annotators have the same start and end boundaries.
• RST-Parseval [31] on interrelations: consisting of Span, Nuclearity, Relation, and Full are used to evaluate unlabeled, nuclearity-, relation-, and fully labeled interrelations respectively between the matched clauses from two annotators.

In the first two rounds of annotation, a total of 10% sentences are double-annotated, and the ratio comes to 16% in the last round, higher than 13.8% in the RST-DT corpus [7]. According to the statistics, the IAA measured by the above two metrics grows as the annotation rounds increase, indicating that the two steps of multi-round annotation and consistency measurement play a significant role in ensuring annotation quality.

As shown in Table 5, the final IAA achieves high consistencies, where

• $F_1$ scores on clauses range from 98.4 to 100,
• RST-Parseval scores on interrelations range from 97.3, 97.0, 93.6, 93.4 to 98.1, 97.8, 94.2, 94.1 in Span, Nuclearity, Relation, and Full, respectively.

Compared with the RST-DT corpus, whose IAA score on EDUs ranges from 95.1 to 100 and IAA scores on rhetorical relations with three metrics, Spans, Nuclearity, Relation, range from 77.8, 69.5, 59.7 to 92.9, 88.2, 79.2, our HCA corpus reaches better consistencies, as the HCA framework has more restrict definitions on the elementary unit (i.e., clauses) and fewer types of the interrelation.

3.3.4. Dataset Detail

The resulting HCA-AMR2.0 dataset is based on AMR 2.0, which contains $39,260$ sentences, and $19,376$ (49.4%) sentences are paired with an HCA tree, while the rest are simple sentences with only one clause. The train, dev, and test set split follows the original split in AMR 2.0. Detailed statistics are listed in Table 6.

4. Model

In this paper, we model hierarchical clause annotation (HCA) as a two-stage task, i.e., clause segmentation and clause parsing, and provide auto-annotation baselines for each subtask. Clause segmentation segments a complex sentence into several clauses, while clause parsing links the clauses with interrelations into a clause tree.

4.1. Clause Segmentation

We encounter clause segmentation task with a sequence tagging model, DiscoDisCo [15] used in the discourse segmentation task: embed the input sentence, encode with a single Bi-LSTM, decode with a linear projection layer, and indicate the first token of each clause in the output tag sequence.

In the embedding layer, we rely on three kinds of word embeddings concatenated together:

(1)

Bi-LSTM encoded character embeddings,

(2)

static word embeddings from fastText, [32]

(3)

and fine-tuning word embeddings from a pretrained language model (PLM).

Additionally, we introduce and embed various grammatical information like lemmas, part-of-speech (POS), and syntactic dependencies generated by Stanza [30]. These tokenwise features embeddings are concatenated with word embeddings, and the word $w_i$ in the input sentence is embedded as

$$u_i=\mathrm{Concat}(u_i^{word}+u_i^{feat})$$

(1)

where $u_i^{word}$ is concatenated from three kinds of word embeddings and $u_i^{feat}$ is grammatical features embeddings.

The embedding ${\mathbf{u}}_{1:n}$ of the input sentence with n tokens are fed through a BiLSTM network, and the output $s_i$ is then calculated by a linear projection layer to predict the segmentation tag for i-th token:

$$s_i=\mathrm{Linear}\left(\mathrm{BiLSTM}({\mathbf{u}}_{1:n},i)\right).$$

(2)

Given $s_i$, we get the predicted tag sequence $\widehat{\mathbf{t}}$, and optimizing the cross-entropy loss of $\mathcal{L}(\widehat{\mathbf{t}},\mathbf{t})$ to train the weights of the PLM, the BiLSTM network and the linear projection layer, where $\mathbf{t}$ is the gold tag sequence.

4.2. Clause Parsing

Given segmented clauses in the input sentence, we model clause parsing as a discourse parsing task, where clauses are considered as EDUs and linked into a clause tree. Therefore, we employ discourse parsers in [16], which contain a span-based parser with the top-down strategy and a shift-reduce transition parser with the bottom-up strategy for their simple architectures and open codes. Overviews of the parsers are shown in Figure 7 and Figure 7. Note that the two parsing strategies share the same word embedding layer to represent text spans.

4.2.1. Text Span Embedding

In the process of clause parsing, the representation of text spans is needed for either “span-to-clause” splitting in the top-down parsing strategy or “clause-to-span” combining in the bottom-up parsing strategy. Therefore, we transform the input sentence into a subword sequence $\{t_1,t_2,\cdots t_n\}$, and obtain the embedding $\{w_1,w_2,\cdots w_n\}$ using a PLM. The embedding for a text span ${\mathbf{u}}_{i:j}$, consisting of the i-th clause to the j-th clause, is obtained by averaging the vector of both edge subwords,

$${\mathbf{u}}_{i:j}=(w_{\mathrm{b}\left(i\right)}+w_{\mathrm{e}\left(j\right)})/2,$$

(3)

where $\mathrm{b}\left(i\right)$ returns the index of the begin subword in the i-th clause and $\mathrm{e}\left(j\right)$ returns that of the end subword in the j-th clause.

4.2.2. Top-Down Strategy

The top-down parser splits each span into smaller ones recursively until the span becomes a single clause. We introduce biaffine networks [33] for span splitting and a loss penalty.

For each position k in a span consisting of the i-th clause to the j-th clause, a scoring function $s_{\mathrm{split}}(i,j,k)$, is defined as follows:

$$s_{\mathrm{split}}(i,j,k)={\mathbf{h}}_{i:k}{\mathbf{Wh}}_{k+1:j}+{\mathbf{v}}_{\mathrm{left}}{\mathbf{h}}_{i:k}+{\mathbf{v}}_{\mathrm{right}}{\mathbf{h}}_{k+1:j}$$

(4)

where $\mathbf{W}$, ${\mathbf{v}}_{\mathrm{left}}$ and ${\mathbf{v}}_{\mathrm{right}}$ are weight matrices in the biaffine layer for splitting a text span. Here, ${\mathbf{h}}_{i:k}$ and ${\mathbf{h}}_{k+1:j}$ are defined as follows:

$$\begin{array}{cc}\hfill {\mathbf{h}}_{i:k}& ={\mathrm{FFN}}_{\mathrm{left}}\left({\mathbf{u}}_{i:k}\right),\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill {\mathbf{h}}_{k+1:j}& ={\mathrm{FFN}}_{\mathrm{right}}\left({\mathbf{u}}_{k+1:j}\right)\hfill \end{array}$$

(6)

Then, the span is split at the end of an inner clause that maximizes Equation 4:

$$\widehat{k}=arg\underset{i\le k<j}{max}{s}_{\mathrm{split}}(i,j,k).$$

(7)

When splitting a span at the end of the $\widehat{k}$-th clause, the score of the nuclearity and relation labels for the two spans is defined as follows:

$$s_{\mathrm{label}}(i,j,\widehat{k},\ell )={\mathbf{h}}_{i:\widehat{k}}{\mathbf{W}}^{\ell }{\mathbf{h}}_{\widehat{k}+1:j}+{\mathbf{v}}_{\mathrm{left}}^{\ell }{\mathbf{h}}_{i:\widehat{k}}+{\mathbf{v}}_{\mathrm{right}}^{\ell }{\mathbf{h}}_{\widehat{k}+1:j}$$

(8)

where ${\mathbf{W}}^{\ell }$, ${\mathbf{v}}_{\mathrm{left}}^{\ell }$, and ${\mathbf{v}}_{\mathrm{right}}^{\ell }$ are weight matrices in the biaffine layer for predicting an inter-clause relation. Then, the label that maximizes Equation 8 is assigned to the spans:

$$\widehat{\ell }=arg\underset{\ell \in \mathcal{L}}{max}{s}_{\mathrm{label}}(i,j,\widehat{k},\ell )$$

(9)

where $\mathcal{L}$ denotes three nuclearity labels, $\{\stackrel{sub.}{⟶},\stackrel{sub.}{⟵},\stackrel{co.}{⟷}\}$, for predicting the nuclearity, and a set of inter-clause relations labels for predicting the exact relation. Note that the parameters in biaffine layers and $\mathrm{FFN}$s for the nuclearity and relation labeling are learned separately.

4.2.3. Bottom-Up Strategy

Formally, in a shift-reduce model with a bottom-up strategy, the parsing state is denoted as a tuple$(S,Q)$, where S is a stack that stores processed clauses and Q is a queue that contains incoming clauses. Each element in S can be an unreduced clause $e_i$ or a combined composite item $e_{i:j}$. At each step, the parser chooses one of the following actions with an FFN classifier and updates the state $(S,Q)$:

• $SHIFT$: pop the first clause off Q and push it onto S.
• $REDUCE$: pop two elements from S and push a new combined composite item that has the popped subtrees as its children onto S as a single composite item.

We employ three FFN classifiers, ${\mathrm{FFN}}_{\mathrm{act}}$, ${\mathrm{FFN}}_{\mathrm{nuc}}$, and ${\mathrm{FFN}}_{\mathrm{rel}}$, where ${\mathrm{FFN}}_{\mathrm{act}}$ predicts an action, and the rest two decide the nuclearity and the label of an inter-clause relation after a $REDUCE$ action. Specifically, the output dimension of ${\mathrm{FFN}}_{\mathrm{act}}$ is 2 ($SHIFT$ or $REDUCE$), that of ${\mathrm{FFN}}_{\mathrm{nuc}}$ is 3 ($\stackrel{sub.}{⟶}$, $\stackrel{sub.}{⟵}$ or $\stackrel{co.}{⟷}$), and that of ${\mathrm{FFN}}_{\mathrm{rel}}$ is the number of inter-clause relations defined in the HCA framework. Three classifier outputs $s_{*}$ are defined as

$$s_{*}={\mathrm{FFN}}_{*}\left(\mathrm{Concat}({\mathbf{u}}_{s_0},{\mathbf{u}}_{s_1},{\mathbf{u}}_{q_0})\right)$$

(10)

where function $\mathrm{Concat}$ concatenates three state vectors: ${\mathbf{u}}_{s_0}$ is the representation of the top clause stored in S, ${\mathbf{u}}_{s_1}$ is that in the second clause of S, and ${\mathbf{u}}_{q_0}$ is that in the first clause in Q. Weights of the PLM and each $\mathrm{FFN}$ are trained by optimizing the cross-entropy loss of $s_{\mathrm{act}}$, $s_{\mathrm{nuc}}$, and $s_{\mathrm{rel}}$.

5. Experiments

In this section, we elaborate on the experimental details of the proposed baseline models for two HCA subtasks, clause segmentation, and clause parsing, on the novel HCA-AMR2.0 corpus.

5.1. Dataset

Except for the HCA-AMR2.0 corpus, we reference three discourse analysis corpora, GUM [34], STAC [35], and RST-DT [7], to further verify the effectiveness of the proposed models and expound the distinctive advantages of HCA-AMR2.0 over these discourse analysis corpora.

• HCA-AMR2.0: The first HCA corpus is annotated on sentences in AMR 2.0 dataset. The source data includes discussion forums collected for the DARPA BOLT AND DEFT programs, transcripts and English translations of Mandarin Chinese broadcast news programming from China Central TV, Wall Street Journal text, translated Xinhua news texts, various newswire data from NIST OpenMT evaluations and weblog data used in the DARPA GALE program.
• GUM: The Georgetown University Multilayer corpus is created as part of the course LING-367 (Computational Corpus Linguistics) at Georgetown University, and its annotation is followed the RST-DT segmentation guidelines for English. The text sources consist of 35 documents of news, interviews, instruction books, and travel guides from WikiNews, WikiHow, and WikiVoyage.
• STAC: The Strategic Conversation dataset is a corpus of strategic chat conversations in 45 games annotated with negotiation-related information, dialogue acts, and discourse structures in Segmented Discourse Representation Theory (SDRT) framework.
• RST-DT: Rhetorical Structure Theory (RST) Discourse Treebank is developed by researchers at the Information Sciences Institute (University of Southern California), the US Department of Defense, and the Linguistic Data Consortium (LDC). It comprises 385 Wall Street Journal articles from the Penn Treebank annotated with discourse structure in the RST framework.

For the clause segmentation subtask, we select all three discourse analysis datasets evaluated on the discourse segmentation task as a comparison, and the data files are all in conll formats. For the clause parsing subtask, we only select the RST-DT dataset evaluated on the discourse parsing task as a comparison, as previous works of discourse parsing experiments on the other two datasets using accuracy as the evaluating metric other than Parseval. The interrelation data files in HCA and RST-DT are preprocessed by Heilman and Sagae’s system [36]. The important dataset statistics related to the experiments are listed in Table 7.

5.2. Experimental Environments

The information on the main hardware and software used in our experimental environments is listed in Table 8.

5.3. Hyper-Parameters

For the hyper-parameters in the models for clause segmentation and clause parsing subtasks, we list their layer, name, and value in Table 9. Note that both models are not trained until reaching the maximum epochs, where the model of clause segmentation is trained in about thirteen epochs for five hours, and the model of clause parsing is finished in about seven epochs for ten hours with the experimental environments introduced in Section 5.2.

5.4. Evaluation Metrics

As described in Section 3.3.3, we introduce two metrics for IAA to evaluate annotation consistencies between two annotators. Thus, we still use P/R/$F_1$ on clauses for clause segmentation subtask and RST-Parseval on inter-clause relations for clause parsing subtask to evaluate the consistencies between gold and predicted annotations.

5.5. Baseline Models

We aim to utilize models proposed by previous works on discourse segmentation and discourse parsing tasks and experiment with our HCA-AMR2.0 corpus to obtain effective baseline models for the novel clause segmentation and parsing subtasks.

5.6. Experimental Results

The experiments of clause segmentation and parsing subtasks are conducted separately, and experimental results are compared with those of previous works on discourse segmentation and parsing tasks, respectively.

5.6.1. Results of Clause Segmentation

We adapt the discourse parser DisCoDisCo [15] into the clause segmentation task and conduct an ablation study on different features, e.g., lemma, syntactic dependency, part-of-speech, and static word embedding fastText, to explore which contributes more to the performances. The experimental results of the clause segmentation task are reported in Table 10, as well as the performances of previous works on the discourse segmentation task for comparison. From the results, we have the following observations:

• Compared with the 98.4-100 IAA scores on clause segmentation mentioned in Section 3.3.3, the adapted DisCoDisCo model achieves satisfactory performances, i.e., 91.3 $F_1$ scores.
• In the ablation study of different embedded features, the static word embedding fastText, which gains a 4.9 $F_1$ score improvement, contributes the most in all features, although other features also have positive impacts on the performances.
• In the discourse segmentation task, the DisCoDisCo model outperforms the GumDrop model in the GUM and RST-DT datasets. Although GumDrop performs better than DisCoDisCo in the STAC dataset, Its performance sharply declined with 14.7 $F_1$ scores when removing extra gold features. Thus, we choose to apply the DisCoDisCo model to the clause segmentation subtask.
• Experimented with the same model DisCoDisCo, performances on clause segmentation are about 3-5 $F_1$ scores lower than on discourse segmentation, indicating that the performance gaps are attributed to the corpora. From the statistics in Table 6, an average of 3.1 clauses constitute a sentence in HCA-AMR2.0, more than that in GUM (2.3 EDUs), STAC (1.1 EDUs), and RST-DT (2.6 EDUs). Besides, only HCA-AMR2.0 contains a certain number of weblog data, which are less formal in English grammar and may cause more obstacles for the DisCoDisCo model.

5.6.2. Results of Clause Parsing

We employ the bottom-up and top-down discourse parsers in [16] as our clause parsing models and conduct experimental trials of the base version of different pretrained language models (PLMs, i.e., BERT [38], RoBERTa [39], SpanBERT [40], XLNet [41], and DeBERTa [42]) to obtain better performances. Table 11 demonstrates the main results of both bottom-up and top-down models on clause parsing, as well as that on discourse parsing for comparison. From the results, we have the following observations:

• Better performances are obtained in clause parsing than in discourse parsing by either top-down or bottom-up parsers with whatever PLMs.
• The best performances of clause parsing are obtained by the bottom-up parser with the pretrained DeBERTa, where the performance reaches up to 97.0 $F_1$ scores on Parseval-Span and falls to 87.7 $F_1$ scores on Parseval-Full.
• All the best performances in either clause parsing or discourse parsing are obtained by parsers with pretrained XLNet and DeBERTa, indicating that these two PLMs are more suitable for relation classification tasks than other PLMs.
• Experimented with a same model and a same pretrained language model, performances on clause parsing are about 6-9 Parseval-Full scores higher than that on discourse parsing, indicating that the performance gaps are attributed to the differences between corpora. RST-DT contains 18 classes of relations partitioned from 78 types of blurry rhetorical relations, while HCA has 18 types of distinguishable semantic relations, half of which are subdivided from Adverbial.

To obtain a clause parser with better performances, we conduct experimental trials with the large versions of PLMs, and the results are illustrated in Table 12. As can be observed from the results,

• All the parsers with the corresponding large PLMs perform better than that with the base PLMs.
• The bottom-up parser with DeBERTa-large achieves a better performance, with a 0.8 $F_1$ score improvement on Parseval-Full over the parser with DeBERTa-base.

6. Discussion

6.1. Potentialities of HCA

As discussed in Section 1, we demonstrated the same hierarchy of the HCA tree, the AMR graph, and the SDG of a complex sentence, indicating the potentialities of utilizing the structural information of the HCA tree to improve semantic parsing. Thus, we provide two case studies where simple transformation rules derived from the HCA tree can be applied to these two semantic parsing tasks.

6.1.1. Case Study for AMR Parsing

For AMR parsing, we employ the state-of-the-art AMR parser proposed in [1] to predict the AMR graph of the exemplified sentence in Section 1. As shown in Figure 8, two dotted red edges are missed by the parser when compared with the gold AMR, while two solid red edges are mistakenly predicted.

From the HCA tree given in Figure 1a, the clause ”I get very anxious” mapping to the subgraph $G_2$ and the clause ”but often the anxiety is so much” mapping to subgraph $G_4$ are coordinate and contrastive. Therefore, we provide a transformation rule:

1.

Transform the inter-clause relation BUT to an AMR node contrast-01 and two AMR edges directing to the root nodes of $G_2$ and $G_4$.

Meanwhile, the clause ”that I can not wait that long” mapping to subgraph $G_5$ is a resultative adverbial clause subordinated to the clause ”but often the anxiety is so much” mapping to the subgraph $G_4$. Note that the verb of the matrix clause is copular “is”, and the subordinate clause modifies the complement “much” in the matrix clause. Therefore, a new transformation rule can be derived:

2.

Transform the inter-clause relation resultative to an AMR node cause-01 and two AMR edges directing to the root node of $G_5$ and the node much in $G_4$.

With these two transformation rules, we can delete the solid red edges which are mistakenly predicted and add the dotted red edges which are missed.

6.1.2. Case Study for Semantic Dependency Parsing

For semantic dependency parsing, we employ the state-of-the-art AMR parser proposed in [2] to predict the semantic dependency graph (SDG) of the exemplified sentence in Section 1. As shown in Figure 9, three dotted red edges are missed by the parser when compared with the gold SDG.

As dicussed in Section 6.1.1, inter-clause relations among three clauses mapping to subgraph $G2$, $G4$, and $G5$ can be derived into the following transformation rules:

• Transform the inter-clause relation BUT to a dependency edge between root nodes (i.e., anxious and much) of $G_2$ and $G_4$.
• Transform the inter-clause relation resultative to a dependency edge between root nodes (i.e., much and can) of $G_4$ and $G_5$.

Moreover, the relative subordinate clause “which does sort of go away after 15-30 mins” mapping to subgraph $G_3$ modifies the complement “anxious” in the matrix clause “I get very anxious” mapping to subgraph $G_2$. Thus, a new transformation rule can be derived:

3.

Transform the inter-clause relation relative to a dependency edge between the root node go $G_3$ and the node anxious) of $G_2$.

With these three transformation rules, we can add the dotted red edges which are missed by the parser.

To sum up, we provide two case studies demonstrating the potentialities of HCA in semantic parsing, where transformation rules derived from the HCA tree are applied to modify two state-of-the-art parsers of the AMR parsing and semantic dependency parsing tasks.

6.2. Future Work

As demonstrated in the experimental results, we adapt discourse segmenter and discourse parser into our HCA subtasks (i.e., clause segmentation and parsing) and achieve satisfactory performances. However, there is still much room for improvement compared with IAA scores of manual annotation. Therefore, we aim to design better models for the HCA task in further research.

In the case studies, we derive some transformation rules from the HCA tree to modify the AMR graph and the SDG, which is explainable to humans but impractical. For this limitation, we aim to explore better ways of integrating the HCA structure in semantic parsing and more downstream NLP tasks.

7. Conclusions

In this paper, we propose a novel framework, hierarchical clause annotation (HCA), to segment complex sentences into clauses and capture inter-clause relations with strict definitions from linguistic research of clause hierarchy. We aim to explore the potentialities of integrating HCA structural features in semantic parsing with complex sentences, avoiding the deficiencies of previous works such as RST-parsing, SRRP, TS, SSD, etc. Following the HCA framework, we build up a large HCA corpus comprising $19,376$ English sentences from AMR 2.0. The annotation consists of silver data transformed from the constituency and syntactic dependency parse trees, and gold data annotated by experienced human annotators using a newly-created tool, ClausAnn. Moreover, we decompose HCA into two subtasks, i.e., clause segmentation and clause parsing, and provide effective baseline models for both subtasks to generate more HCA data.