LAI: Label Annotation Interaction based Representation Enhancement for End to End Relation Extraction

1. Introduction

As a core information extraction (IE) task, end-to-end relation extraction (E2ERE) can be split into named entity recognition (NER) subtask for entity identification and relation extraction (RE) subtask for capturing inter-entity relations from plain texts. As postulated by [1], E2ERE is challenging for its difficulty in capturing affluent correlations between entities and relations. IE research has traditionally converted NER and RE tasks into span-based tasks [1,2,3,4,5]. Though these methodologies have incrementally advanced model performance from various perspectives, they are still impeded by two pivotal limitations: overdetached of sub-tasks leads to insufficient information exchange between entitiy and relation, and the disparity in modeling strategies between entity and relation result in semantic gaps. In this paper, we mainly focus on enhancing semantic interaction during modeling process to achieve enhanced span representation for E2ERE.

To address the challenges above, prevailing researches mainly focuses on reorganizing the input or intermediate network layers of pre-train language model (PLM), attempting to enhance the semantic information of representation through the integration of specialized symbols or extrinsic prior knowledge. We roughly divided into three types (as shown in Figure 1): Vanilla based method is a straightforward approach to acquiring a given span representation by feeding raw text tokens series into pre-trained encoder. Marker based method inserts independent entity markers like $\left[\mathrm{M}\right],[\setminus \mathrm{M}]$ amidst text tokens to highlight the presence of entities, aims at attracting more model attention. Enumerate based method enumerates all posible entity candidates from plain text and then concatenate them after text tokens, entity tokens share the position ids with candidates as well. For these methods above, the distinguished LSTM-CRF [6] is a typical vanilla based sequence labeling method, and PURE [7] is a combination of marker based method and enumeration based method, which adopts marker based method during NER phase and enumerate based method for RE phase, that achieved SOTA performance. PL-Marker [8] is a typical enumeration based method that promoted the SOTA further.

In addition to the above three methods, what we develop in this paper can be classified as the fourth class named annotate & enumeration based method. It’s an novel semantic enhancement approach with external knowledge, inspired by external knowledge based aproaches [3,9,10]. We argue that a thorough comprehension of label semantics will significantly enhance the IE model abilities, what serves as a premise for our work.

As shown in Figure 1, our principal improvement over preceding methods lies in the insertion of external prior knowledge (i.e., label annotations) into the PLM input sequence, aiming to leverage PLM’s internal layers to enhance semantic interaction between label and text. This represents the first round semantic interaction in LAI-Net framework.

Unlike the lexicon adapter-based methods [11,12] and lattice-based methods [13,14,15], we manually expand the label information and embed it between text tokens and enumerated candidates then feed the series into a pre-trained model. We further enhance the representation by combining word vectors using downstream neural networks. Formally, the augmented representations derived from the aforementioned methods can be summarized as:

$$\begin{array}{cc}\hfill & {\mathbf{h}}_{\mathrm{span}}^V=f\left({\mathbf{h}}_{\mathrm{span}}^s;{\mathbf{h}}_{\mathrm{span}}^e\right)\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill & {\mathbf{h}}_{\mathrm{span}}^M=f\left({\mathbf{h}}_M;{\mathbf{h}}_{\setminus M}\right)\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill & {\mathbf{h}}_{\mathrm{span}}^E=f\left({\mathbf{h}}_{\mathrm{span}}^s;{\mathbf{h}}_{\mathrm{span}}^e;{\mathbf{h}}_M;{\mathbf{h}}_{\setminus M}\right)\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill & {\mathbf{h}}_{\mathrm{span}}^{A\&E}=f\left({\mathbf{h}}_{\mathrm{span}}^s;{\mathbf{h}}_{\mathrm{span}}^e;{\mathbf{h}}_M;{\mathbf{h}}_{\setminus M};{\mathbf{h}}_a^s;{\mathbf{h}}_a^e\right)\hfill \end{array}$$

(4)

where $f(·)$ is an tensor operator, equipped to execute series of operations including tensor addition, tensor multiplication, tensor concatenation, etc., or even could be a neural network. And superscripts $s,e$ denote the commencement and termination tokens of a span or annotation, while the subscripts $\mathrm{span},M,\setminus M$ represent the disparate token types.

Based on the first round semantic interaction, we further meticulously crafted with selectively designed downstream network layers. The main innovations and improvements lie in the following aspects:

• We explore a novel two-round semantic interaction approach for enhancing span representations, wherein the first round interaction reorganizes the PLM input with annotated information in what we term an annotation & enumeration-based method, and the second round interaction employs GCN built atop Gaussian Graph Generator modules to facilitate label semantic fusion.
• We conduct a coarse screening with a entity candidate filter to eliminate out spans that are clearly not real entities, which also promotes the saving of computing resources.
• Experiments demonstrates that our method, while slightly lagging behind current SOTA in NER performance, takes the lead in the downstream RE task, surpassing the current SOTA performance.

2. Related Work

Recently, academic interest in span representation enhancement has surged, providing a substantial impetus to E2ERE. Traditional neural network based methods often ignore non-local and non-sequential context information from input text [16], what is exactly GCNs [17,18] excel in. GCNs, what our discussion centered on, have been widely used to model the interaction between entities and relations in the text, and has been demonstrated as a typical and effective approach. GCN-based approaches typically leverage a predefined graph structure, which constructed from plain text, to facilitate information propagation among nodes, thus capturing text’s non-linear structure and enhancing NER and RE models’ capabilities to capture both global graph structures and representations of nodes and edges. Mostly GCN based methods [2,19,20,21,22,23] utilizes different approaches to define nodes (sentences, words, tokens, spans, labels, etc.), and these nodes can be connected by syntactic dependency edges [24,25,26,27], re-occurrence edges [16], co-occurrence edges [28,29], adjacent word edges [16,27,30], adjacent sentence edge [27], etc. And then perform convolution operations on the graph to facilitate the flow of information between nodes, which enables nodes to efficiently acquire both local and global information. This further refines node representations and downstream network performance. Building on these advances, our work also adopt a GCN based method to design the interaction between entities and relations.

3. Methodology

3.1. Task Definition

Given a sentence formularized as $S=\{w_1,w_2,\dots ,w_m\}$$=\{t_1,t_2,\dots ,t_n\}$ with m words (or n tokens, $n\ge m$ naturally), the goal of E2ERE task is to recognize a set of entity spans and relationships of entity pairs automatically, which can be written as $(e_i,r_{i,j},e_j)\in \mathcal{T}$. Every entity e, which attaches specific type (e.g. person (PER), organization (ORG)), is a sequence of tokens. Every relation $r_{i,j}$ represents the relationship between $e_i$ and $e_j$, and also attaches a specific type (e.g. organization affiliation relation (ORG-AFF))1. Formularly, we define the set of possible entity types and relation types as $\mathcal{E}$, $\mathcal{R}$ respectively.

3.2. First Round Semantic Interaction

During data processing, we concatenate text tokens, annotation tokens, and marker tokens sequentially to formulate a unified input sequence (the bottom element of Figure 2). The PLM encoder conducts the first round semantic interaction, then output the encoded representation, which is semantic amalgamation of the three types tokens.

Text Tokens    Our approach breaks down the words from raw text into text token sequences as part of the model input.

Annotation Tokens    Inspired by [3,31], we augment semantic information by manually annotating the entity (or relation) abbreviated label both in NER and RE phase. For example, the abbreviated entity type GPE can be annotated as "geography political entity", a fully-semantic unbroken phrase. Correspondingly, the abbreviated relation type ORG-AFF can be annotated as "organization affiliation". Each label is manually expanded to enrich semantic content and then tokenized into annotation tokens (highlighted by red rectangle in bottom of Figure 2), which are appended to the text tokens sequentially.

Marker Tokens    We enumerate all potential consecutive token sequences (i.e. entity candidates) not exceeding a predefined limitation of length c (with $c\le n$) within a sentence, labeling each with an entity type. If $c=2$, as shown in Figure 2, the set of all the possible spans from sentence "chalabi is the founder and leader of the iraqi national congress." can be written as $\mathbf{\Psi }=\{$"chalabi", "chalabi is", "is", "is the", "the founder", "founder", "founder and", "and", "and leader"$\dots \}$. The i-th span can be written as ${\mathrm{span}}_i=[{\mathrm{span}}_i^s,{\mathrm{span}}_i^e]$, where ${\mathrm{span}}_i^s$ and ${\mathrm{span}}_e^e$ are indicative of start and end position id of entity span respectively. Therefore, entity candidate series can also be written as $\mathbf{\Psi }=\left\{\right[0,0],[0,1],[1,1],[1,2],[2,2],[2,3],[3,3],[3,4],[4,$$4],[4,5\left]\right\}$ given position id perspective. Thus, the number of candidate for a sentence with m words:$\left|\mathbf{\Psi }\right|=m·c+(c-c^2)/2$.

In model input, we define a start marker (M) and an end marker ($\setminus M$), which form a pair of marker tokens, respectively represent the start and end of an entity span and are appended subsequent to annotation tokens. The start and end marker share the same position embedding with corresponding span’s start token and end token respectively, while keeping the position id of original text tokens unchanged. From an PLM encoder input perspective, every marker is a token element of tokens series, called marker token. As shown in Figure 2, entity chalabi is highlighted by light yellow bordered square, and its corresponding markers noted by a colorful non-bordered square with line frame differred in various entity labels (white means non-entity). In conclusion, the complete input sequence can be represented as Eq.(5), where $a_i$ is a token broken from label annotation, ${\mathrm{M}}_i^s,{\mathrm{M}}_i^e$ represent start and end marker token respectively.

$$\begin{array}{c}\hfill \tilde{S}=\{\left[\mathrm{CLS}\right],t_0,\dots ,t_{n-1},\left[\mathrm{SEP}\right],a_0,\dots ,a_{N-1},\left[\mathrm{SEP}\right],{\mathrm{M}}_0^s,\dots ,{\mathrm{M}}_{\left|\mathbf{\Psi }\right|-1}^s,{\mathrm{M}}_0^e,\dots ,{\mathrm{M}}_{\left|\mathbf{\Psi }\right|-1}^e\}\end{array}$$

(5)

Partial Attention    Although special tokens, such as [CLS] and [SEP] serve to isolate different types of tokens, there still exists semantic interference among them. The straightforward blend of annotation tokens and marker tokens with text tokens may disrupt semantic consistency of raw text. To mitigate this, we devise a partial attention mechanism, allowing selective semantic influence among the differ types of token. This mechanism can effectively control the information flow (could be regard as a kind of visibility) between different tokens, by adjusting the value of elements of the attention mechanism mask matrix. It suppresses the information interaction among tokens mutual invisible, while enhancing the information interaction among tokens mutual visible. Experimental results show that partial attention effectively improves model performance. See Appendix B for more detail information about partial attention.

3.3. Second Round Semantic Interaction

To refine semantic integration, we introduce second round semantic interaction, employing a semantic integrator that explicitly model interactions between entity candidates and label annotations. The semantic integrator consists of multiple GCN layers with randomly generated adjacency matrix, treats both entity spans and label annotations as nodes, and establishs connections between nodes through the construction of a graph $\mathcal{G}$. So that the interactions between nodes can be explicitly modeled. A GCN typically necessitates a manually predefined and fixed adjacency matrix to depict the inter-nodes connections. The fixed adjacency matrix fixes the perspective from which the model understands the semantics. However, it’s naturally to note that the inter-nodes connection cannot be predetermined accurately when considering our task. Otherwise, our task would be meaningless. Therefore, inspired by [22,32], we forgo a static adjacency matrix in favor of a multi-view graph, called $\mathbf{G}$aussian $\mathbf{G}$raph $\mathbf{G}$enerator based $\mathbf{G}$raph $\mathbf{C}$onvolutional $\mathbf{N}$etworks (${\mathrm{G}}^4\mathrm{CN}$). ${\mathrm{G}}^4\mathrm{CN}$ unlike vanilla GCN which takes a fixed adjacency matrix, it gives up fixing the edge weights and set the edge weights via trainable neural networks during the network initial stage, which allows model to assimilate semantic contexts from multiple perspectives.

First, we attach every node with a Gaussian distribution $\mathcal{N}(\mu ,\sigma )$, where both $\mu ,\sigma$ are generated by trainable neural networks, formulated as in Eq.(6), (7), and set the activation function $\varphi$ as the SoftPlus function, since the standard deviation of Gaussian distribution is bounded on $(0,+\infty )$. Then, simulate edge weight through coumpute the KL divergence $w_{ij}^e$ between two Gaussian distribution of node, where $w_{ij}^e=\mathrm{KL}\left(\mathcal{N}({\mu }_i,{\sigma }_i)\parallel \mathcal{N}({\mu }_j,{\sigma }_j)\right)$. So we can obtain a number of Gaussian distributions for the multi-view graph, each ${\mathcal{N}}_i$ here corresponds to a node representation $v_i$. Thus, a vanilla GCN’s value-fixed adjacency matrix $\mathbf{A}={\left(a_{ij}^e\right)}_{k\times k}$ can be modified to be a ${\mathrm{G}}^4\mathrm{CN}$’s value-varied adjacency matrix ${\mathbf{A}}^{\prime }={\left(w_{ij}^e\right)}_{k\times k}$, where k is number of nodes. The whole process can be formulated as Eq.(8), where ${\mathbf{H}}_{\mathrm{span}},{\mathbf{H}}_{\mathrm{anno}}^{\mathrm{ent}}$ are matrixs formed by concatenating multiple nodes (span or annotation) in constructed heterogeneous graph $\mathcal{G}$, and $\mathbf{GCN}(·)$ is vanilla GCN, see detail formulas in Appendix A.

$$\begin{array}{c}\hfill \{{\mu }_i^1,{\mu }_i^2,{\mu }_i^3,\dots ,{\mu }_i^N\}=g_{\theta }\left(v_i\right)\end{array}$$

(6)

$$\begin{array}{c}\hfill \{{\sigma }_i^1,{\sigma }_i^2,{\sigma }_i^3,\dots ,{\sigma }_i^N\}=\varphi \left(g_{\theta }^{\prime }\left(v_i\right)\right)\end{array}$$

(7)

$$\begin{array}{cc}\hfill {\tilde{\mathbf{H}}}_{\mathrm{span}}& ={\displaystyle \frac{1}{2}}\left({\mathbf{H}}_{\mathrm{span}}+\mathbf{GCN}\left({\mathbf{A}}_{\mathrm{ner}}^{\prime },\left[{\mathbf{H}}_{\mathrm{span}};{\mathbf{H}}_{\mathrm{anno}}^{\mathrm{ent}}\right]\right)\right)\hfill \end{array}$$

(8)

3.4. Name Entity Recognition

Span and Annotation Representation We extract the contextualized representations $\mathbf{h}$ for individual token s from PLM output, and naturally obtain the involved mathematical formulas for spans and annotatations as Eq.(9) and Eq.(10), where ${\mathbf{h}}_{\mathrm{anno}}\in {\mathbb{R}}^d,{\mathbf{h}}_{\mathrm{span}}\in {\mathbb{R}}^d$. And ${\mathbf{h}}_a^s,{\mathbf{h}}_a^e$ is embedding of first and last token of a certain type of label annotatation, respectively. ${\mathbf{h}}_t^s,{\mathbf{h}}_t^e$ is embedding of first token and last token of a entity candidate respectively, and ${\mathbf{h}}_M^s,{\mathbf{h}}_M^e$ indicates the embedding of start and end token of marker respectively. Linear layer $\mathbf{FC}$ used to harmonize dimensional space.

$$\begin{array}{cc}\hfill {\mathbf{h}}_{\mathrm{anno}}& ={\mathbf{FC}}_a\left([{\mathbf{h}}_a^s;{\mathbf{h}}_a^e]\right)\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill {\mathbf{h}}_{\mathrm{span}}& ={\mathbf{FC}}_{\mathrm{span}}\left([{\mathbf{h}}_{\mathrm{span}}^s;{\mathbf{h}}_{\mathrm{span}}^e;{\mathbf{h}}_M^s;{\mathbf{h}}_M^e]\right)\hfill \end{array}$$

(10)

Entity Candidates Filter Before the development of the entity candidates filter module, we attempted to train the model, but it ended in failure. The reasons can be summarized as: 1) During the initialization phase of training, model parameters are randomly assigned values, leading to lots of candidate entities being randomly predicted as real entities in early phase. This further causes the adjacency matrix of the graph neural network to become excessively large in scale, resulting in extremely slow network training and significant resource consumption. 2) Among the numerous candidate entities enumerated, positive sample entities are extremely few while negative sample entities are abundant. This easily induces the model to tend towards classifying all entities as non-entities to achieve the minimum loss value, which is not the desired outcome. To overcome the aforementioned issues, we devise a binary classifier acts as a entity filter, performing coarse screening for all enumerated entities by discarding non-genuine entities, thus optimizing subsequent predictions.

As for loss function, the primary aim of entity filter is to maximize the likelihood function, what drove us follow [2] to adopt likelihood loss function as Eq.(11), where ${\mathbf{\Psi }}_g\subseteq \mathbf{\Psi }$ indicates a set of real entity spans. In addition to intuitive time consumption optimization, experimental results indicate that entity filter successfully alleviates model weakening engendered by negative samples and enhances overall performance.

Span Classifier We conduct span representations classification through a linear classifier, utilizing cross-entropy loss to direct the learning process. The combined loss function ${\mathcal{L}}_{\mathrm{ner}}={\mathcal{L}}_{\mathrm{filter}}+{\mathcal{L}}_{\mathrm{span}}$ is optimized during training, using ${\mathcal{L}}_{\mathrm{span}}$ from Eq.(11)-(12), with dropout layers for regularization.

In addition, [8] had proved that packing a series of related spans into a training instance can promote the NER model performance, that naturally prompt us to follow the effective measures when reorganize input.

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{filter}}& =-{\displaystyle \frac{1}{\left|\mathbf{\Psi }\right|}}\sum _{i=1}^{\left|\mathbf{\Psi }\right|}logP({\mathrm{span}}_i\in {\mathbf{\Psi }}_g|{\mathrm{span}}_i\in \mathbf{\Psi })\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{span}}& =-{\displaystyle \frac{1}{|{\mathbf{\Psi }}_g|}}{\sum }_{{\mathbf{\Psi }}_g}log{P}_{\mathrm{span}}\hfill \end{array}$$

(12)

3.5. Relation Extraction

Subject marker In RE phase, we design our Annotate & Enumeration Based method (as demonstrated in Figure 1) to acquire the enhanced representation. Concretely, we adopt the marker based approach (shown in Figure 1), and insert a pair of subject markers, called solid markers by [8], into left and right of subject entity, and enumerated object candidate spans following on the heels of annotatation tokens to extract relations involving the subject entity.

Entity pairs Representation We match subject and object representations up pairwise to obtain a series of entity pairs (${\mathbf{h}}_{\mathrm{pair}}=[{\mathbf{h}}_{\mathrm{subj}};{\mathbf{h}}_{\mathrm{obj}}]$). And the label semantic confused pair representation formulas can be written as Eq.(13), where ${\mathbf{H}}_{\mathrm{pair}},{\mathbf{H}}_{\mathrm{anno}}^{\mathrm{rel}}$ are matrixs formed by concatenating entity pair or relation label annotation representations.

$$\begin{array}{c}\hfill {\tilde{\mathbf{H}}}_{\mathrm{pair}}={\displaystyle \frac{1}{2}}\left({\mathbf{H}}_{\mathrm{pair}}+\mathbf{GCN}\left({\mathbf{A}}_{\mathrm{re}}^{\prime },\left[{\mathbf{H}}_{\mathrm{pair}};{\mathbf{H}}_{\mathrm{anno}}^{\mathrm{rel}}\right]\right)\right)\end{array}$$

(13)

The loss function of RE phase is the cross-entropy loss.

4. Experiments

Datasets We utilize 3 widely used standard corpora: 1) ACE05 spans various domains (newswire, online forums), and contains 7 entity types and 6 relation types between entities. 2) SciERC [33] is a scientific dataset built from AI conference/workshop proceedings across four communities. It includes 7 entity types and 7 relation types. 3) ADE [34] consists of 4,272 sentences and 6,821 relations extracted from medical reports.

Metrics The model with best F1 performance on test set will be selected on a fixed number of epochs. Both micro and macro average metrics are used to evaluate the model performance, former for ACE05/SciERC and latter for ADE. For NER task, an entity prediction is correct if and only if its type and boundaries both match with those of a gold entity. For RE task, a relation prediction is considered correct if its relation type and the boundaries of the two entities match with those in the gold data. We also report the strict relation F1 (denoted RE+), where a relation prediction is considered correct if its relation type as well as the boundaries and types of the two entities all match with those in the gold data. We show detailed experimental settings in Appendix B.

4.1. Main Results

To ensure the credibility of experimental results, we conducted multiple experiments with different random seeds on all datasets (relevant settings are detailed in Appendix B.1), and the stability of running are presented in Appendix B.10.

4.1.1. Results Against Horizontal Comparison

Table 1 presents a horizontal comparison with baselines, focusing on comparing excellent models developed in recent years, including several previous SOTA models (detail in Appendix B.2). In terms of NER task, our method is on par with the current SOTA, with the discrepancy across three datasets ranging from 0.39% to 0.1%, which achieves sub-optimal performance. Starting from a slightly lower NER performance compared to the SOTA, our method still achieved a performance gain of 2-10% on relation F1 and strict relation F1, consistently outperforming all selected baselines, even with the error propagation between NER and RE task. For example, on ACE05, our method exhibits a 0.24% disadvantage in NER phase, but takes the lead in the downstream RE phase, surpassing the current SOTA by 0.41%, setting a new SOTA record. Similarly, our method, despite having a 0.39% and 0.1% disadvantage in NER performance respectively on SciERC and ADE, achieved performance improvements of 10% and 0.52%. All these improvements demonstrate that two-rounds semantic interactions indeed further utilizes the predicted entities from the NER phase, significantly improving the performance of relation recognition with slightly compromising NER performance.

4.1.2. Results Against Significant Hyperparameters

Table 2 delineates the impact of varying significant hyperparameter of ${\mathrm{G}}^4\mathrm{CN}$ on the performance of LAI-Net. For the number of GCN layers, we consider a range from 0 to 5, with zero indicating the absence of GCN for semantic interaction and more layers corresponding to increased communication times among nodes within the graph. As for the number of attention heads, we opt for values of 1, 2, 3, 4, and 6. The deliberate exclusion of the value 5 is attributed to the fact that the encoder’s hidden dimension is not divisible by 5, a constraint inherent to the multi-head attention mechanism.

The table permits an intuitive observation that: (1) an increase in the GCN layers number does not linearly translate to enhanced performance; an excessive GCN layers number can exert a deleterious effect on the model, with the more optimal layers number identified as either 1 or 2; (2) concerning the number of attention heads, the more optimal solution exhibits some variation across different datasets, yet it is unequivocally clear that neither an excessively high (e.g. 6) nor a disproportionately low (e.g. 1) number of attention heads can fully capitalize on the GCN’s capabilities.

4.2. Inference Speed

There is justification to assess whether the adoption of GCNs to enhance F1 performance has made trade-offs respect to inference efficiency, for the reason that GCNs have certain efficiency disadvantages typically. Hence, We conducted a comparison between PL-Marker and LAI-Net in terms of inference speed, that experiment was evaluated on a GeForce RTX 3090 24GB GPU with a evaluate batch size of 16. We use Bert-base model for ACE05 and SciBert for SciERC.

As shwon in Table 3, we indeed sacrificed varying degrees of inference efficiency in exchange for varying degrees of improvement in F1 performance. Overall, the greater the sacrifice, the greater the benefit.

4.3. Ablation Study

4.3.1. Ablations Against Entity Filter

Considering the binomial surge in candidate entity quantity accompanying increased entity length in NER phase, the relatively paltry positive examples can be easily overwhelmed by a vast array of negative samples, thereby instilling the network a strong propensity to categorize all samples as negative, leading to difficulty in accurately identifying genuine entities. To mitigate this, LAI-Net incorporates a deliberately inserted filter prior to the entity classifier to preliminarily sieve out spans that are clearly non-entity. To validate whether said filter genuinely facilitates the NER process, we devised associated ablation experiments. As depicted in the two rightmost columns of Table 2, the presence of the filter effectively improves NER performance, with advantages most conspicuous on the ACE05 dataset (surpassing no-filter models by 1.74% in terms of F1 scores) and improvements of varying degrees are also observed on other datasets.

4.3.2. Ablation Against Two Rounds of Interaction

We conducted ablation experiment specifically targeting the two-round semantic interaction. Drawing upon the outcomes of the experiment, we can directly evaluate the extent to which our devised dual semantic interaction genuinely augments the model’s performance. What should be noted is that the second round interaction is built upon the annotated label information (that’s what the first round interaction do), so the second round interaction ceases to exist once the label annotation information is no longer injected. However, the existence of the second round interaction does not affect the first round interaction. As shown in Table 4, no matter which round of semantic interaction is eliminated, it invariably leads to adverse effects of varying magnitudes on the newly SOTA we have developed, encompassing the precision, recall, and F1-score metrics. From the perspective of performance degradation, the detrimental effect of eliminating the second round interaction on the basis of LAI-Net is significantly more pronounced than the impact of further removing the first round of interaction on the basis of w/o 2nd round interaction, with the discrepancy peaking at over 21-fold (${\displaystyle \frac{4.67\%}{4.89\%-4.67\%}}$ for RE+ in ACE05).

4.3.3. Ablations Against Attention Mask Matrix

In order to evaluate the efficacy of partial attention matrices, we selected three distinct attention mask matrices for ablation (detail explaned in Appendix B.11), namely: the full attention matrix, wherein all tokens are mutually visible; the anno-token visible attention matrix, where annotation tokens and text tokens are intervisible; and the anno-token invisible attention matrix, with annotation tokens and text tokens also being intervisible. Irrespective of the attention masking matrix employed, tokens of the same type remain mutually visible during the computation of attention scores.

As Table 5 elucidates, across all three datasets, the anno-token invisible attention matrix exhibits a markedly superior performance with strict F1 metrics compared to the other attention matrice types. The anno-token visible attention matrix comes in second place, with its largest deficit compared to the top-performing technique capping out at 4.7% across the varied tasks encapsulated within the trio of benchmarks. Meanwhile, the fully attention matrix acts a performance lagging behind the peak achieved score on each respective dataset (max up to 6.28%), indicative of appreciably inferior capabilities amongst the range of workloads tested.

We attempt to analyze the underlying reasons, which may lie in the following points: (1) the mutual visibility mechanism between annotation tokens and text tokens establishes a conduit for semantic communication, thereby enhancing the semantic richness of the embeddings for both annotations and text. On the one hand, this enables annotation tokens to fully comprehend background information of text, which directly impacts their high-dimensional semantic representation. On the other hand, text tokens can also fully integrate the information from annotations, which directly facilitates the screening and categorization of candidate entities. (2) conversely, the full attention matrix allows for the intermingling of semantic information among annotation tokens, text tokens, and entity candidate tokens, which may lead to an excessive infusion of additional semantics, resulting in semantic redundancy, and ultimately causing a decline in model performance.

5. Conclusion

Faced with the challenges posed by sub-task excessive separation and modeling gaps between entities and relations, we propose LAI-Net, which leverages a two-phase semantic interaction and attains SOTA performance in both NER and RE tasks. The key novelty is the multi-phase semantic interaction framework to effectively inject external knowledge and unify representations for entities and relations. We manually annotate label abbreviations to fully-semantic unbroken phrase for expand lexical semantic, and then leverages ${\mathrm{G}}^4\mathrm{CN}$ to fuse information from latent multi-view. And an entity candidate filter is embed to coarse screening of candidate spans for NER. Experiments on several datasets show that our LAI-Net allows high-level semantic information flow and facilitates the E2ERE task, successfully achieves the SOTA performance.

Appendix B.1. Datasets and Preprocess

We selected two standard corpus (ACE05, SciERC) in terms of E2ERE task.

1) ACE052 is collected from a variety of domains (such as newswire and online forums). It includes 7 entity types, and include 6 relation types between entities. For data processing, we use the same entity and relation types, data splits3, and pre-processing as [38,39] (351 training, 80 development and 80 testing).

2) SciERC [33] is a scientific-oriented dataset, which is built from 12 AI conference/workshop proceedings in four AI communities, and includs 7 entity types and 7 relation types.

In terms of experiment, we run our model 10 times with different random seeds, and report averaged results of all the runs.

Appendix B.2. Chosen of Baselines

To ensure fairness and comprehensiveness in performance comparison as much as possible, we have established several principles for selecting baselines The first is to select well-recognized baselines from those cited in the SOTA works published in recent years. The second principle is to cover models of encoder-only, decoder-only and encoder-decoder architectures as comprehensively as possible, and what the priority is given to models that have similar or comparable parameter scales to our PLM during the selection process for encoder-only based baselines. Thirdly, one or more datasets should be the same as those used in our work. Lastly, we tend to select methods that formally published in relatively authoritative conferences or journals, which is not mandatory.

Appendix B.3. PLMs and Hardware Devices

For fair comparison with previous works, we employ bert-base-uncased [40] as the encoders for ACE05, and use the in-domain scibert-scivocab-uncased4 [41] as the encoder for SciERC, and all the experiments are executed using three GeForce RTX 3090 24GB GPUs.

Appendix B.4. Optimizer and Learning Rate Settings

We use AdamW optimizer during training. We set the learning rate as 4e-4 for both NER and RE task. We had tried to set different learning rate for different layers, and experiment results show that it’s useless.

Appendix B.5. Maximum Length Settings

We respectively set the maximum length of reorganized sentence C as 150, 150 on ACE05, SciERC. As enumerating possible spans, we set the maximum span length L as 8 for all datasets, and limit the number of entity candidates as 220 for every train/eval sample.

Appendix B.6. Batch Size and Epoch Settings

In NER phase, we respectively set batch size as 16 per GPU for SciERC, 20 per GPU for ACE05. And in RE phase, we set the batch size as 40 per GPU for SciERC, 50 per GPU for ACE05. We set the epoch as 80 for all Datasets in NER phase, and 60 for all dataset in RE phase.

Appendix B.7. Avoidance the Negative Influence of Annotation

Considering that our method increases the length of the training instances, which could impact the model’s performance, we strive to alleviate this negative impact as much as possible. Firstly, avoid using overly long annotation information, but simply restore label abbreviations(such as restoring the PER tag to "person" and the PER-ORG tag to "person-organization"), and the length of the annotations used in the experiments does not exceed 10 words. Secondly, using adjusted attention mask matrices to minimize semantic confusion caused by the insertion of annotations.

Figure A2. Diagrams of three types attention mask matrix with an assumption that there is two entity candidates.

Figure A2. Diagrams of three types attention mask matrix with an assumption that there is two entity candidates.

Appendix B.8. Cold Start Settings for NER

It should be noted that, due to the enumeration of a large number of non-entity spans as negative samples in NER pahse, which extremely likely to lead to unable-convergence train, or the phenomenon that the model directly predicts all spans as non-entities. Therefore, we set up a specific cold start process for training. In the first half of training, we use the true labels as the filter result for next entity classification in order to calculate the loss to update the parameters. This guides the model during early learning before exposing it to negatively labeled non-entity spans. In actual training, we set the cold boot epoch number as 15, 40 for ACE05, SciERC respectively. Moreover, according to unpublished experimental results, experiments without cold start configuration were utterly incapable of acquiring any knowledge whatsoever, with all performance metrics equaling zero throughout the training phase.

Appendix B.9. Symmetry of Relation for RE

We formulate the directed relation as $r_{i,j}$, with the subject entity $e_i$ always pointing to object entity $e_j$. Therefore, a triplet with positive relation canbe written as $e_i\to r_{i,j}\to e_j$, and its reverse formula is $e_j\to r_{j,i}\to e_i$. $r_{i,j},r_{j,i}$ may refer to different relation types. In RE phase, we consider the subject be left and the object be right by default. Either ($e_i\to r_{i,j}\to e_j$) or ($e_j\to r_{j,i}\to e_i$) will be predicted if it’s a really relaiton. Only when they are both predicted to be true (that is, the probability value is greater than the threshold), the triplet ($e_i,r_{i,j},e_j$) and triplet ($e_j,r_{j,i},e_i$) will be established.

Appendix B.10. Stability of Training

To ensure the reliability of the experimental data, we repeated the experiments multiple times for each hyperparameter combination, each time initializing the network with a different random seed, and then calculated the mean and standard deviation of the experimental results, as shown in the Figure A1. The first row of chart in the Figure A1 represents the fluctuations in NER and RE performance across three datasets for different numbers of GCN layers, The second row of chart in the Figure A1 represents the fluctuations in NER and RE performance across three datasets for different numbers of attention heads.

Appendix B.11. Parital Attention Mask

Obviously, the series of annotation tokens and marker tokens manually attached after the text tokens does not form a coherent and semantically complete sentence. It inevitably affects the semantic construction from text tokens, and impair the representational ability of word vectors during the PLM encoding process.

To address this potential issue, we adopt a specialized partial attention mechanism to selectively mitigate or enhance the semantic impact of tokens from differed types. In details, by adjusting the value of elements of the attention mechanism mask matrix, we can control the visibility among three types of tokens.

Partial attention can effectively control the information flow between different tokens. It suppresses the information interaction between text and annotations (i.e. invisible) while enhancing the information interaction between text and candidates (i.e. visible).

We show three different masking matrices in Figure A2, for which we conduct some ablation experiments. In conjunction with Figure A2, we further elucidate the meaning of the discrete elements within mask matrix. Train our gaze upon the first row of Figure A2 (a), which delineates the tokens discernible by the text token. The pale green region signifies it can see corresponding tokens, the white space those it cannot see, and the faint yellow elements the marker tokens within its purview. Notably, the two faint yellow squares on the left denote the start marker tokens, while the two on the right denote the end marker tokens.