Linguistic-aided Sentence Representation Learning

1. Introduction

In recent years, the field of natural language processing has experienced an unprecedented interest in the area of pre-trained language models. The focus of contemporary research in this domain has been categorized into three distinct yet interconnected streams: (1) The application of context-aware word representations in a variety of downstream tasks, as evidenced in works like [1,2,3]; (2) A deep dive into understanding the extensive knowledge that these models garner from vast text corpora [6,7]; (3) The enhancement of word representations by incorporating external, particularly syntactic, information [8,9]. In line with these developments, our research presents the innovative Syntactic Enhancement Network (SEN), a groundbreaking framework that integrates syntactic structures into pre-trained language models with finesse. Unlike conventional models, SEN adopts a unique tripartite approach to dependency syntax, ensuring compatibility with both RNN and Transformer-based models. This novel method stands out for its efficiency and compatibility with mini-batch training methodologies, setting it apart from traditional dependency parser tree models [10,11,12].

Our exploration of SEN’s capabilities is conducted through two distinct sentence-level tasks. The first task, sentence completion, involves choosing the most apt word or phrase from a selection to complete a sentence effectively. For this, we utilize the well-established Microsoft Research Sentence Completion Challenge (MSCC) [13], which is composed of sentences from the classic Sherlock Holmes novels, each with a single missing word and five potential choices. Prior research [15,16] has validated the importance of pre-trained models and syntactic information in this particular context. The second task is centered around relation extraction, with the objective of classifying pairs of entities in sentences. For this, we use the renowned SemEval2013 DDI Extraction challenge dataset, a benchmark dataset for biological relation extraction, again highlighting the crucial role of syntactic information in this area [4,11,12,17,18,19,20].

Furthermore, we are excited to introduce the English Sentence Gap-Filling (ESG) dataset, an innovative resource specifically designed for sentence completion tasks, derived from a range of standardized English examinations. This dataset broadens the scope of question types and grammatical phenomena beyond what is available in the MSCC, as demonstrated in Table 1. A comprehensive exploration of the ESG dataset is provided in Section 4.1.

The contributions of this paper are significant and varied:

• We introduce the Syntactic Enhancement Network (SEN), a pioneering approach that skillfully weaves complex syntactic structures into the fabric of pre-trained language models.
• We rigorously test the SEN across two pivotal sentence-level tasks, employing diverse experimental designs to thoroughly evaluate its performance.
• We develop the extensive English Sentence Gap-Filling dataset, tailored for sentence completion in English examinations, poised to become a benchmark in the field.

2. Related Work

Recent advancements in natural language processing have been significantly influenced by the development of pre-trained language models, notably ELMO and BERT, as introduced by Peters et al. [21] and Devin et al. [22] respectively. ELMO’s innovation lies in its utilization of a BiLSTM trained with a dual language model objective, while BERT, diverging from BiLSTM, leverages a bidirectional Transformer encoder as conceptualized by Vaswani et al. [23] for predicting words in a masked context. Building upon these foundational works, subsequent research has explored various applications and enhancements of these models. For instance, Sun et al. [1] and Nogueira et al. [3] applied the BERT model to tasks like sentiment analysis and passage ranking, achieving groundbreaking results. Additionally, studies by Liu et al. [27] and Tenney et al. [6] demonstrated the capability of ELMO and BERT in capturing both syntactic and semantic nuances in language. Furthermore, Zhang et al. [8] and Sun et al. [28] proposed the integration of knowledge graphs and entity information with contextualized embeddings to further refine language representations. In the domain-specific context, Lee et al. [29] and Beltagy et al. [30] showed how fine-tuning BERT with specialized data can significantly enhance performance in specific fields.

Focusing on the sentence completion task, Zweig et al. [31] introduced baseline models based on n-gram language models and latent semantic analysis, achieving accuracies of 39% and 49% respectively. Mikolov et al. [15] enhanced these results by integrating skip-gram with an RNN-based language model, leading to a 58.9% accuracy in the MSCC. Park et al. [35] further improved upon these results with a bidirectional word-level RNN. Beyond architectural changes, there has been a significant interest in incorporating syntactic information into models. For instance, Piotr Mirowski et al. [36] achieved a 10% absolute improvement over previous RNN-based baselines by integrating syntactic information into the RNN language model. Zhang et al. [10] employed a tree-structured LSTM model to infuse dependency path information into the model.

In the context of the DDI2013 Extraction task, Zhang et al. [12] combined shortest dependent path information and sentence sequences using a hierarchical RNN network, surpassing other state-of-the-art methods. Miwa et al. [11] adopted a bidirectional Tree-LSTM to capture substructural information within dependency trees, leading to significant error reductions in F1-score. From these preceding studies on sentence completion and relation extraction tasks, it is evident that models leveraging syntactic information predominantly rely on tree structures. In our proposed Syntactic Enhancement Network (SEN), we linearize this tree structure into a triple form and utilize expanded syntactic information for improved sentence representations. This approach demonstrates versatility and effectiveness in both sentence completion tasks and relation extraction tasks.

3. The Proposed Framework

In this section, we elaborate on our Syntactic Enhancement Network (SEN) model. The architecture of SEN, encompasses three primary components: (1) Encoding Layer: employing a pre-training language model to generate word embeddings; (2) Dependency Syntax Integration Layer: merging syntactic information with pre-trained word embeddings, followed by a BiLSTM to yield refined sentence representations; (3) Output Layer: determining the most probable answer or label based on the final semantic representations of sentences. We represent a sentence $S=\left[w_1,w_2,\cdots ,w_n\right]$ as a sequence of n words.

3.1. Encoding Layer

The encoding layer’s primary function is to transform each word in the sentence into a fixed-length vector using pre-trained word embeddings. We initially utilize a bidirectional LSTM network (Hochreiter and Schmidhuber [40]), akin to ELMO, and refer to it as biLM. Subsequently, we apply the BERT pre-trained model based on Transformer architecture to acquire a distinct set of contextualized word embeddings. Specifically, for biLM, the forward phase calculates the probability of sentence by modeling the likelihood of word $w_k$ given $\left(w_1,\cdots ,w_{k-1}\right)$, while the backward phase processes the sentence in reverse. The joint objective maximizes the log likelihood of both passes as follows:

$$\begin{array}{c}p\left(w_1,w_2,\dots ,w_n\right)=\prod _{k=1}^{n}p\left(w_k|w_1,w_2,\dots ,w_{k-1}\right)\hfill \\ p\left(w_1,w_2,\dots ,w_n\right)=\prod _{k=1}^{n}p\left(w_k|w_{k+1},w_{k+2},\dots ,w_n\right)\hfill \end{array}$$

(1)

$$\begin{array}{c}\sum _{k=1}^n\left(logp\left(w_k|w_1,\dots ,w_{k-1};{\overleftarrow{\theta }}_{LSTM}\right)\right.\hfill \\ +logp\left(w_k|w_{k+1},\dots ,w_N;{\overrightarrow{\theta }}_{LSTM}\right))\hfill \end{array}$$

(2)

For BERT, given its unique architecture, a modified language modeling objective is adopted, masking some tokens and predicting their likelihood through a softmax layer. During training, BERT’s parameters are updated while biLM’s remain fixed.

3.2. Dependency Syntax Integration Layer

This subsection describes the fusion of explicit syntax information with pre-trained word embeddings. Dependency syntax, illustrating grammatical relations between words, is traditionally represented by binary trees. In this structure, each word relies on a unique head. These head-dependent relations, essential for applications like coreference resolution and information extraction, are commonly depicted in a triple form: (dependent, relation, head). For example, (fruit, nsubj, is) suggests that fruit modifies is with a nsubj relation. We transform these dependency trees into an expanded form.

Using this triple form, syntax information is effortlessly integrated with contextualized word embeddings to create novel sentence representations. We employ two fusion functions to map a triple into a continuous vector, defined as:

$$x_i=g\left(w_i,r_i,w_i^H\right)$$

(3)

Here, $w_i$ and $w_i^H\in {\mathbb{R}}^d$, with $w_i^H$ being the head of $w_i$ and d the embedding dimension. $r_i$ is a low-dimensional vector representing different dependency relations. The fusion function $g(·)$ has two implementations: vector concatenation and a gating mechanism controlling information flow:

$$g\left(w_i,r_i,w_i^H\right)=w_i\oplus r_i\oplus w_i^H$$

(4)

$$g\left(w_i,r_i,w_i^H\right)=w_i+\sigma \left(r_i\right)\otimes w_i^H$$

(5)

The processed sentential sequence is denoted as $S^p=[x_1,x_2,\dots ,x_n]$.

$S^p$ is then inputted into a BiLSTM network. The following equations detail the calculation of each gate and memory cell unit in the BiLSTM:

$$\begin{array}{c}{h}_i=\left[\overrightarrow{h_t}\oplus \overleftarrow{h_t}\right]\hfill \\ h_t=o_{t}tanh\left(c_t\right)\hfill \\ f_t=\sigma \left(W_f·\left[C_{t-1},h_{t-1},x_t\right]+b_f\right)\hfill \\ i_t=\sigma \left(W_i·\left[C_{t-1},h_{t-1},x_t\right]+b_i\right)\hfill \\ g_t=tanh\left(W_{xc}·\left[C_{t-1},h_{t-1},x_t\right]+b_c\right)\hfill \\ c_t=i_t{g}_t+f_t{c}_{t-1}\hfill \\ o_t=\sigma \left(W_o·\left[C_t,h_{t-1},x_t\right]+b_o\right)\hfill \end{array}$$

(6)

The final sentence representation $S^f=\left[x_1^f,x_2^f,\dots ,x_n^f\right]$ combines the forward and backward output at the last position.

3.3. Output Layer

The output layer utilizes a fully connected feed-forward network for final predictions. However, the objective functions differ for sentence completion and relation extraction tasks. In sentence completion, we employ a pairwise approach to rank answer options. The loss function is defined as:

$$loss=\sum _{i=1,2,3}^{N}max(0,-f\left(S_{true}^f\right)+f\left(S_{false\_i}^f\right)+margin)$$

(7)

$f(·)$ computes scores for each candidate sentence, selecting the highest-scoring completion. Here, $u\in {\mathbb{R}}^k$ is a learnable parameter, and margin is a threshold to increase score separation between $S_{true}^f$ and $S_{false\_i}^f$.

For DDI extraction, we use negative log-likelihood as our loss function:

$$loss=-logp\left(y_i|S_{p_i}^f\right)=-log\left(\frac{exp\left(y_i\right)}{{\sum }_j^{5}exp\left(y_j\right)}\right)$$

(8)

where $S_{p_i}^f$ is an entity pair in sentence $S^f$, and $y_i$ is the relation class of $p_i$. The softmax function computes the probability that an entity pair $S_{p_i}^f$ belongs to class $y_i$.

4. Experiments

4.1. Settings

ESG dataset We sourced our English Sentence Gap-Filling (ESG) dataset primarily from high school English multiple-choice questions found on online educational platforms. Prior to training, the dataset underwent preprocessing as follows: Initially, to avoid redundancy, questions with an editing distance below 8 were filtered out. Then, utilizing the Stanford Parser, we implemented part-of-speech tagging and dependency parsing. The final dataset, comprising 62,834 questions, was divided into training and test sets in a 9:1 ratio.

DDI dataset The DDIExtraction 2013 dataset, derived from the DrugBank and DrugMedline databases, is accessible at https://www.cs.york.ac.uk/semeval-2013/task9/. This task revolves around correctly classifying drug-drug interaction relations into five categories: advice, effect, mechanism, int, and negative. Table 2 offers a detailed breakdown of this data.

4.2. Configurations

In the sentence completion task, we employed the PyTorch implementation of BERT with models provided by Google.1 Given BERT’s use of BPE for tokenization, which splits some words into sub-words, we used the first word piece embedding to maintain consistent sequence lengths with the original sentences. We chose the bert-base-cased model (${\mathrm{BERT}}_{{\mathrm{base}}_{\mathrm{u}}\mathrm{ncased}}$), with a maximum sequence length of 128, a mini-batch size of 16, and BertAdam as the optimizer. The learning rate was set to $\alpha$ = $2e-5$ with warmup = $0.05$. The relation vector size was fixed at 200 and the hidden size of the BiLSTM at 768. The model was trained for 6 epochs, with evaluations after each epoch. Additionally, we gathered a 26G unannotated text corpus from Wikipedia, Gigaword, and English learning websites to train the biLM model.

For the DDI task, we opted for the BioBERT [29] model, a domain-adaptive version of BERT pre-trained on biomedical corpora. We set the sequence length at 350 and the minibatch size at 32. The learning rate and warmup were adjusted to $3e-5$ and $0.1$, respectively. The BiLSTM’s hidden size was 512, with other hyperparameters mirroring those of the sentence completion task. We replaced entities in the sentence with "drug1" and "drug2" for those requiring prediction, and "drug0" for others. This approach, as shown in prior studies, helps models better grasp semantic entity relationships.

4.3. Results and Analysis

Main results The results on the ESG dataset are summarized in Table 3. Our baseline model, using randomly initialized word vectors and a BiLSTM layer, served as the comparison point. In the table, models marked as ’biLM’ and ’${\mathrm{BERT}}_{{\mathrm{base}}_{\mathrm{u}}\mathrm{ncased}}$’ were trained without the syntax integration layer. Accuracy was the chosen metric for evaluation. As evidenced, the best performance (91.2%) was achieved by the SEN model utilizing BERT. Pre-trained language models substantially enhanced performance, and the SEN model with BERT outperformed the one with biLM. Syntax integration on top of pre-trained LMs improved accuracy by 2% and 0.6% for biLM-based and BERT-based models, respectively. Furthermore, fusion methods (equations 4 and 5) had varying impacts depending on the model. For instance, the gate mechanism proved more effective for the biLM+SEN model, while the reverse held true for the ’${\mathrm{BERT}}_{{\mathrm{base}}_{\mathrm{u}}\mathrm{ncased}}$+SEN’ model.

Inspired by domain-specific fine-tuning approaches like SciBERT[30] and BioBERT[29], we introduced a BERT variant, ’${\mathrm{BERT}}_{\mathrm{finetune}}$’, pre-trained on an additional 1.6G corpus containing nuanced words and phrases from error samples. This fine-tuned model exhibited stable performance improvements on the ESG dataset.

Error Analysis and Visualization An error analysis revealed a notable decrease in error rates for verb differentiation, tense consistency, and phrase collocation, attributing this improvement to the integration of syntax information and the fine-tuning step. Additionally, we utilized Bertviz to visualize attention patterns in the ’${\mathrm{BERT}}_{{\mathrm{base}}_{\mathrm{u}}\mathrm{ncased}}$’ and ’${\mathrm{BERT}}_{{\mathrm{base}}_{\mathrm{u}}\mathrm{ncased}}$+SEN’ models, highlighting the impact of syntax integration on model learning.

DDI Task Evaluation We benchmarked our model against recent DDI task methods, focusing on precision, recall, and micro F1-score metrics. The results, detailed in Table 5, demonstrate that our model not only achieved competitive performance but also benefited from the integration of syntactic expansion information.

Ablation Study We performed an ablation study on the inputs of the dependency fusion layer by varying the composition of the triple in equation 4. The results in Table 6 indicate that the dependent relation vector significantly contributes to model performance improvements.

5. Conclusion and Future Exploration

This study was dedicated to exploring the synergistic integration of syntactic information with sophisticated pre-trained language models. To achieve this, we innovatively developed a triplet-based structure for expanding dependency trees, leading to the creation of the Syntactic Enhancement Network (SEN). This novel approach was rigorously tested across both sentence completion and relation extraction tasks. The results from these experiments were quite illuminating, leading to two primary insights. Firstly, the utilization of an advanced pre-trained language model can significantly boost the overall performance of the system. Secondly, the fusion of dependency syntax information with contextualized word embeddings is instrumental in achieving more nuanced and accurate sentence representations. Moreover, we constructed a new dataset for sentence completion tasks, tailored to reflect real-world application scenarios more closely. This dataset not only served as a practical testbed for our SEN model but also contributes to the broader research community by providing a resource that closely mirrors actual usage patterns. Looking ahead, our future endeavors will concentrate on further refining the combination of external knowledge with pre-trained language models. We aim to extend the applicability of our SEN model to a broader array of downstream tasks, thereby enhancing its versatility and effectiveness in various natural language processing applications. This ongoing research will contribute to the evolving landscape of language processing technologies, offering more sophisticated tools for understanding and utilizing natural language.