Research on Machine Reading Comprehension Integrating Longdistance Semantic Relations and Linguistic Features

I. Introduction

II. Related Work

A. Pre-Trained Language Models

Pre-trained language models obtain generic language representations by pre-training on large-scale open domain corpora, and then fine-tune them in specific downstream tasks to learn domain-specific knowledge[5]. The introduction of Transformer[6] marks the further improvement of model understanding and generalization ability, more accurate semantic feature extraction, and the ability of machines to learn the relationship between problems and answers through multi-layer neural networks. Traditional four-layer network models such as QANet[7] and BiDAF[8] have gradually faded from researchers' attention. In 2018, the Google research team launched the groundbreaking pre-trained language model BERT[9], an innovative initiative that inspired a lot of natural language processing research based on pre-trained language models, and also triggered the rise of pre-trained paradigms in the field of natural language processing[10]. Joint Laboratory of HIT and iFLYTEK Research has released a variety of Chinese pre-training model resources and related supporting tools. "Pre-training+fine-tuning" technology can solve different languages and different NLP tasks with one set of technologies, effectively improving the development efficiency. These include Chinese MacBERT[11], Chinese ELECTRA[12], Chinese XLNet[13], etc. In the process of model input, three parts of information need to be fused, the first is the Token Embedding, each lexicon has a corresponding index in the vocabulary, BERT maps these indexes through a learnable word embedding matrix into a continuous vector representation, these vectors capture the semantic information of the lexicon. To distinguish between different sections of input text, such as context and questions in question answering tasks, BERT introduces Segment Embedding. These embeddings are learnable vectors that are added to the representation of each tag in each paragraph. Since the Transformer architecture itself does not have the ability to capture sequence order, BERT adds Positional Embedding to the input representation to provide information about where the tags are in the sequence. Finally, all embeddings are added up as the final input, and the input form is shown in the Figure 1

The pre-trained language model utilizes the transformer architecture to handle long-distance dependencies through a self-attention mechanism, which is critical for understanding complex logic and reasoning in text. At the same time, the Mask Language Model (MLM) and the Next Sentence Prediction (NSP) can further optimize the semantic features of the text, and improve the generalization ability and adaptability of the model. The structure of the pre-trained language model represented by BERT is shown in the figure below.

Figure 2. Model structure.

Figure 2. Model structure.

B. Graph Convolutional Network

Traditional four-layer network model and pre-trained language model can only process data in Euclidian space. In the field of natural language processing, text structure, syntax and even the sentence itself exist in the form of graph data[14]. To seamlessly integrate structured data into a language model, you need to provide a comprehensive view of each node in the graph, including its neighbors, relationships to itself, and a clear description of the node[15]. Graph neural networks provide the ability to model non-European spatial data, which further expands the application range of deep neural networks[16]. Graph convolutional neural network[17] generates a vector representation by aggregating information of nodes and their neighbors, so that it can process graph data with complex structural relationships. Its core lies in graph convolution operation, which can capture local and global features in graph structure and optimize model performance through forward and backpropagation processes.

Graph convolutional neural networks have shown strong capabilities in many fields, such as social network analysis, recommendation system, traffic prediction and so on. The graph structure is represented by G=(V, E), where V represents the set of nodes and E represents the set of edges. n represents the number of nodes and m represents the number of edges. In GCN, only undirected graphs are usually considered. In this paper, the relationship between the named entities of the text is constructed as a graph, and its vector representation is obtained based on it. The convolutional neural network structure of the graph is shown in the figure.

Figure 3. Network structure.

Figure 3. Network structure.

III. 3 Semantic Fusion

A. Named Entity Extraction

Named entities, such as personal names, place names, organization names, dates, times, etc., are specific and identifiable important information units in a text. In machine reading comprehension, the system needs to accurately identify these entities so that it can accurately locate relevant information when answering questions or summarizing text. As can be seen from the following example, named entities occupy a core position in the semantics of paragraphs and there is a lot of repetition, which provides the possibility to collect entity information to enrich the semantic representation of text.

Figure 4. Example of a text segment.

Figure 4. Example of a text segment.

The resulting named entity recognition result contains the entity name, the start index in the text, the end index, and the entity type. The purpose of the start and end indexes is to pinpoint the position of the entity within the text or question, making it easier to combine with word granularity information in the original text. In this paper, Stanford open source toolkit Stanza[18] is used to extract named entity information. Stanza is a powerful multi-language named entity extraction tool, which provides a convenient natural language processing interface for Python users and supports text analysis in more than 60 languages. Including part-of-speech tagging, named entity recognition, dependency parsing and other NLP functions.

B. Construction of Entity Co-Occurrence Graph Based on Sentence Co-Occurrence Relation

As for the co-occurrence relationship of sentences, it can be understood that a paragraph is divided according to the unit of sentences, and the number of simultaneous occurrences of two named entities in the same sentence is counted respectively, which is used to measure the correlation strength between two entities. If two named entities appear frequently in the same sentence, the number of co-occurrences between them will increase, indicating that they are more relevant. If two entities appear n times in the same sentence, the edges in their co-occurrence graph are weighted to n. The paragraphs understood by reading and questions to be answered are joined together as the model input, and the entity co-occurrence diagram based on the sentence co-occurrence relationship is obtained, as shown in the figure below.

Figure 5. Graph based on the intersentence co-occurrence relationship.

Figure 5. Graph based on the intersentence co-occurrence relationship.

C. Construction of Entity Co-Occurrence Graph Based on Sliding Window

During the construction of entity co-occurrence graph based on sliding window, the weight of the connected edge between two nodes is no longer just the number of times the two entities co-occur in the sentence, but the point mutual information value (PMI)[19] is adopted as the measurement standard.

The PMI value is standardized to the interval of [-1, 1], where the PMI limit value -1 indicates that the two entities have never appeared at the same time within the text range under investigation; when the PMI value is 0, it indicates that the two entities are independent in appearance and have no correlation; when the PMI value is 1, it means that the two entities always occur at the same time completely. The introduction of standardized PMI enables more precise quantification of the strength of co-occurrence relationships between entities. The PMI value is calculated by the co-occurrence relationship of two words in the sliding window. Set a sliding window with a length of 10, and add 1 to the weight of two entities when they appear in the same document. The calculation formula for the PMI value is as follows.

$$PMI(i,j)=\log {\displaystyle \frac{p(i,j)}{p(i)p(j)}}$$

(1)

$$P(i,j)={\displaystyle \frac{\#W(i,j)}{\#W}}$$

(2)

$$p(i)={\displaystyle \frac{\#W(i)}{\#W}}$$

(3)

Where, #W (i) is the number of sliding Windows in the corpus containing word i, #W (i, j) is the number of sliding Windows containing both words i and j, and #W is the total number of sliding Windows in the corpus. This method can adjust the window size and move step according to the text length, and dynamically generate the feature map, but it ignores the sentence as the original text unit, which may lose part of the original semantic information. The entity co-occurrence diagram based on sliding window is shown in the figure below.

Figure 6. Graph based on slding window co-occurrence relationship.

Figure 6. Graph based on slding window co-occurrence relationship.

D. Named Entity Feature Generation

When the entity cooccurrence graph is input into the graph convolutional neural network, information consolidation and feature update are performed for each node in the hidden layer. This process involves two aspects. The first is neighbor information aggregation. For each node in the graph, the model will consider the characteristics or information of its neighbor nodes and aggregate the information. The second is node feature update, which usually combines the original node feature with the aggregated neighbor information through an update function to generate a new, richer node representation. This process involves two aspects. The first is neighbor information aggregation. For each node in the graph, the model will consider the characteristics or information of its neighbor nodes and aggregate the information. The second is node feature update, which usually combines the original node feature with the aggregated neighbor information through an update function to generate a new, richer node representation.As shown in the figure below, each node in the figure represents the named entity of the text segment, and its adjacency matrix is A. Assuming that a, b, c, d, and e are named entities, the adjacency matrix and degree matrix examples are shown in the figure below.

Figure 7. Examples of adjacency matrix and degree matrix.

Figure 7. Examples of adjacency matrix and degree matrix.

The matrix after aggregating the node features of the graph is shown in the figure below.

Figure 8. Example of information aggregation matrix.

Figure 8. Example of information aggregation matrix.

Through the stacking of multi-layer GCN, higher-level structure information of the graph can be gradually captured, thus generating a richer node representation. The specific propagation mode is shown as follows.

$$H^{(l+1)}=\sigma ({\tilde{D}}^{-{\displaystyle \frac{1}{2}}}\tilde{A}{\tilde{D}}^{-{\displaystyle \frac{1}{2}}}{H}^{(l)}{W}^{(l)})$$

(4)

GCN is A multi-layer graph convolutional neural network, each convolutional layer only processes first-order neighborhood information, and multi-order neighborhood information transfer can be realized by superimposing several convolutional layers. Each layer of GCN is multiplied by adjacency matrix A and feature matrix H(l) to obtain a summary of each vertex neighbor feature, and then multiplied by a parameter matrix W(l). The matrix H(l+1) of the features of the aggregated adjacent vertices is obtained by a nonlinear transformation of the activation function. Assuming a two-layer GCN is constructed and the activation functions are ReLU and Softmax respectively, the overall forward propagation formula is as follows:

$$Z=f(X,A)=soft\max (A^{\text{'}}\mathrm{Re}LU(A^{\text{'}}X{W}^{(0)})W^{(1)})$$

(5)

By stacking multiple GCN layers, features can be propagated over a wider area of the graph. Each layer receives the output of the previous layer as input and repeats the process of feature propagation, transformation, and activation. After multi-layer feature propagation, each node will obtain an updated feature representation, which encodes the structure and feature information of the node in the graph and can be used to obtain the final named entity feature representation. The propagation process of node features is shown in the figure below.

Figure 9. Feature propagation process.

Figure 9. Feature propagation process.

E. Feature Fusion

Structured entity knowledge can be effectively integrated into unstructured original text information by combining context embedding vector based on pre-trained language model and entity node embedding vector based on graph structure learning. This process helps the model to further capture deep linguistic features, thus improving the model's ability in semantic extraction and entity information perception. This fusion strategy enables the model to better understand and utilize the complex semantic relationships in the text, thus improving the overall performance. The overall model structure is shown in the figure.

Figure 10. Model architecture.

Figure 10. Model architecture.

Suppose that in a machine reading comprehension task, the input text segment is represented by $c=(c_1,c_2,c_3,\mathrm{...}{c}_{k_c})$, where kc represents the number of characters in the text segment, once the named entities are extracted, an entity matching matrix M is used to mark the position of each entity in the text segment, so M is a kc×kn matrix, where each element is calculated as follows.

Where kn represents the number of named entities extracted in paragraph c, and nj represents the NTH named entity extracted. An example M for input text is shown at the bottom of Figure 11. After the sequence-based embedding obtained by the pre-trained model and the entity embedding obtained by the graph structure, the two need to be fused to obtain a unified embedding representation combining the two features, and then the answer prediction is carried out. Here, $a_i$ is used to represent the word in the text paragraph, and $e_{\mathrm{j}}$ is used to represent the named entity in the paragraph. When feature fusion is carried out, the word embeddings, that is, the embeddings obtained by the pre-trained model, are represented by $E(a_i)$; the named entity embeddings, that is, the embeddings obtained by the graph structure, are represented by $E(e_j)$, and the fusion mode is as follows.

In the fusion process, $v_i^{(l)}$ is used to represent the i th character of layer l, and ${\mu }_{i,k}^{(l)}$ is used to represent the k th named entity related to $v_i^{(l)}$. The specific calculation formula for each character is as follows.

$$v_{\mathrm{i}}^{(l)}{ }^{*}=v_{\mathrm{i}}^{(l)}+{\displaystyle \sum _k{\mu }_{i,k}^{(l)}}$$

(8)

Extending to the whole layer, the calculation formula is as follows.

$$V_{ }^{(l)}{ }^{*}=V_{ }^{(l)}+M\times U_{ }^{(l)}$$

(9)

Where V(l) is the embedding matrix of all characters, and its association with U (l) can be done directly through M.

IV. Experiment

A. Dataset

Dureader was adopted as the basic data set of the experiment. Dureader is a large-scale Chinese machine reading comprehension data set, which contains a large number of question and answer pairs under real scenarios, and aims to evaluate the model's ability to understand natural language text and answer questions. It includes three types of data, namely entity type, description type and non-correct type. Each type is also divided into fact and opinion, and the data contains 200k questions, 1000k original text and 420k answers. In addition, a part of question and answer data is extracted from the Dureader data set as a supplement, and the data format is normalized by means of data cleaning and regularization, as shown in the figure below.

Figure 11. Data format.

Figure 11. Data format.

V. Summary and Prospect

References

1. YAN Weihong, Li Shaobo, Shan Lili, Sun Chengjie, Liu Bingquan. An extractive machine reading comprehension model with explicit lexical and syntactic features. Applications of Computer Systems,2022,31(09):352-359.
2. Bian R, T. Application of pre-training model integrating knowledge in reading comprehension [D]. Harbin Institute of Technology, 2021.
3. Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proceedings of Advances in Neural Information Processing Systems，2017: 5998-6008.
4. Lu, Yinquan et al. KELM: Knowledge Enhanced Pre-Trained Language Representations with Message Passing on Hierarchical Relational Graphs. arXiv:2109.04223, 2021.
5. Liu, Weijie, et al. "K-BERT: Enabling Language Representation with Knowledge Graph." Proceedings of the AAAI Conference on Artificial Intelligence 34.3(2020):2901-2908.
6. Liu, S.; Zhang, X.; Zhang, S.; Wang, H.; Zhang, W. Neural Machine Reading Comprehension: Methods and Trends. Appl. Sci. 2019, 9, 3698. [Google Scholar] [CrossRef]
7. Seo M, Kembhavi A, Farhadi A, et al. Bidirectional attention flow for machine comprehension. arXiv:1611.01603, 2016.
8. Yu A W, Dohan D, Luong M, et al. QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension[C]//Proceedings of the International Conference on Learning Representations, 2018 : 13-17.
9. Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019: 4171-4186.
10. Zhang Qi, GUI Tao, Zheng Rui, Huang Xuanjing. Large-scale Language Model: From Theory to Practice [M]. Publishing House of Electronics Industry, 2024.
11. Cui, Y.; Che, W.; Liu, T.; Qin, B.; Wang, S.; Hu, G. Revisiting Pre-Trained Models for Chinese Natural Language Processing. Findings of the Association for Computational Linguistics: EMNLP 2020. LOCATION OF CONFERENCE, COUNTRYDATE OF CONFERENCE; pp. 657–668.
12. Clark, Kevin, et al. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. arXiv:2003.10555, 2020.
13. Yang, Zhilin, et al. XLNet: Generalized Autoregressive Pretraining for Language Understanding[C]// Advances in Neural Information Processing Systems, 2019.
14. Chen Yulong, Fu Qiankun, Zhang Yue. Application of graph neural networks in natural language processing. Journal of Chinese Information Processing, 2021,35(3):23.
15. Xu, Y.; Zhu, C.; Xu, R.; Liu, Y.; Zeng, M.; Huang, X. 2021. Fusing context into knowledge graph for commonsense question answering. In Findings of ACL-IJCNLP 2021, pages 1201-1207.
16. Zhang, S.; Tong, H.; Xu, J.; Maciejewski, R. Graph convolutional networks: a comprehensive review. Comput. Soc. Networks 2019, 6, 1–23. [Google Scholar] [CrossRef] [PubMed]
17. Kipf T N, Welling M. Semi-supervised Classification with Graph Convolutional Networks. arXiv:1609.02907, 2016.
18. Qi P, Zhang Y, Zhang Y, et al.Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. 2020. acl-demos.14.
19. Yao L, MaoC, Luo Y. Graph convolutional networks for text classification[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2019, (33 ):.7370-7377.