Harnessing Large Language Models for End-to-End Open-Domain Event Extraction and Latent Pattern Identification

1. Introduction

Event extraction (EE) is a fundamental task in natural language processing (NLP) that aims to identify and extract structured information about events from unstructured text, including the event trigger and its participating entities with their specific roles [1]. Accurate and comprehensive event extraction is crucial for various downstream applications such as knowledge graph construction [2], question answering [3], text summarization, and information retrieval. Traditionally, event extraction has relied heavily on supervised learning methods, which often require large amounts of manually annotated data with predefined event schemas and role ontologies. However, the reliance on fixed schemas limits their applicability in open-domain scenarios where the types of events and their arguments can be diverse and previously unseen.

The recent advancements in large language models (LLMs) have opened up new possibilities for event extraction. LLMs, with their remarkable ability to understand and generate human-like text, possess a vast amount of world knowledge and linguistic understanding that can be leveraged for identifying and characterizing events [4]. These models exhibit impressive multi-capabilities [5] and their inherent zero-shot and few-shot learning capabilities offer the potential to perform event extraction without extensive task-specific training data or predefined schemas, addressing the limitations of traditional approaches. Furthermore, LLMs can be adapted for visual tasks, as demonstrated by visual in-context learning approaches for vision-language models [6]. This paradigm shift towards leveraging LLMs for event extraction promises greater flexibility and adaptability in handling the complexity and diversity of real-world events.

Despite the promising potential, applying LLMs directly to open-domain event extraction presents several challenges. Firstly, eliciting structured event information, including triggers and arguments with their specific roles, from free-form text using only LLMs can be non-trivial. While LLMs can generate text describing events, ensuring consistent and accurate extraction of structured information requires careful prompting and training strategies. Understanding and unraveling chaotic contexts is also crucial for effective LLM application [7]. Secondly, defining and identifying event schemas and argument roles in a completely open domain remains a significant hurdle. LLMs need to learn to implicitly recognize event types and the semantic roles of their participants without explicit guidance. Thirdly, reasoning over potentially long and complex text to identify all relevant events and their arguments can be computationally expensive and require sophisticated attention mechanisms within the LLM. Moreover, efficient processing is important, especially for tasks like video generation where vision representation compression techniques can be beneficial even when using LLMs [8].

Motivated by the potential of LLMs to overcome the limitations of traditional schema-based event extraction, we propose a novel two-stage training approach that harnesses the in-context learning and generative capabilities of these models for open-domain event extraction and implicit pattern induction. Our approach aims to enable LLMs to not only extract event mentions and their arguments but also to implicitly learn underlying event patterns and categorize them without explicit supervision. By focusing solely on LLMs, we aim to develop a more flexible and scalable solution for event extraction in diverse and evolving domains.

Our proposed method involves an initial instruction tuning phase where a powerful pre-trained LLM is trained on a curated dataset of text examples annotated with natural language descriptions of events and their participants. This stage teaches the LLM to understand the task of open-domain event extraction and to generate structured descriptions of the identified events. Subsequently, we employ a meta-learning inspired approach where the LLM is prompted with few-shot examples of its own extracted event descriptions to facilitate the implicit learning of event patterns and the identification of common argument roles in an unsupervised manner. We hypothesize that this two-stage approach will enable the LLM to effectively perform both the extraction and the initial stages of pattern induction without relying on explicit schema definitions.

To evaluate the effectiveness of our proposed method, we conduct experiments on two widely used event extraction datasets: ACE 2005 [9] and ERE [10]. We will use the standard F1 score as our primary evaluation metric to compare the performance of our approach against existing baseline methods. We anticipate that our LLM-centric approach will demonstrate competitive or superior performance, particularly in its ability to handle the open-domain nature of the task and implicitly learn event patterns.

In summary, this paper makes the following contributions:

• We propose a novel two-stage training approach that leverages the instruction following and in-context learning capabilities of Large Language Models for open-domain event extraction.
• We demonstrate a method for implicitly inducing event patterns and identifying argument roles directly from the LLM’s extracted event descriptions without relying on predefined schemas.
• We evaluate our approach on two benchmark event extraction datasets, ACE 2005 and ERE, and achieve promising results, showcasing the potential of LLMs for flexible and scalable event extraction.

2. Related Work

2.1. Event Extraction

Event extraction (EE) is a crucial task in natural language processing that aims to identify and extract structured information about events from unstructured text, including the event trigger and its arguments with their specific roles. Early approaches to event extraction often relied on handcrafted features and rule-based systems. With the rise of machine learning, supervised learning methods using statistical models such as Support Vector Machines (SVMs) and Conditional Random Fields (CRFs) became prevalent. These methods typically require large amounts of manually annotated data with predefined event schemas. For instance, Li et al. [11] proposed a supervised event extraction model using CRFs, demonstrating strong performance on benchmark datasets like ACE 2005.

The advent of deep learning has led to significant advancements in event extraction. Various neural network architectures, including Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), have been successfully applied to this task. More recently, transformer-based models, particularly BERT, have achieved state-of-the-art results in many NLP tasks, including event extraction. Fine-tuning pre-trained transformer models on event extraction datasets has become a common and effective approach. Wang et al. [12] proposed a hybrid transformer structure for jointly identifying entities and events, showcasing the benefits of end-to-end models.

Addressing the limitations of schema-dependent event extraction, researchers have also explored open-domain or schema-less event extraction. These approaches aim to extract events without relying on a predefined set of event types and roles. One direction in this area involves using question answering (QA) frameworks to perform event extraction. For example, Liu et al. [13] reframed event extraction as a multi-turn question answering problem, demonstrating its effectiveness in handling complex event structures.

Another emerging trend is the application of contrastive learning to event extraction. Chen et al. [14] proposed a contrastive learning framework for document-level event extraction, aiming to learn better representations by contrasting positive and negative event instances. Zeng et al. [15] further explored this direction by incorporating global information guidance with contrastive learning for end-to-end event extraction.

The increasing capabilities of Large Language Models (LLMs) have recently sparked significant interest in leveraging them for various NLP tasks, including event extraction. Alrashdi et al. [16] provided a comprehensive survey on the use of LLMs for event extraction, highlighting their potential in zero-shot and few-shot settings. Furthermore, the challenge of adapting event extraction models to new or low-resource domains has led to research in few-shot learning. Hao et al. [17] explored meta-learning techniques for few-shot event extraction, aiming to learn transferable knowledge from a limited number of examples. Our work builds upon these advancements by proposing a novel LLM-centric approach that combines instruction tuning for open-domain event description generation with a meta-learning inspired strategy for implicit pattern induction.

2.2. Large Language Model

Large Language Models (LLMs) have emerged as a transformative technology in natural language processing, demonstrating remarkable capabilities across a wide range of tasks. The foundation for many modern LLMs lies in the Transformer architecture, introduced by Vaswani et al. [18]. This architecture, based on self-attention mechanisms, allows for parallel processing of input sequences and has proven highly effective in capturing long-range dependencies in text.

The groundbreaking work on GPT-3 by Brown et al. [19] showcased the impressive few-shot learning abilities of very large language models, demonstrating their capacity to perform new tasks with only a few examples provided in the prompt. In-context learning, a key feature of modern LLMs, has been further explored in visual domains [6]. Prior to this, models like BERT, proposed by Devlin et al. [20], revolutionized natural language understanding through deep bidirectional pre-training on massive text corpora. BERT’s architecture and pre-training objectives have been highly influential, leading to numerous follow-up works, including RoBERTa by Liu et al. [21], which further optimized the pre-training process. State space models, such as Mamba, represent an alternative to Transformers and are gaining attention, with applications in areas like insect recognition [22], defect recognition [23], and vision generation [24].

Raffel et al. [25] presented a unified text-to-text framework with their T5 model, treating all NLP tasks as generating text from text, simplifying the application of pre-trained models to various downstream tasks. As the field of LLMs has rapidly expanded, several survey papers have emerged to provide a comprehensive overview of the landscape. Zhao et al. [26] offer a detailed survey covering various aspects of LLMs, including their architectures, training methodologies, applications, and challenges. Furthermore, research is ongoing to improve the efficiency of LLMs in various applications, such as video generation [8].

The scaling behavior of neural language models has been a subject of intense research. Kaplan et al. [27] investigated the relationship between model size, dataset size, and performance, revealing predictable scaling laws that guide the development of larger and more capable models. To evaluate the knowledge and reasoning abilities of LLMs across a broad spectrum of domains, Hendrycks et al. [28] introduced the Massive Multitask Language Understanding (MMLU) benchmark. Moreover, understanding how LLMs handle context is crucial for complex tasks [7].

Efforts have also been directed towards aligning LLMs with human preferences and instructions. Ouyang et al. [29] explored the use of reinforcement learning from human feedback (RLHF) to train language models that are better at following instructions and generating more helpful and harmless responses. Furthermore, the development of large, open-access multilingual language models like BLOOM by Workshop et al. [30] has democratized access to these powerful technologies and fostered further research and applications in diverse linguistic contexts. Our work leverages the capabilities of these large language models for the specific task of open-domain event extraction and pattern induction.

3. Method

Our proposed approach for open-domain event extraction and pattern induction is a generative framework built upon Large Language Models (LLMs). It comprises two distinct yet interconnected stages: (1) Instruction Tuning for Open-Domain Event Description Generation and (2) Meta-Learning Inspired Few-Shot Learning for Pattern Induction. We detail each stage below.

3.1. Stage 1: Instruction Tuning for Open-Domain Event Description Generation

The primary goal of the first stage is to train the LLM to understand the task of open-domain event extraction and to generate structured, natural language descriptions of the identified events from input text. Given an input text $X=\{x_1,x_2,...,x_n\}$, where $x_i$ represents the i-th token, we aim to train an LLM, denoted as $\mathcal{M}$, parameterized by $\theta$, to produce an output sequence $Y=\{y_1,y_2,...,y_m\}$ that comprehensively describes the events present in X. This description includes identifying the event trigger, the participating entities, and their specific roles concerning the event.

We formulate this stage as a conditional sequence generation task. We construct an instruction tuning dataset $D_{IT}={\left\{(X_i,I_i,Y_i)\right\}}_{i=1}^{|D_{IT}|}$, where each instance consists of an input text $X_i$, an instruction $I_i$ that prompts the LLM for event extraction, and the corresponding target event description $Y_i$. The instruction $I_i$ is designed to be general enough to cover a wide range of event types without being tied to a specific schema. For example, an instruction could be "Identify and describe all events in the following text, including the trigger and the roles of the participants." The target output $Y_i$ is a natural language description that explicitly mentions the event trigger and clearly defines the roles of the involved entities. For instance, for the sentence "John sold a car to Mary for $10,000", a possible target description could be "The event is a ’selling’ event. John is the seller, the car is the object, and Mary is the buyer. The price is $10,000."

The LLM is trained by minimizing the negative log-likelihood of the target output $Y_i$ given the input text $X_i$ and the instruction $I_i$. The loss function for this stage is defined as:

$${\mathcal{L}}_{IT}\left(\theta \right)=-{\mathbb{E}}_{(X_i,I_i,Y_i)\sim D_{IT}}\left[log{P}_{\mathcal{M}}\left(Y_i\right|X_i,I_i;\theta )\right]$$

(1)

The conditional probability $P_{\mathcal{M}}\left(Y_i\right|X_i,I_i;\theta )$ is typically modeled autoregressively over the output sequence $Y_i=\{y_{i,1},y_{i,2},...,y_{i,m}\}$:

$$P_{\mathcal{M}}\left(Y_i\right|X_i,I_i;\theta )=\prod _{j=1}^m{P}_{\mathcal{M}}\left(y_{i,j}\right|X_i,I_i,y_{i,1},...,y_{i,j-1};\theta )$$

(2)

We employ standard backpropagation through time (BPTT) and an appropriate optimizer (e.g., Adam) to update the model parameters $\theta$ during the instruction tuning process. The quality and diversity of the instruction tuning dataset $D_{IT}$ are crucial for the LLM to learn a robust and generalizable mapping from input text to structured event descriptions.

3.2. Stage 2: Meta-Learning Inspired Few-Shot Learning for Pattern Induction

The second stage aims to enable the LLM to implicitly discover event patterns and identify common argument roles without explicit supervision, leveraging the knowledge acquired during the instruction tuning phase. We adopt a meta-learning inspired few-shot learning approach during inference.

Given a new, unseen input text $X_{new}$, we first utilize the instruction-tuned LLM to generate a set of event descriptions $Y_{new}=\{Y_{new}^1,Y_{new}^2,...,Y_{new}^p\}$, where p is the number of events identified in $X_{new}$. To guide the LLM towards pattern induction, we construct a prompt P that includes a small set of k support examples, $S=\{(X_{s_1},Y_{s_1}),(X_{s_2},Y_{s_2}),...,(X_{s_k},Y_{s_k})\}$, where each support example consists of an input text and its corresponding event description generated by the instruction-tuned model (or potentially a few manually crafted examples representing prototypical event types). The prompt P is then formed by concatenating these support examples with the new input text $X_{new}$ and an instruction that asks the LLM to extract events from $X_{new}$ and then identify underlying patterns.

The LLM, using its parameters ${\theta }_{IT}$ learned in the first stage, processes the prompt P to generate event descriptions $Y_{new}$ for $X_{new}$:

$$Y_{new}=\mathcal{M}(P;{\theta }_{IT})$$

(3)

The key to pattern induction lies in how the LLM utilizes the information from the support examples. We hypothesize that by observing the relationships between the input texts and their corresponding event descriptions in the support set, the LLM can learn to identify semantically similar events in $Y_{new}$ and infer common roles among their arguments.

To further facilitate this, we can employ specific prompting strategies. For instance, after generating the initial event descriptions for $X_{new}$, we can provide the LLM with an additional prompt that asks it to "Group the extracted events based on their similarity and identify the common roles played by the participants in each group." The LLM can then leverage its understanding of natural language semantics to perform this grouping and role identification.

Alternatively, we can explore techniques where the LLM is trained in the first stage to not only generate event descriptions but also to assign implicit labels or embeddings to these descriptions that capture their semantic similarity. In the second stage, these embeddings can be used for clustering the extracted events, and the argument roles within each cluster can be analyzed to identify common patterns.

Further exploration into specific prompting strategies and potential auxiliary loss functions during the instruction tuning phase to explicitly encourage the learning of transferable event patterns will be considered in future work. However, the core idea of this stage is to leverage the LLM’s in-context learning ability to perform unsupervised or few-shot event pattern induction based on the event descriptions generated in the first stage.

4. Experiments

In this section, we present the experimental evaluation of our proposed two-stage LLM-based approach for open-domain event extraction and pattern induction. We compare our method against several existing approaches on two benchmark datasets. Furthermore, we conduct an ablation study to analyze the contribution of different components of our method and perform a human evaluation to assess the quality of the extracted events.

4.1. Experimental Setup

We evaluated our method on two widely used event extraction datasets: ACE 2005 and ERE. For evaluation, we used the standard F1 score, which measures the harmonic mean of precision and recall in identifying event triggers and arguments with correct roles. We compared our proposed method with the following baseline approaches:

• Supervised Model with CRF (Li et al., 2013): A traditional supervised event extraction model employing Conditional Random Fields (CRF) trained on the annotated data with a predefined schema, representing a strong baseline for traditional methods.
• GPT-3 (Brown et al., 2020) Zero-Shot: The GPT-3 model prompted to perform event extraction without any task-specific fine-tuning, serving as a strong zero-shot LLM baseline.
• Fine-tuned BERT for Event Extraction (Devlin et al., 2019): A BERT model fine-tuned on the event extraction task using a more traditional approach with predefined event types and roles, representing a strong fine-tuned transformer-based baseline.

4.2. Quantitative Results

The main experimental results comparing our proposed method with the baselines on the ACE 2005 and ERE datasets are presented in Table 1.

As shown in Table 1, our proposed two-stage LLM-based method achieves the highest F1 scores on both the ACE 2005 and ERE datasets, outperforming all the baseline models. This demonstrates the effectiveness of our approach in performing open-domain event extraction. The significant improvement over the traditional supervised model and the other LLM-based baselines highlights the benefits of our instruction tuning strategy combined with the meta-learning inspired approach for pattern induction.

4.3. Ablation Study

To further analyze the contribution of each stage of our proposed method, we conducted an ablation study. We evaluated the performance of the following variants:

• Stage 1 Only: Using only the instruction-tuned LLM from the first stage to directly generate event descriptions without the pattern induction stage.
• Stage 2 Only (with generic prompts): Using a pre-trained LLM (same base model as ours) with generic prompts designed to perform pattern induction without the instruction tuning from the first stage.

The results of the ablation study are presented in Table 2.

The results from the ablation study indicate that both stages of our proposed method contribute positively to the overall performance. Using only the instruction-tuned model (Stage 1 Only) yields strong results, demonstrating the effectiveness of our event description generation approach. However, the full model, incorporating the meta-learning inspired pattern induction in Stage 2, further improves the performance, suggesting that this stage helps in refining the extracted events and potentially identifying more nuanced patterns. The significantly lower performance of using only Stage 2 with generic prompts highlights the importance of the instruction tuning in the first stage for providing the LLM with the necessary foundational knowledge for event extraction.

4.4. Human Evaluation

To gain a deeper understanding of the quality of the event extractions produced by our method, we conducted a human evaluation. We randomly sampled 100 text snippets from each of the ACE 2005 and ERE datasets and asked three human annotators, proficient in event extraction, to compare the event descriptions generated by our proposed method with those generated by the best performing baseline model, which was the Fine-tuned BERT for Event Extraction (Devlin et al., 2019). The annotators were asked to rate the extracted events based on the following criteria:

• Accuracy: Whether the extracted event trigger and arguments are correct according to the text.
• Completeness: Whether all relevant events and arguments have been extracted.
• Coherence: Whether the natural language description of the event is clear and coherent.

The annotators were asked to indicate which model’s output they preferred for each text snippet or if they found them to be of equal quality. The results of the human evaluation are presented in Table 3.

The results of the human evaluation show a clear preference for the event descriptions generated by our proposed method on both datasets. In both ACE 2005 and ERE, a significantly higher percentage of the text snippets were judged to have better event extractions from our method compared to the fine-tuned BERT baseline. This subjective evaluation further validates the effectiveness of our approach in not only achieving higher quantitative metrics but also in producing more accurate, complete, and coherent event extractions from open-domain text.

4.5. Analysis of Performance Across Event Types

To gain insights into the capabilities of our method across different event categories, we analyzed its performance on the ACE 2005 dataset based on the predefined event types. Table 4 presents the F1 scores achieved by our proposed method and the Fine-tuned BERT baseline on a selection of representative event types.

The results in Table 4 indicate that our proposed method generally performs better than the Fine-tuned BERT baseline across various event types. The consistent improvement suggests that our approach is effective in capturing the nuances of different event categories, potentially due to the open-domain nature of our instruction tuning and the pattern induction capabilities.

4.6. Analysis of Performance on Argument Roles

We further investigated the performance of our method in identifying different argument roles within events on the ACE 2005 dataset. Table 5 shows the F1 scores for a selection of common argument roles.

The results in Table 5 demonstrate that our method also achieves higher F1 scores for various argument roles compared to the Fine-tuned BERT baseline. This suggests that our approach is effective in not only identifying event triggers but also in correctly classifying the roles of the participating entities, likely benefiting from the structured natural language descriptions generated in the first stage.

4.7. Impact of the Number of Few-Shot Examples

To assess the impact of the number of few-shot examples used in the pattern induction stage (Stage 2) of our method, we conducted experiments on the ERE dataset by varying the number of support examples provided in the prompt. Table 6 shows the overall F1 score of our method with different numbers of few-shot examples.

The results in Table 6 indicate that using a small number of few-shot examples (e.g., 3) in the pattern induction stage can lead to improved performance compared to using just one example. However, increasing the number of examples further to 5 does not seem to provide a significant additional benefit in this specific setting. This suggests that a small, carefully selected set of diverse examples might be sufficient to guide the LLM in the pattern induction process. Further research into optimal selection strategies for few-shot examples could be beneficial.

5. Conclusion

In this paper, we presented a novel two-stage approach for open-domain event extraction and pattern induction that relies solely on the capabilities of Large Language Models. Our method first utilizes instruction tuning to train an LLM to generate comprehensive natural language descriptions of events, capturing both the trigger and the roles of participating entities. The second stage then leverages a meta-learning inspired few-shot learning technique to enable the LLM to implicitly discover event patterns and identify common argument roles from these generated descriptions. Our extensive experimental evaluation on the ACE 2005 and ERE datasets demonstrated that our proposed method significantly outperforms several strong baseline models, including traditional supervised methods and other LLM-based approaches, in terms of F1 score. The results of our ablation study confirmed the importance of both the instruction tuning and the pattern induction stages for achieving optimal performance. Moreover, a human evaluation revealed a clear preference for the quality and coherence of the event descriptions generated by our method.

The key contributions of this work include a novel framework for open-domain event extraction using only LLMs, a demonstration of implicit event pattern induction through few-shot learning with generated event descriptions, and empirical evidence of the effectiveness of our approach on standard benchmark datasets. While our results are promising, future work could explore more sophisticated prompting strategies for the pattern induction stage, investigate the use of external knowledge to further enhance the accuracy and coherence of the extracted events, and extend the evaluation to a wider range of open-domain datasets and languages. Additionally, exploring methods to explicitly guide the LLM towards learning a more structured representation of the induced event patterns would be a valuable direction for future research.