Modeling the Mutual Dynamic Correlations of Words in Written Texts Using Multivariate Hawkes Processes

1. Introduction

Analyzing document data as a time series has been a commonly employed approach in the past [1,2,3,4,5,6]. One notable advantage of representing documents as time series data is the ability to quantitatively capture long-range correlations among various components of the documents [7,8,9,10,11,12]. This advantage stems from the application of mathematical tools in time series analysis, such as autocorrelation functions [13,14,15], waiting time distributions [16,17], and similar methods. For instance, autocorrelation functions can be utilized to determine the significance of words within a given document [13]. Words exhibiting strong long-range autocorrelation across a text are often considered to be closely related to the document’s central theme [13,14,15,16,17].

In a previous study [17], we utilized univariate Hawkes processes to model word occurrence patterns in texts, demonstrating their effectiveness in capturing the dynamic, long-range correlations between different positions in a considered text. Hawkes processes are a type of stochastic process designed to model events occurring over time, influenced by prior occurrences [18]. They are particularly effective at capturing self-exciting phenomena, where the occurrence of one event increases the probability of subsequent events, whether in the short-term or long-term [19,20,21,22,23,24,25,26,27,28]. This characteristic makes Hawkes processes highly suitable for describing the occurrences of significant words in a text that are associated with specific concepts or ideas. Such key words often reappear during the explanation of crucial concepts. In other words, important words related to the subject matter of the text exhibit self-excitatory behavior, correlating with their own past occurrence signals. As a result, their patterns can be effectively modeled using Hawkes processes. This was confirmed in our previous study [17], which showed that many keywords in famous academic books follow such patterns.

However, the methodology proposed in the previous study [17] has certain limitations. Specifically, it employed univariate Hawkes processes, which can capture autocorrelation—measuring how a word’s occurrence signal is correlated with itself over different time lags. However, this approach cannot account for cross-correlation, which quantifies the correlation between the occurrence signals of two different words as a function of a time lag. It is common for significant concepts in documents to be explained from multiple perspectives, using several key terms. In such cases, these key terms should be interrelated or correlated, such that they mutually influence one another’s appearance within the document.

To characterize such documents, the model used should be capable of capturing the mutual correlations among keywords. This requirement underscores the necessity of multivariate Hawkes processes [29,30,31] for text modeling. As will be elaborated later, multivariate Hawkes processes can effectively represent multiple, mutually correlated stochastic processes, such as the occurrence signals of several keywords within a text. The objective of this study is to employ multivariate Hawkes processes to model documents. To the best of our knowledge, this represents the first application of multivariate Hawkes processes to document analysis. Furthermore, it will be demonstrated that employing multivariate Hawkes processes, as opposed to univariate ones, significantly enhances the accuracy in describing word occurrence signals. Moreover, we discovered that Hawkes graphs, [32] which are graphical representations of multivariate Hawkes processes, are highly effective in intuitively conveying the contents of analyzed documents. These graphs sensitively depict the interrelationships among multiple keywords appearing in the document. From the graph, it is possible to identify a central keyword that is closely tied to the document’s theme, as well as several other keywords used to describe or introduce the central concept. While Hawkes graphs may appear similar to diagrams depicting word co-occurrence networks [33,34], they are more advantageous as they incorporate information about causal relationships and dependencies among keywords. Thus, another significant benefit of utilizing the multivariate Hawkes process lies in its ability to provide an intuitive representation of document content through Hawkes graphs.

The remainder of this paper is organized as follows. In the next section, we outline the methodology, describing the characteristics of the univariate and multivariate Hawkes processes employed, including their kernel functions and log-likelihood functions. Additionally, we explain how word occurrence signals were extracted from each document and how these signals were modeled using the univariate and multivariate Hawkes processes. This section also details the simulation procedures used to validate our modeling approach with both types of Hawkes processes. The subsequent section presents our results, highlighting the advantages of multivariate Hawkes processes over univariate ones. This section also elaborates on the construction and practical aspects of Hawkes graphs. Finally, in the concluding section, we summarize our findings and propose directions for future research.

2. Methodology

2.1. Converting Text as Time Series Data

To treat written texts as time series data, we assign a serial number to every sentence in a considered document and assign a time role to this sentence number [13,14,15,16,17]. The occurrence signal of a considered word is then defined as

$$x\left(t\right)=\left\{\begin{array}{c}0\left(\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{d}\mathrm{d}\mathrm{o}\mathrm{e}\mathrm{s}\mathrm{n}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{p}\mathrm{p}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{i}\mathrm{n}\mathrm{t}\mathrm{h}\mathrm{e}t-\mathrm{t}\mathrm{h}\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}\right)\\ 1\left(\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{d}\mathrm{a}\mathrm{p}\mathrm{p}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{t}\mathrm{h}\mathrm{e}t-\mathrm{t}\mathrm{h}\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}\right)\end{array},\right.$$

(1)

which is a binary variable expressing word occurrence event, and accumulated value of $x\left(t\right)$, i.e.,

$$N\left(s\right)=\sum _{t=1}^{s}x\left(t\right)$$

(2)

becomes a counting process for the word occurrence event, i.e., $N\left(s\right)$ represents the number of occurrences of a considered word along the documents. We treat $x\left(t\right)$ and $N\left(s\right)$ as time series data and seek a stochastic process that can well describe the behavior of $x\left(t\right)$and $N\left(s\right)$. As already clarified, for words that are not directly connected to the theme of a given document and therefore that do not exhibit any dynamic correlations, their occurrences can be accurately modeled using either a homogeneous Poisson process or an inhomogeneous Poisson process [13,14,15,16]. In this paper, we investigate stochastic processes that characterize $x\left(t\right)$ and $N\left(s\right)$ for significant words which are closely related to the document’s theme and therefore have long-range dynamic correlations.

2.2. Maximum Likelihood Estimation of Hawkes Processes

To illustrate our methodology, we briefly introduce the intensity and likelihood functions of univariate and multivariate Hawkes processes, which we consider to be suitable stochastic models for explaining the observed word occurrence signals, Equation (1), for real words in the analyzed text.

The univariate Hawkes process is mathematically defined by its intensity function, $\lambda \left(t\right)$, representing the conditional event rate (word occurrenc rate) at time $t.$ For a process with an exponential decay kernel, the intensity function is given as [35,36]:

$$\lambda \left(t\right)=\mu +\sum _{\left\{t_i<t\right\}}\alpha \exp \left\{-\beta \left(t-t_i\right)\right\}$$

(3)

where $\mu$ denotes the baseline intensity, $t_i$represents the times of previous events ($t_i<t$), and $\alpha$ and $\beta$ are positive parameters of the exponential decay function. Given that event times $\{t_1,t_2,t_3,\dots ,t_n\}$ are observed within a time interval $[0,T]$, where $T$ denotes the total number of sentences in the text, the log-likelihood function of the univariate Hawkes process is expressed as [35,36]:

$$l\left(\mu ,\alpha ,\beta \right)=\sum _{i=1}^n\log \left(\mu +\sum _{\left\{t_j<t_i\right\}}\alpha \exp \left\{-\beta \left(t-t_i\right)\right\}\right)-\mu T-{\displaystyle \frac{\alpha }{\beta }}\sum _{i=1}^n\left(1-\exp \left\{-\beta \left(T-t_i\right)\right\}\right).$$

(4)

In the framework of maximum likelihood estimation (MLE), $\mu$, $\alpha$ and $\beta$ are treated as fitting parameters to adapt the univariate Hawkes process to the observed sequence of event times.

The multivariate Hawkes process expands upon the univariate Hawkes process by introducing multiple dimensions, allowing it to capture interactions among various types of events. In our case, we anticipate that the multivariate Hawkes process can effectively model the cross-correlation between word occurrence signals of different words within a given text. If we select $d$ words as the focus of our analysis and consider the$i$-th word, the intensity function of the multivariate Hawkes process for events of type $i$ (i.e., the intensity function for the occurrence of the $i$-th word) is defined as [36,37]:

$${\lambda }_i\left(t\right)={\mu }_i+\sum _{j=1}^d\sum _{\left\{{k:t}_k^j<t\right\}}{\alpha }_{ij}\exp \left\{-{\beta }_{ij}\left(t-t_k^j\right)\right\}$$

(5)

where ${\mu }_i$is the baseline intensity for events of type $i$, $d$ is the total number of event types (the total number of analyzed words), $t_k^j$ represents the time of the $k$-th event of type $j$ (i.e., the $k$-th occurrence of the $j$-th word), ${\alpha }_{ij}$ denotes the influence strength of events of type $j$ on those of type $i$, and ${\beta }_{ij}$ is the decay rate of the excitation. The log-likelihood function for the multivariate Hawkes process is expressed as:

$$\begin{gathered} l\left(\mathit{\mu },\mathit{\alpha },\mathit{\beta }\right)=\sum _{i=1}^d\sum _{k=1}^{n_i}\log \left[{\mu }_i+\sum _{j=1}^d\sum _{\left\{m:t_m^j<t_k^i\right\}}{\alpha }_{ij}\exp \left\{-{\beta }_{ij}\left(t_k^i-t_m^j\right)\right\}\right] \\ -\sum _{i=1}^d\left[{\mu }_{i}T-\sum _{j=1}^d\sum _{m=1}^{n_j}{\displaystyle \frac{{\alpha }_{ij}}{{\beta }_{ij}}}\left(1-\exp \left\{-{\beta }_{ij}\left(T-t_m^j\right)\right\}\right)\right] \end{gathered}$$

(6)

Here, $n_i$ represents the number of occurrences of events of type $i$, $\mathit{\mu }$ is the baseline intensity vector, defined as$\mathit{\mu }=({\mu }_1,{\mu }_2,\dots ,{\mu }_d)$, while $\mathit{\alpha }=\left\{{\alpha }_{ij}\right\}$ and $\mathit{\beta }=\left\{{\beta }_{ij}\right\}$ are matrices comprising the fitting parameters introduced in Equation (5). Equation (6) can be derived in the following manner. According to the general theory of point processes, the likelihood function of the multivariate Hawkes process is given by [38] (Chapter 7):

$$L(\mathit{\mu },\mathit{\alpha },\mathit{\beta })=\prod _{i=1}^d\left[\prod _{k=1}^{n_i}{\lambda }_i\left(t_k^i\right)\right]\exp \left(-{\int }_0^T{\lambda }_i\left(t\right)dt\right)$$

(7)

where ${\lambda }_i\left(t\right)$ is defined by Equation (5). By substituting Equation (5) into Equation (7), taking the logarithm of both sides, and performing further integral calculations, we obtain the log-likelihood function given in Equation (6).

In the framework of MLE for univariate Hawkes processes, the goal is to find optimized parameters $\mu$, $\alpha$ and $\beta$ that maximize the log-likelihood function, Equation (4), given a list of event times $\{t_1,t_2,t_3,\dots ,t_n\}$. Similarly, for multivariate Hawkes processes, the objective is to optimize the vector $\mathit{\mu }$ and matrices $\mathit{\alpha }$ and $\mathit{\beta }$ to maximize the log-likelihood function, Equation (6), given lists of event times for each type of event, $\{t_1^i,t_2^i,\dots ,t_{n_i}^i\}$ for $1\le i\le d$. In practical MLE procedures, we used the minimize() function from the Python library scipy.optimize to determine the optimal parameter values by minimizing $-l(\mu ,\alpha ,\beta )$ and $-l(\mathit{\mu },\mathit{\alpha },\mathit{\beta }).$ The L-BFGS-B (Limited-memory Broyden–Fletcher–Goldfarb–Shanno with Box constraints) method was employed to define upper and lower bounds for the fitting parameters, ensuring that the solutions remained within the specified ranges.

2.3. Selecting Important Words in Used Texts

Six texts employed in this study are listed in Table 1. One of the six documents is a famous historical pamphlet (“Common Sense” by Thomas Paine) ; the other five are well-known academic books. They are chosen so as to represent wide range of written texts, In the table, short names of each book and some information are also shown. The preface, contents and index pages were deleted before starting the text preprocessing because they may act as noise and may affect the final results.

We select 20 important words from each of the six texts to analyze their occurrence signals by using the univariate and the multivariate Hawkes processes. This means that 20 pairs of Equations, (3) and (4), are considered for modeling the 20 words with univariate Hawkes processes while we set $d=20$ in Equations (5) and (6) for modeling with the multivariate Hawkes process. The selected 20 important words having long-range dynamic correlations are listed in Table 2. Note that only nouns are included in the table because we intended to investigate the causal relationships between concepts represented by the nouns in the document under consideration (e.g., word “A” is used to describe word “B”, etc.). This investigation was performed based on the correlations derived from analysis using the multivariate Hawkes process.

The accuracy of representing word occurrence signals using the multivariate Hawkes process improves as the number of analyzed words increases. This is because a larger set of words allows for the consideration of all possible event causes that mutually excite one another. In most documents, key concepts are explained from multiple perspectives, utilizing numerous words to convey complex ideas. Such texts often exhibit correlations where many words influence each other.

In a multivariate Hawkes process, each dimension represents the occurrence signal of a single word. Consequently, limiting the process to only 20 dimensions restricts its ability to capture correlations among words, making it insufficient for describing the cross-correlations commonly found in texts. However, the maximum likelihood estimation (MLE) of a multivariate Hawkes process requires extensive convergence calculations, making it computationally intensive and practical only for cases with fewer than 20 dimensions. Therefore, establishing clear selection criteria for the 20 key words, as outlined below, is critical to this study.

First, we calculated the autocorrelation functions (ACFs) for all words that appear more than 50 times in each text, using the formula:

$$\varphi \left(s\right)={\displaystyle \frac{\frac{1}{T-s}\sum _{t=1}^{T-s}\left\{x\left(t\right)-\overline{x}\right\}\left\{x\left(t+s\right)-\overline{x}\right\}}{\frac{1}{T}\sum _{t=1}^T{\left\{x\left(t\right)-\overline{x}\right\}}^2}}$$

(8)

where $\overline{x}$ represents a mean value of $x\left(t\right)$. Then these ACFs were fitted by a Kohlausch-Williams-Watts (KWW) function:

$$\varphi \left(t\right)=\exp \left\{-{\left({\displaystyle \frac{t}{\tau }}\right)}^{\beta }\right\}$$

(9)

where $\tau$ (a relaxation time) and $\beta$ (a shape parameter) are fitting parameters that satisfy $\tau >0$ and $0<\beta \le 1$. The KWW function has been demonstrated to effectively represent real ACFs of important words with long-range dynamic correlations [13,14,15,16,17]. Next, we evaluated the mean relaxation times $\langle \tau \rangle$ for each word using the equation:

$$〈\tau 〉=\underset{0}{\overset{\infty }{\int }}\exp \left\{-{\left({\displaystyle \frac{t}{\tau }}\right)}^{\beta }\right\}dt={\displaystyle \frac{\tau }{\beta }}\mathsf{\Gamma }\left({\displaystyle \frac{1}{\beta }}\right)$$

(10)

where $\Gamma \left(x\right)$ denotes the gamma function. Finally, we select top 20 words in terms of $\langle \tau \rangle$ as the key words to be analyzed for each document. A large mean relaxation time, $\langle \tau \rangle$, indicates that a word exhibits both strong dynamic correlations and long-range memory, making it relevant to the key concept or theme of the text. Therefore, we consider words with large $\langle \tau \rangle$ to be significant. In Table 2, the values of the fitting parameters, $\tau$ and $\beta$, and the mean relaxation time, $\langle \tau \rangle$, are shown for each of the 20 words. Figure 1 shows the word occurrence signal $x\left(t\right)$, the calculated ACF and the fitted KWW function for a picked word “formation” in the Darwin text.

2.4. Validation of Modeling with Hawkes Processes

In this study, we use univariate and multivariate Hawkes processes to model word occurrence signals in the considered texts. The validity of the modeling is confirmed by following procedures.

To confirm the validity of using the univariate Hawkes process, we first evaluated the optimal values of the fitting parameters $\mu$, $\alpha$, and $\beta$ in Equations (3) and (4) using the maximum likelihood estimation (MLE). By the principle of the maximum likelihood, the univariate Hawkes process with the obtained parameter values is expected to best replicate the occurrence signals of the word in question. Using the optimized parameter values, we generated virtual word occurrence signals for the top 20 important words by simulating each of the 20 univariate Hawkes processes. These simulated word occurrence signals were compared to the observed signals in the real texts. The comparison mainly focused on the autocorrelation functions (ACFs): if the ACF of the simulated signal for a given word closely matches the ACF of the real signal, we conclude that the modeling using the univariate Hawkes process is effective.

To confirm the validity of the multivariate Hawkes process, we followed a similar approach. First, we determined the parameter values of the 20-dimensional multivariate Hawkes process using MLE given the word occurrence signals of the top 20 important words. Next, we simulated the 20-dimensional Hawkes process to generate virtual word occurrence signals for these 20 words. Finally, two types of ACFs—calculated from the simulated signals and the real signals for each of the 20 words—were compared to assess the effectiveness of the modeling using the multivariate Hawkes process.

3. Results and Discussion

3.1. Validity Confirmation of Hawkes Processes

As outlined above, we primarily compare the characteristic quantities derived from simulated signals with those obtained from word occurrence signals observed in real texts. The comparisons focused on the following four quantities:

• The total number of occurrences of the word throughout the text.
• The relaxation time $\tau$ of the ACF, derived from the fitting parameter of the KWW function (Equation (9)).
• The shape parameter $\beta$ of the ACF, also obtained from the fitting parameter of the KWW function.
• The Bayesian Information Criterion (BIC), calculated during the fitting of the KWW function to ACFs.

The results of these comparisons are displayed in plots, where the vertical axis represents one of the four characteristic quantities calculated from simulated signals, while the horizontal axis represents the corresponding quantity calculated from observed occurrence signals in the real text.

Figure 2 through Figure 7 present the results of these comparisons. Plots (a)-(d) in Figures 2 through 7 illustrate the comparison between the univariate Hawkes processes and the actual word occurrence signals, whereas plots (e)-(h) depict the comparison between the multivariate Hawkes processes and the actual word occurrence signals. Word occurrence signals were simulated 20 times for both univariate and multivariate Hawkes processes, resulting in 20 vertical values for each characteristic quantity corresponding to each word. The upper end of the error bars represents the maximum value among these 20 features, while the lower end denotes the minimum value. The vertical positions of the blue circles in Figures 2 through 7 indicate the averages of these 20 values.

If the simulated signal for a given word closely matches the real signal, the four characteristic quantities across plots (a)-(h) should align near or directly along the $y=x$ line. Thus, we can conclude that the modeling using univariate or multivariate Hawkes processes is effective if the blue circles are positioned along the straight line indicating direct proportionality. To assess the degree of linear correspondence between the vertical and horizontal quantities, we calculated correlation coefficients, which are displayed in the titles of the plots in Figures 2 through 7. Consequently, the validation criterion is how closely the correlation coefficient approaches 1.

Figure 2. Characteristics comparing real and simulated word occurrence signals. In all plots, the horizontal axis represents characteristics of the real signals in the Darwin text, while the vertical axis represents characteristics derived from the simulated signals. Characteristics derived from simulated signals generated from univariate Hawkes processes are shown in plots (a)–(d), while those from the 20-dimensional multivariate Hawkes process are shown in plots (e)–(h). The red lines represent the function $y=x$.

Figure 2. Characteristics comparing real and simulated word occurrence signals. In all plots, the horizontal axis represents characteristics of the real signals in the Darwin text, while the vertical axis represents characteristics derived from the simulated signals. Characteristics derived from simulated signals generated from univariate Hawkes processes are shown in plots (a)–(d), while those from the 20-dimensional multivariate Hawkes process are shown in plots (e)–(h). The red lines represent the function $y=x$.

Figure 3. The same meaning plots as in Figure 2 for Einstein text.

Figure 3. The same meaning plots as in Figure 2 for Einstein text.

Figure 4. The same meaning plots as in Figure 2 for Faraday text.

Figure 4. The same meaning plots as in Figure 2 for Faraday text.

Figure 5. The same meaning plots as in Figure 2 for Freud text.

Figure 5. The same meaning plots as in Figure 2 for Freud text.

Figure 6. Similar plots to those in Figure 2 are presented for the Paine text, except that an 8-dimensional multivariate Hawkes process was used to calculate the vertical quantities in plots (e)–(h).

Figure 6. Similar plots to those in Figure 2 are presented for the Paine text, except that an 8-dimensional multivariate Hawkes process was used to calculate the vertical quantities in plots (e)–(h).

Figure 7. The same meaning plots as in Figure 2 for Plato text.

Figure 7. The same meaning plots as in Figure 2 for Plato text.

Our primary focus is to evaluate the extent to which the effectiveness of modeling improves when transitioning from the univariate Hawkes process to the multivariate Hawkes process. To assess this, we compared the description accuracies of the univariate Hawkes process and the multivariate Hawkes process across Figures 2 through 7: comparisons between plots (a) and (e) for the number of occurrences, between plots (b) and (f) for the relaxation time $\tau$, between plots (c) and (g) for the shape parameter $\beta$, and between plots (d) and (h) for the Bayesian Information Criterion (BIC).

These comparisons clearly demonstrate that the multivariate Hawkes process significantly enhances modeling effectiveness compared to the univariate Hawkes process. This improvement is evidenced by the scatter plots of the four characteristic quantities being distributed closer to the straight line $y=x$ in the case of the multivariate Hawkes process, with correlation coefficients approaching 1. Indeed, Table 3 confirms that the correlation coefficient for each quantity is closer to 1 when modeling with the multivariate Hawkes process is applied.

This result is unsurprising, given the nature of the document, which presents concepts each of which is explained by using multiple key terms. These key terms are frequently repeated throughout the explanation, creating inherent correlations among them. As a result, the multivariate Hawkes process effectively captures these relationships.

In contrast, the univariate Hawkes process is limited to capturing self-correlation, meaning it only reflects the correlations found within the occurrence signal of a single word.

3.2. Hawkes Graphs

Hawkes graphs are directed graphs that intuitively represent multivariate Hawkes processes. In these graphs, each node corresponds to a specific type of event in the multivariate Hawkes process, while each edge indicates how one type of event (represented at the root of the arrow) influences or enhances another type of event (located at the tip of the arrow). Examples of Hawkes graphs are presented in Figure 8 and Figure 9, which illustrate the multivariate Hawkes processes optimized to model word occurrence signals in the Paine text and the Plato text, respectively. Each node represents the occurrence events of an important word, and its corresponding background intensity, ${\mu }_i$, as defined in Equation (5), is also displayed within the node.

In a multivariate Hawkes process, various types of events mutually excite one another, and the magnitude of their mutual excitation is quantified by the following coefficient [32]:

$$a_{ij}={\displaystyle \frac{{\beta }_{ji}}{{\alpha }_{ji}}}=\underset{0}{\overset{\infty }{\int }}{\alpha }_{ji}\exp \left(-{\beta }_{ji}t\right)dt$$

(11)

This coefficient measures how much the event of type-$i$ enhances the occurrence of the type-$j$ event. In Figure 8 and Figure 9, the values of $a_{ij}$ are displayed alongside the edges. To improve readability, edges with $a_{ij}<0.01$ are not shown in the graphs.

By constructing a Hawkes graph for a given text using the optimized parameter values of the multivariate Hawkes process, it becomes easier to infer the central concepts or notions of the text. For instance, based on Figure 8, we identify “time” as one of the central notions in the Paine text. This conclusion is drawn from observing that five edges associated with the “time” node are incoming arrows, while only one edge flows outward from this node. The predominance of incoming arrows indicates that the concepts represented by the source nodes are predominantly used to describe the notion of “time.”

3.3. Finding Important Notions in Texts

The idea for inferring the central notions of a text from the Hawkes graph, as mentioned above, can be refined as follows. The difference between the inflows into the $i$-th node and the outflows from the same node is defined as:

$$\mathsf{\Delta }{a}_i=\sum _{j=1}^d{a}_{ji}-\sum _{j=1}^d{a}_{ij}.$$

(12)

This quantity intuitively represents the net extent to which the notion corresponding to the $i$-th node is explained by other important notions.

The values of $\Delta a_i$are displayed in Table 4, which ranks the importance of each notion in descending order for each text. The most important notion for each text—represented by the top-ranked word in Table 4—can be considered appropriately identified based on the content of each text, as will be discussed below.

Clearly, the notion of “species” in the Darwin text, “relativity” in the Einstein text, and “dream” in the Freud text are all closely tied to the main themes of the respective documents, serving as the central concepts each text seeks to explain. In the Faraday text, the principles of chemistry and physics are explored through the fascinating subject of candles. The most significant topic in the Faraday text is the investigation of the substances that constitute candles and how they transform during combustion. Thus, it is reasonable to identify “substance” as the most critical concept in the text. The Paine text is a groundbreaking pamphlet published during the American Revolution, emphasizing the urgency of declaring independence. In this text, the author employs the notion of “time” as a motivational framework for driving transformative action. Hence, “time” is regarded as central because the author explains the nature of the current “time” within this context. In the Plato text, the notion of “man” takes center stage because the author’s philosophical reflections on justice, governance, and enlightenment are deeply rooted in his understanding of human nature and elaborated upon through this perspective.

3.4. Advantages of Analyzing Texts with Multivariate Hawkes Processes

Based on the results outlined so far, the following advantages of utilizing the multivariate Hawkes process have been established:

• As demonstrated in Figures 2 through 7, the accuracy of modeling word occurrence signals in documents is significantly improved, making it more reliable when using the multivariate Hawkes process compared to the univariate one.
• A Hawkes graph can be generated from the parameter values obtained through the analysis using the multivariate Hawkes process. This facilitates an intuitive understanding of the relationships among the concepts that emerge in the document.
• The importance of each concept identified in a text can be assessed using the optimized parameters of the multivariate Hawkes process. The most significant concepts suggested for each text analyzed in this study are confirmed to be valid when considering the content of the text.

4. Conclusions

The occurrence signals of important words in six texts (one historical pamphlet and five renowned academic books) were modeled using univariate and multivariate Hawkes processes. The modeling procedure for the univariate Hawkes process was conducted as follows. First, optimized parameter values for the univariate Hawkes process were determined using maximum likelihood estimation (MLE) based on the occurrence signals of a given word in a considered text. Next, word occurrence signals were generated by simulating the univariate Hawkes process with optimized parameters. Finally, the validity of the univariate Hawkes process modeling was confirmed by verifying that the four characteristic quantities derived from the simulated word occurrence signals closely matched those obtained from the real signals observed in the text.

The modeling procedure for the multivariate Hawkes process followed a similar approach. First, optimized parameters for the $d$-dimensional multivariate Hawkes process were evaluated using MLE based on the occurrence signals of $d$ important words, where $d$ represents the number of important words analyzed simultaneously with the multivariate Hawkes process. In this study, the typical value of $d$ was set to be 20. Next, occurrence signals of $d$ important words were simultaneously simulated using the multivariate Hawkes process with the optimized parameters. Finally, the validity of the multivariate Hawkes process modeling was confirmed by ensuring that the four characteristic quantities derived from the simulated word occurrence signals closely matched those obtained from the real signals observed in the text.

By validating the modeling with univariate and multivariate Hawkes processes, we observed that the correlation coefficients of the four characteristic quantities between real and simulated signals are notably closer to one in the case of the multivariate Hawkes process compared to the univariate one. This result demonstrates that the multivariate Hawkes process provides greater accuracy in describing word occurrence signals.

Additional advantages of the multivariate Hawkes process include its ability to intuitively represent the relationships among concepts described in a document through a Hawkes graph and to evaluate the importance of words in the text based on their relationships. For instance, the most important words in the analyzed documents were inferred using the optimized parameters of multivariate Hawkes processes, and these findings were confirmed to be valid.

Potential directions for future research utilizing the multivariate Hawkes process, which proved useful in this study, include:

• Enhancing the accuracy of word occurrence signal descriptions by increasing the number of important words analyzed simultaneously.
• Treating the matrix $a_{ij}$ (defined by Equation (11)) as an adjacency matrix and applying graph theory methods, such as spectral clustering.

Since increasing the dimensionality of the multivariate Hawkes process poses challenges related to the computational cost of the MLE method, these directions are deferred to future studies.