Dementia Detection by In-Text Pause Encoding

1. Introduction

In dementia, language difficulties primarily manifest in terms of word access, word meaning, and the pragmatic aspects of communication. For instance, individuals affected by AD frequently tend to use semantically “empty" words like “thing" or “stuff," which lack specificity and nuance [1]. They also tend to employ relatively lower portions of nouns and, notably, fewer verbs that carry significant informational content [2]. This can result in their communication appearing less precise and more challenging to follow. Furthermore, their overall discourse may seem disorganized, making it harder for others to engage in meaningful conversations with them.

On the other hand, it is generally believed that other language components such as syntax (the structure of sentences), phonology (the sound system of language), and articulation (the physical production of speech sounds) remain relatively well-preserved until the later stages of the disease [3]. However, this particular conclusion remains a subject of controversy within the research community, with some experts challenging the notion that these aspects of language remain unaffected throughout the course of Alzheimer’s Disease, but not in the late stages of the disease [4,5].

The advent of large language models (LLMs) has led to significant achievements across various domains, including sentiment analysis, question answering, summarization, and more [6,7]. These models have demonstrated their ability to tackle intricate tasks effectively. Notably, models like BERT [8] and similar variants [9,10,11] have shown their capacity for comprehending the context within textual data, encompassing diverse aspects of language, such as semantics and syntax [12].

In recent years, researchers have integrated LLMs into their studies on various forms of dementia. They have approached this by either analyzing textual data or speech independently or by combining both modalities simultaneously, as referenced in a series of studies [13,14,15,16,17]. In most cases, obtaining transcriptions of speech involves the use of automatic speech recognition (ASR) models. Our primary focus here is on the text-based aspects of their methodologies. For example, one study [14] introduced a novel approach by incorporating pauses (special characters) in the textual data, aiming to leverage these pause-related cues within the textual context. This inclusion led to performance enhancements compared to utilizing plain text alone. Another prevalent strategy in dementia detection is the utilization of transfer learning, as demonstrated in multiple studies [14,16,18], highlighting its effectiveness in enhancing performance.

It’s worth noting that training LLMs can be challenging due to limited datasets for dementia detection. To address this challenge, several authors proposed various data augmentation techniques in studies [19,20,21], which have proven to be effective in augmenting the available data for training models. Additionally, some researchers have explored the robustness and sensitivity of LLMs in predicting Alzheimer’s disease. In a particular study [22], the investigation focused on evaluating the robustness and sensitivity of BERT-like models in Alzheimer’s disease prediction. This research is crucial not only for the development of more reliable classification models, but also for gaining a better understanding of the capabilities and limitations of these models.

In this paper, we aim to detect dementia through textual input with in-text pause encoding. In our methodology, three different pauses (short, medium, and long) are extracted from the audio and encoded with the corresponding text by different encoding techniques. We perform a combinatorial search to find the best combination of in-text pause encoding scheme that results in superior performance. To the best of our knowledge this type of approach is novel and the in-text pause encoding techniques have not been tested on the Pitt Corpus Cookie Theft dataset [23]. Our main results indicate that this approach is effective in improving performance compared to the baseline. These results show the advantage of incorporating pause information within the text.

The rest of this paper is organized as follows: In Section 2, the data preparation, pause encoding, and modeling are discussed. In Section 3, the results are presented. Lastly, we present our conclusions in Section 4.

2. Methodology

In our experiments, the Pitt Corpus Cookie Theft dataset [23] is used. This dataset contains audio and transcripts from different individuals for different tasks. In the first step of our methodology, we clean the dataset from the special characters that would not be used in our experiments. Next, we encode different types of pauses within the textual input to the model, as explained in Section 2.2. In Section 2.3, different model architectures are proposed. These models are carefully selected to study the effect of model complexity on performance.

2.1. Data Preparation

In here we outline the procedure for data cleaning in our study, which utilizes transcripts from the Pitt Corpus Cookie Theft dataset [23]. These transcripts are rich in detail, including the patients’ demographic information like gender and age, as well as clinical data such as dementia severity. Additionally, they contain syntactic details to ensure language consistency, timestamps, and dialogues between researchers and participants. This dataset contains 243 and 305 recordings and CHAT style transcriptions for control and dementia groups, respectively. throughout our experiments, we use the transcriptions for our training and evaluation. For our analysis, we specifically exclude certain special characters found in the conversations, such as “xxx", “(.)", “&-uh", “. exc", among others. Table 1 illustrates the process. Each of these symbols has its meaning, which can be found in the DementiaBank documentation [23].

2.2. In-Text Pause Encoding

In this research, the relationship between speech pauses and the likelihood of dementia is a key focus, a topic extensively explored in various studies [14,24]. These studies often classify different types of dementia based on the frequency and duration of pauses in speech.

We choose to use transcripts for our analysis rather than audio recordings. While audio data offers a richer source of information, its high dimensionality presents significant computational challenges. To circumvent these challenges and efficiently process the data, we rely on transcripts. There are various transcription tools available that can accurately convert speech to text, facilitating our analysis. These tools allow us to effectively analyze the data without the computational burden associated with processing high-dimensional audio files.

In the Cookie Theft dataset used in our study, specific special characters indicate different lengths of pauses: short, medium, and long. The symbols “(.)", “(..)", and “(...)" represent these pauses, respectively. To simplify processing, we substitute these symbols with more model-friendly terms: “ShPause", “MePause", and “LoPause". In [24], authors are using a similar concept to encode pauses within the text. Their approach is different from ours in how they incorporate encoded pauses in the text. Their methodology is applied to the ADReSS dataset [23], and they utilize temporal word alignment, which is not the case in our approach. Additionally, we analyze the impact of incorporating frequency of each pause type within the text.

We refer to the baseline model with the text input with all symbols removed as $B_0$. The models performed better when they were provided with a secondary numerical vector input corresponding to the frequencies of the pauses in the form

$$\left[\begin{array}{c}\hfill \#\mathrm{Sh}\end{array},\begin{array}{c}\hfill \#\mathrm{Me}\end{array},\begin{array}{c}\hfill \#\mathrm{Lo}\end{array}\right],$$

where #Sh, #Me and #Lo represent the number of short, medium and long pauses, respectively. All variants of the models discussed below include this vector as a secondary input and employ the architecture described in the next section.

We consider the combinations of 4 different ways of encoding pauses within the text: In-place (I), End-Sequence (S), Frequency (F), and Vector (V). As an example, consider the original text with pauses (after replacing the pause symbols with our previously defined terms) in the form:

$$“\mathrm{seg}1\begin{array}{c}\hfill \hspace{0.17em}\mathrm{ShPause}\end{array}\mathrm{seg}2\begin{array}{c}\hfill \hspace{0.17em}\mathrm{LoPause}\end{array}\mathrm{seg}3\begin{array}{c}\hfill \hspace{0.17em}\mathrm{ShPause}\end{array}\dots ”$$

where seg1 indicates a segment of text, and ShPause represents a short pause, and so on. We refer to this as a text with in-place encoding. A text without in-place encoding takes the form:

$$“\mathrm{seg}1\hspace{0.17em}\mathrm{seg}2\hspace{0.17em}\mathrm{seg}3\dots ”.$$

The end-sequence encoding creates a sequence of the pauses in the order in which they are present and takes the form:

$$“\begin{array}{c}\hfill \mathrm{ShPause}\end{array}\begin{array}{c}\hfill \hspace{0.17em}\mathrm{LoPause}\end{array}\begin{array}{c}\hfill \hspace{0.17em}\mathrm{ShPause}\end{array}\dots ”.$$

This sequence is concatenated at the end of the original text. The frequency encoding creates a text with the count of each pause type attached to each pause type. It takes the form:

$$\begin{array}{c}\hfill \#\mathrm{Sh}\end{array}+“\begin{array}{c}\hfill \mathrm{ShPause}\end{array}”+\begin{array}{c}\hfill \#\mathrm{Me}\end{array}+“\begin{array}{c}\hfill \mathrm{MePause}\end{array}”+\begin{array}{c}\hfill \#\mathrm{Lo}\end{array}+“\begin{array}{c}\hfill \mathrm{LoPause}\end{array}”,$$

and it also concatenated at the end of the original text. Finally, the vector encoding is similar to the frequency encoding except that it does not contain the pause type. It takes the form:

$$\begin{array}{c}\hfill \#\mathrm{Sh}\end{array}+“”+\begin{array}{c}\hfill \#\mathrm{Me}\end{array}+“”+\begin{array}{c}\hfill \#\mathrm{Lo}\end{array}+“”.$$

Our simplest model does not contain any of these encodings and is denoted as $E_{I_0,S_0,F_0,V_0}$. The model inputs that include in-place and frequency encoding will be denoted as $E_{I_1,S_0,F_1,V_0}$. Other inputs are denoted similarly. We will explore all 16 combinations of the pause encoding in Section 3. Note that the frequency and vector encodings contain the same information. They are used to verify that the LLM does process that information in the same way.

2.3. Modeling

As mentioned in Section 1, there are many models using LLMs to process textual data. In our experiments, we will be using BERT-base-uncased model [8] as the base for feature extraction. We conducted extensive experiments to find the best setup to optimize performance. As shown in Figure 1, our architecture has BERT as its base, followed by one dropout layer and two linear layers to perform classification. Also, to process the frequency vector, an encoder-decoder model is introduced. This model consisted of linear layers.

For our setup, we use cross-entropy as one of our optimization terms as follows

$$L_{cl}=\sum _{t}E(y_t,{\widehat{y}}_t),$$

where $E,y_t$, and ${\widehat{y}}_t$ are the cross-entropy loss, ground-truth label, and predicted label, respectively. For the auto-encoder, we introduce the following optimization term

$$L_F=MSE(X_F,{\widehat{X}}_F),$$

where $MSE$ is the mean square error loss, and $X_F$ and ${\widehat{X}}_F$ are true frequency input and its reconstruction, respectively. So, the total optimization cost becomes

$$L=L_{cl}+\lambda L_F,$$

where $\lambda$ is a hyper-parameter to be optimized.

Note that the baseline model does not have the auto-encoder network because processing the frequency vector is not part of the raw text for analysis. For all other experiments, this network is present.

3. Results and Discussion

This section details the outcomes of various experiments1 which utilized different in-text encoding methods, as outlined in Section 2.2, and various models discussed in Section 2.3. All experiments employed the BERT [8] base uncased model in conjunction with the AdamW optimizer [25] with default parameters, and each model was trained for 10 epochs. For all experiments, we used $\lambda =0.75$. To ensure the reliability of the results, a 20-fold cross-validation was conducted for each encoding scheme. The performance of the encodings scheme is presented in Table 2. The metrics used to evaluate performance include accuracy and f1-score, with the best result highlighted in red.

For the baseline, $B_0$, where no pause encoding is included, the model achieves $0.56$ and $0.42$ in accuracy and f1-score, respectively. When adding the numeric vector of counts as secondary input using our proposed architecture, the accuracy remains at $0.56$, but the f1-score increases to $0.45$. That is a 7% improvement on f1-score. However, the most significant improvements occur when the encoding are incorporated.

When just in-text encoding ($E_{I_1,S_0,F_0,V_0}$) is used the performance reaches $0.74$ and $0.77$ in accuracy and f1-score, respectively, which shows a significant improvement. For other encodings, the gap is even larger. The $E_{I_0,S_1,F_0,V_0}$ model achieved the highest performance for all the metrics overall with $0.86$ and $0.88$ for accuracy and f1-score. Compared to the baseline, the f1-score increases by a factor greater than 2.

Table 2. Performance for All Models. The best overall performance is highlighted in red.

Model	Acc.	f1
$B_0$	$0.56\pm 0.11$	$0.42\pm 0.27$
$E_{I_0,S_0,F_0,V_0}$	$0.56\pm 0.11$	$0.45\pm 0.33$
$E_{I_1,S_0,F_0,V_0}$	$0.74\pm 0.08$	$0.77\pm 0.06$
$E_{I_0,S_1,F_0,V_0}$	$\mathbf{0.86}\pm 0.06$	$\mathbf{0.88}\pm 0.05$
$E_{I_0,S_0,F_1,V_0}$	$0.81\pm 0.07$	$0.85\pm 0.06$
$E_{I_0,S_0,F_0,V_1}$	$0.83\pm 0.05$	$0.85\pm 0.04$
$E_{I_1,S_1,F_0,V_0}$	$0.80\pm 0.06$	$0.83\pm 0.05$
$E_{I_1,S_0,F_1,V_0}$	$0.84\pm 0.06$	$0.86\pm 0.06$
$E_{I_1,S_0,F_0,V_1}$	$0.81\pm 0.07$	$0.84\pm 0.06$
$E_{I_0,S_1,F_1,V_0}$	$0.81\pm 0.05$	$0.84\pm 0.05$
$E_{I_0,S_1,F_0,V_1}$	$0.82\pm 0.07$	$0.84\pm 0.05$
$E_{I_0,S_0,F_1,V_1}$	$0.83\pm 0.05$	$0.85\pm 0.05$
$E_{I_0,S_1,F_1,V_1}$	$0.83\pm 0.06$	$0.84\pm 0.06$
$E_{I_1,S_1,F_0,V_1}$	$0.82\pm 0.06$	$0.84\pm 0.05$
$E_{I_1,S_0,F_1,V_1}$	$0.82\pm 0.07$	$0.85\pm 0.06$
$E_{I_1,S_1,F_1,V_0}$	$0.81\pm 0.06$	$0.83\pm 0.05$
$E_{I_1,S_1,F_1,V_1}$	$0.83\pm 0.06$	$0.85\pm 0.05$

Table 2. Performance for All Models. The best overall performance is highlighted in red.

Model	Acc.	f1
$B_0$	$0.56\pm 0.11$	$0.42\pm 0.27$
$E_{I_0,S_0,F_0,V_0}$	$0.56\pm 0.11$	$0.45\pm 0.33$
$E_{I_1,S_0,F_0,V_0}$	$0.74\pm 0.08$	$0.77\pm 0.06$
$E_{I_0,S_1,F_0,V_0}$	$\mathbf{0.86}\pm 0.06$	$\mathbf{0.88}\pm 0.05$
$E_{I_0,S_0,F_1,V_0}$	$0.81\pm 0.07$	$0.85\pm 0.06$
$E_{I_0,S_0,F_0,V_1}$	$0.83\pm 0.05$	$0.85\pm 0.04$
$E_{I_1,S_1,F_0,V_0}$	$0.80\pm 0.06$	$0.83\pm 0.05$
$E_{I_1,S_0,F_1,V_0}$	$0.84\pm 0.06$	$0.86\pm 0.06$
$E_{I_1,S_0,F_0,V_1}$	$0.81\pm 0.07$	$0.84\pm 0.06$
$E_{I_0,S_1,F_1,V_0}$	$0.81\pm 0.05$	$0.84\pm 0.05$
$E_{I_0,S_1,F_0,V_1}$	$0.82\pm 0.07$	$0.84\pm 0.05$
$E_{I_0,S_0,F_1,V_1}$	$0.83\pm 0.05$	$0.85\pm 0.05$
$E_{I_0,S_1,F_1,V_1}$	$0.83\pm 0.06$	$0.84\pm 0.06$
$E_{I_1,S_1,F_0,V_1}$	$0.82\pm 0.06$	$0.84\pm 0.05$
$E_{I_1,S_0,F_1,V_1}$	$0.82\pm 0.07$	$0.85\pm 0.06$
$E_{I_1,S_1,F_1,V_0}$	$0.81\pm 0.06$	$0.83\pm 0.05$
$E_{I_1,S_1,F_1,V_1}$	$0.83\pm 0.06$	$0.85\pm 0.05$

A possible reason for the success of this method might be attributed to how these encoded pause characters create recognizable patterns in the text, simplifying the task for the model to identify and classify the inputs. Additionally, the occurrence frequency and distribution of each pause type within the text might serve as a distinctive feature, helping the model differentiate between the dementia group and the control group. This distinction could significantly contribute to the model’s ability to accurately separate these groups. Also, it should be mentioned that our proposed encoding scheme resulted in more stability in performance as could be observed in Table 2. All the models with encoding have a smaller standard deviation, which shows the robustness that these encodings introduce to the model. Perhaps illustrating the power of transfer learning from an LLM.

Table 3 shows the aggregate difference between models with and without an encoding. The best performance take place when frequency encoding $E_{F_1}$, vector encoding $E_{V_1}$ and end-sequence encoding $E_{S_1}$ are present. All get an average f1 score of 0.85. This is consistent with the best mode performance found in Table 2, and validates the fact that these models contain very similar information. We observe that the averages for models that include an encoding are smaller than those that do include an encoding (e.g., $E_{I_1}$ has higher performance overall than $E_{I_0}$). This difference is due to the low performance of $E_{I_0,S_0,F_0,V_0}$ which contains no encoding.

4. Conclusions

We implemented and analyzed a novel encoding strategy within the text, incorporating special characters to enhance the performance of the model. This method, as evidenced by the results in Table 2, outperformed the baseline model, which did not include in-text pause encoding. Additionally, this strategy demonstrated greater stability across various experiments, indicated by its lower variance. However, further research is necessary to fully understand the limitations and potential applications of this approach across diverse datasets and utilize other special language cues such as verbs, nouns, and repetition of “uh" and “um" utterances in the text.