End-to-End Multi-Modal Speaker Change Detection with Pre-trained Models

1. Introduction

Speaker change detection (SCD) is an important task in audio processing, particularly for applications such as speech segmentation, speaker diarization, and automatic transcription. Accurate detection of speaker transitions is critical in multi-person conversations, especially in meeting scenarios, where challenges such as overlapping speech and background noise can make correct detection difficult.

Traditional SCD methods rely on audio features, including pitch, Mel Frequency Cepstral Coefficient (MFCC), and cepstral coefficients, with various approaches such as metric-based and neural network-based methods. However, these approaches often under-perform in complex acoustic environments, such as those with low signal-to-noise ratios or simultaneous speech. Other multi-modal approaches to SCD have been explored, but they face practical limitations. Their approaches rely on ground-truth transcripts, which are often impractical in real-world scenarios due to transcription errors and limited availability. Another issue is that binary cross entry loss (BCE loss) is used as main objective function, which make model fails to learn useful features from imbalanced data. Usually SCD dataset contains many negative samples (no change points) and less positive samples (change points), it has imbalanced features.

To address these challenges, we propose an end-to-end multi-modal SCD model with a self-attention layer, based on large pre-trained models, and enhance it with focal loss to mitigate the issue of imbalanced data. The model employs a pre-trained large audio model to extract acoustic features from raw audio and a large language model (LLM) to derive semantic and contextual information from transcripts generated by an automatic speech recognition (ASR) system. By leveraging both modalities in a practical, real-world setting, this approach enhances detection accuracy in challenging acoustic conditions while avoiding the reliance on manually annotated transcripts. Furthermore, this study investigates the impact of different configurations, including audio-only, text-only, and multi-modal setups, to evaluate their effectiveness in SCD. A comparison between BCE loss and different focal loss configurations is carried out. Additionally, we explore different fine-tuning strategies, examining the impact of updating the pre-trained models to optimize performance while balancing computational efficiency. The evaluation is conducted using the multi-talker meeting corpus [1], a widely recognized dataset in speech and audio processing research. Experimental results showed that the updating the pre-trained model’s parameters during the training outperform significantly than freezing them for all modalities. Compared to the text-only model, the audio-only model outperforms it in various settings, and the multi-modal configuration surpasses both. Self-attention layer introduced in the approach effectively improve the F1 score compared to the model without using it for various modalities. The proposed approach outperforms the baselines and also outperforms comparable results to the previous model enhanced with multi-modal.

The rest of the work is organized as follows: (i) Section 2 reviews related studies on SCD, including multi-modal approaches. (ii) Section 3 presents the approach, including task formalization and the proposed model. (iii) Section 4 provides details on the experiments, including model setup, data statistics, baselines, and a discussion of the results. (iv) Section 5 discusses the limitations of the approach and potential future directions. (v) Section 6 concludes the work with a summary of the key findings.

2. Related Work

Existing approaches to SCD problems can be categorized into two groups: i) unsupervised and ii) supervised approaches.

Most unsupervised approaches are considered metric-based methods that calculate the discrepancy between two consecutive frames or segments. A change point is detected if the distance between the two segments exceeds a predefined threshold, otherwise, it is considered that no change point exists. In this direction, one of the simplest metric-based approaches [2] is to use a distance function, such as Kullback-Leibler (KL) divergence [3], to measure the similarity between two consecutive segments modeled with Gaussian mixture models (GMMs). It is very sensitive to a predefined threshold, difficult to generalize to unseen audio data. Early methods to SCD are the combination of GMMs with Bayesian information criterion (BIC) [4,5,6]. Specifically, each segment will be modeled with separate GMMs (referred to as two speaker models), and two consecutive segments will be modeled with a GMM (refers as single speaker model). BIC is used to calculate a score that estimates how well these GMMs will predict the given segments. Generalized Likelihood Ratio (GLR) [7] can also be applied with GMMs. It measures the likelihood ratio between two distributions of these segments. Studies have shown that using neural network-based speaker embeddings [8,9] outperforms traditional audio features such as mel-frequency cepstral coefficients (MFCCs), pitch, and filter banks in SCD. In the work [10], the authors applied a neural network to calculate speaker embeddings at the frame level such as d-vector, and used it with metric-based segmentation to detect speaker changes. It achieved a fast and effective SCD model using deep speaker vectors.

In supervised direction, many machine learning methods have been applied to SCD, such as conditional random field (CRF)

3.2. Multi-Modal SCD Model

Speaker change detection in audio streams can be addressed using a multi-modal approach that integrates audio and text data. This model employs two pre-trained large models: i) large audio models to extract acoustic features from raw audio and ii) large language model to derive semantic information from the transcript. The combination of these modalities enhances the accuracy of detecting speaker transitions.

Audio Feature Extraction.

To extract robust audio representations, we leverage a pre-trained model that transforms raw waveforms into sequences of latent acoustic representations. This model encodes segment-level hidden states that capture meaningful speech features. To refine these representations, we apply self-attention to enhance contextual dependencies, followed by a pooling operation to obtain fixed-size feature vectors.

The raw input waveform X is processed into frame-wise hidden states Z generated by a pre-trained model, where each frame contains a high-dimensional vector. These representations are then refined through a self-attention mechanism, producing updated embeddings. To obtain a compact representation, temporal mean pooling is applied, resulting in a fixed-size vector ${\mathbf{h}}^{\mathrm{audio}}$.

Text Feature Extraction.

The audio is segmented into 1-second intervals, and an automatic speech recognition (ASR) system is applied to generate transcripts for each segment, rather than relying on ground truth transcriptions which are impractical for real-world applications. The text processing leverages a pre-trained large language model, Bert [18], its contextual embeddings. The transcript is tokenized and input into Bert, producing embeddings for each token. The embedding of the [CLS] token is selected to represent the sentence’s semantic content:

$${\mathbf{h}}^{\mathrm{text}}={\mathbf{z}}_{\mathrm{CLS}},$$

(1)

where ${\mathbf{z}}_{\mathrm{CLS}}$ encapsulates the overall meaning of the text.

Feature Fusion and Classification.

To integrate audio and text modalities, a simple concatenation is performed between the two. The resulting audio and text embeddings are then concatenated:

$${\mathbf{h}}^{\mathrm{fusion}}=[{\mathbf{h}}^{\mathrm{audio}};{\mathbf{h}}^{\mathrm{text}}],$$

(2)

where $[·;·]$ indicates concatenation.

The final audio representation serves as input to a fully connected neural network for classification. The classifier consists of three linear layers with decreasing dimensions. Each linear transformation is followed by layer normalization to stabilize training and improve generalization. Non-linearity is introduced using the ReLU activation function, and dropout regularization is applied after each hidden layer to mitigate overfitting. The final output is passed through a sigmoid activation function to produce a probability score for the classification task.

To optimize the model, we employ focal loss, which is designed to address class imbalance by down-weighting well-classified examples and focusing more on hard-to-classify instances. Given the predicted probability and the ground-truth label, the focal loss is computed as:

$${\mathcal{L}}_{\mathrm{focal}}=\alpha {(1-p_t)}^{\gamma }{\mathcal{L}}_{\mathrm{BCE}},$$

where ${\mathcal{L}}_{\mathrm{BCE}}$ is the binary cross-entropy loss, $p_t$ represents the predicted probability for the true class, $\alpha$ is a weighting factor that balances the contribution of positive and negative samples, and $\gamma$ controls the down-weighting of easy examples.

By combining structured feature extraction with a robust classification strategy, the model effectively learns meaningful audio representations while addressing class imbalance in the dataset.

4. Experiments

4.1. Model Setup

The model utilizes pre-trained Wav2Vec2 and Bert, specifically "wav2vec2-base" and "bert-base-cased" variants. Training employs the AdamW optimizer with a learning rate scheduler, ensuring stable updates and effective convergence. The base model components (Wav2Vec2 and Bert) use a learning rate of $1\times {10}^{-5}$, while the classifier has a higher learning rate of $1\times {10}^{-3}$. A weight decay of $0.01$ is applied for regularization. To enhance stability, a warmup ratio of $0.1$ is used, meaning the learning rate gradually increases during the first 10% of total training steps before following a linear decay. Training is conducted for 10 epochs, with the total number of training steps computed as the product of the number of epochs and the length of the training dataloader. Additionally, a random seed is set to ensure reproducibility across experiments.

The classifier is designed to process the features, with a hidden size of 768. The head of self-attention is set to 8. The classifier consists of three fully connected layers with 512 and 256 hidden units, each followed by layer normalization, ReLU activation, and dropout (0.4), leading to a final output layer for classification.

Each of the three configurations: multi-modal, text-only, and audio-only supports two training modes: i) a freeze mode (F) and ii) a non-freeze mode (NF). In freeze mode, all Wav2Vec2 and Bert parameters are fixed, and only the classification layers are trained. In non-freeze mode, the configurations differ as follows: the both base models with its classification layers are trained. For focal loss of the model, $\alpha$ is set to 0.31, and $\gamma$ is set to 2.0. For convenience in naming, we use eMD to refer to the proposed approach. Using the different settings described above, we define two versions of eMD: i) A multi-modal model without self-attention and cross-attention layers, which consists only of the three fully connected layers mentioned earlier for classification. This model is denoted as eMD. ii) A model with self-attention layers, which enhances feature interaction. This version is denoted as eMD${‌}_{att.}$

4.2. Baselines

To check how well the proposed models work, we compare them with these baselines:

i) Neural Network (NN) Baseline: This model takes a flattened vector of 13-dimensional MFCC features as input. It has fully connected layers with non-linear activation functions. Then, a sigmoid output layer predicts the chance of a speaker change. This baseline captures feature patterns in each segment but does not model time relations between segments.

ii) Bidirectional Long Short-Term Memory (BiLSTM): This model takes sequences of 13-dimensional MFCC feature vectors. Its bidirectional structure helps capture both past and future time relations. BiLSTM outputs are pooled over time to form a fixed-size representation. Then, a fully connected layer with a sigmoid activation is used for classification.

iii) Fine-Tuned Approach with Multilayer Perceptron (MLP): This model is a strong baseline similar to this work. It uses only audio and has a single linear classification layer. It refers as to Wav2Vec2-MLP.

4.3. Dataset Statistics

Experiments were conducted using the AMI meeting corpus [1], a well-established audio dataset commonly utilized in speech and audio processing research. The AMI corpus provides a diverse collection of meeting recordings with noisy, interrupt etc, allowing for the evaluation of the model under a variety of conditions and scenarios. The dataset used for training and evaluating the multi-modal speaker change detection model is detailed in two tables.

Table 1 presents the overall statistics of the training dataset, which includes 137 meetings with an average of 790.99 change points per meeting. The dataset exhibits an average of 306.61 overlap points, indicating instances of simultaneous speech, and an average of 3.99 speakers per meeting. The average length of each meeting is 34.11 minutes, with a total duration of 4673.69 minutes. The class distribution shows that 30.85% of the data are positive instances (indicating speaker changes), while 69.15% are negative instances (no speaker changes).

Table 2 provides statistics for the test set, covering eight meetings identified by specific IDs (e.g., EN2002a, EN2002b, etc.). It reports the total number of change points, overlap points, and speakers per meeting, as well as the audio length in minutes. For example, the EN2002a meeting has 1366 change points, 678 overlaps, 4 speakers, and a duration of 43.90 minutes. The test set averages 821.31 change points, 328.19 overlaps, 3.94 speakers, and 33.38 minutes per meeting. Additionally, the class distribution for each meeting is provided, with percentages for positive (P) and negative (N) classes. On average, 31.40% of the test set instances are positive, and 68.60% are negative, reflecting a similar imbalance to the training set. These statistics highlight the complexity and variability of the dataset, particularly due to overlapping speech and the distribution of speaker changes.

4.4. Evaluation Metrics

To assess the performance of the baseline and proposed methods, several metrics are employed. The false alarm rate (FAR) measures the proportion of incorrect speaker change detections, calculated as the ratio of false positives to all non-change points:

$$\mathrm{FAR}={\displaystyle \frac{\mathrm{False}\mathrm{Positives}}{\mathrm{False}\mathrm{Positives}+\mathrm{True}\mathrm{Negatives}}}$$

(3)

The missed detection rate (MDR) evaluates the frequency of undetected speaker changes, determined as the ratio of false negatives to all true change points:

$$\mathrm{MDR}={\displaystyle \frac{\mathrm{False}\mathrm{Negatives}}{\mathrm{True}\mathrm{Positives}+\mathrm{False}\mathrm{Negatives}}}$$

(4)

The Hit rate measures how well the model correctly identifies actual speaker change points.

$$\mathrm{Hit}={\displaystyle \frac{\mathrm{True}\mathrm{Positive}}{\mathrm{Total}\mathrm{Actual}\mathrm{Positives}}}$$

(5)

Additionally, precision is used to indicate the accuracy of detected speaker changes, recall assesses the ability to identify all true speaker changes, and the F1 score provides a balanced measure of precision and recall. These metrics collectively offer a comprehensive evaluation of the model’s effectiveness in speaker change detection. It should be noted that the average results (Precision, Recall, F1-score, FAR and MDR) reported below are calculated using the Macro method.

4.5. Results

4.5.1. Results of eMD Model

Table 3 and Table 4 detail the performance of eMD model for SCD using freeze and non-freeze models across three modalities: Text, Audio, and Multi-modal.

The Multi-modal approach consistently outperformed single modalitie configurations in both Freeze and Non-Freeze settings. For instance, in the Freeze models, the Multi-modal modality achieved a Hit Rate of 79.88% in meeting EN2002a (45.51% P, 54.49% N), compared to 72.28% for Text and 78.34% for Audio. In Non-Freeze models, the Multi-modal Hit Rate reached 82.98% in EN2002d (40.20% P, 59.80% N), surpassing Text (70.43%) and Audio (82.26%) modalities. Non-Freeze models exhibited marked improvements over Freeze models, particularly in Audio and Multi-modal settings. The FAR in Audio modality decreased from a range of 19.08%–40.48% in Freeze models to 8.32%–25.99% in Non-Freeze models. Similarly, the Multi-modal MDR reduced from 20.12%–30.53% (Freeze) to 17.02%–33.33% (Non-Freeze), indicating enhanced detection accuracy.

The class imbalance significantly impacted the Text modality in Freeze models, resulting in elevated MDR and reduced Hit Rates. For example, in meeting ES2004b (25.46% P, 74.54% N), the Text modality recorded an MDR of 43.72% and a Hit Rate of 56.28%, reflecting a bias toward the majority negative class. This imbalance compromised the detection of speaker changes. Non-Freeze models, particularly Multi-modal, effectively addressed these challenges. In ES2004b, the Multi-modal Non-Freeze model improved the Hit Rate to 72.53% and reduced the MDR to 27.47%, demonstrating the advantage of adaptive learning and multi-modal integration in handling imbalanced distributions.

Figure 2 compares the performance of Freeze (F) and Non-Freeze (NF) models across three setups of eMD model: text-only, audio-only, and multi-modal. In part Figure 2, the stacked bar chart shows the average FAR and MDR. The hatched bars represent FAR, while the solid bars represent MDR. The light blue bars correspond to Freeze mode, and the red bars correspond to Non-Freeze mode. Generally, the multi-modal configuration shows a lower FAR and MDR compared to the text-only and audio-only setups. All Non-Freeze models outperform the Freeze models in all three settings. In the audio-only mode, it can be seen that the FAR decreases significantly from 31.49% to 25.98%, while the MDR drops from 25.24% to 16.93%.

Part Figure 2 shows the averaged results of F1 scores. The text-only configuration achieves an F1-score of 54.22% with Freeze and 55.26% with Non-Freeze. The audio-only configuration records an F1-score of 60.51% under Freeze and 62.58% under Non-Freeze. The multi-modal configuration performs best, with an F1-score of 70.54% for Freeze and 70.91% for Non-Freeze. The results demonstrates that the multi-modal approach outperforms both text-only and audio-only configurations, with a slight improvement in Non-Freeze mode.

4.5.2. Results of eMD${‌}_{att}$ Model

Table 5 and Table 6 shows the results of eMD${‌}_{att}$ model. It applies a self-attention layer only to the multimodal and audio-only configuration, as previous experiments showed that adding self-attention to the text-only models did not improve performance. In the Freeze (F) mode (Table 5), where feature extraction layers remain unchanged, the audio-only setup outperforms the text-only setup. This suggests that audio inherently provides more informative cues for speaker change detection than text. The multi-modal setup performs best, leveraging both modalities to achieve the highest Hit Rates across all meetings.

In the Non-Freeze (NF) mode Table 6, where feature extraction layers are fine-tuned, overall performance improves. The text-only model sees a reduction in FAR from 87-92% to 77-87%, and MDR drops to 9.07% in the best case. The audio-only model continues to outperform text-only, with FAR decreasing to 16.80% (IS1009c) and 17.23% (IS1009b), and MDR remaining below 14%. The multi-modal approach remains the most effective, maintaining FAR around 20% and MDR between 9–16%. The multi-modal setup achieves the most significant improvements across the most imbalanced meetings (TS3003c, IS1009c, ES2004b). Fine-tuning substantially reduces FAR, dropping from 57.93% to 21.06% in TS3003c, from 76.07% to 14.04% in IS1009c, and from 78.26% to 21.97% in ES2004b. These results suggests that utilizing both text and audio in a fine-tuned model can help reduce the impact of class imbalance, making multimodal processing a promising approach for speaker change detection in imbalanced scenarios.

The precision, recall and F1 score for eMD and eMD${‌}_{att}$ model reported in the Appendix A.

Figure 3 compares the performance of Freeze (F) and Non-Freeze (NF) models across three setups of eMD${‌}_{att}$ model: text-only, audio-only, and multi-modal. In part Figure 3, the stacked bar chart shows the average FAR and MDR. The hatched bars represent FAR, while the solid bars represent MDR. The light blue bars correspond to Freeze mode, and the red bars correspond to Non-Freeze mode. It is clearly shown that eMD${‌}_{att}$ using attention layer the fine-tuned model (NF) reduce both FAR and MDR significantly compared to Freeze model. For example, the multimodal with Freeze mode achieves high FAR of 69.81% and a low MDR of 10.54, however, fine-tuned (the red bar) trade-off between these two rates and reduce false alarm from 69.81% to 21.78%. Part Figure 3 reports the F1-score for these models, and it shows that the highest F1 score 73.39% was obtained by multi-modal with NF.

4.5.3. Optimization Parameters for Focal Loss

In order to test how focal loss [19] influence the model’s performance in SCD imbalanced situation, we predefine various value of parameters of focal loss, such as $\alpha =[0.1,0.2,0.3,0.4]$ and $\gamma =[0,1,2,3]$. The former $\alpha$ is weighting factor that balances class importance in imbalanced datasets. A lower $\alpha$ (< 0.5) gives more weight to the majority class (non-change points), while a higher $\alpha$ (> 0.5) gives more weight to the minority class (speaker change points). $\gamma$ is a focusing parameter that reduces the weight of easy samples, making the model focus more on hard-to-classify examples. A higher $\gamma$ increases this effect, but too high a value can cause training instability.

We selected a highly imbalanced meeting from the training set to analyze different parameter combinations and reduce training time. Figure 4 presents the curves for different combinations of values of $\alpha$ and $\gamma$. Traditional Binary Cross-Entropy (BCE) loss is also included for comparison and is denoted as BCEloss. Figure 4 and Figure 4 illustrate how the FAR and MDR change with different values of $\alpha$ and $\gamma$. It can be observed that $\gamma =0$ and $\gamma =1$, when using BCEloss, achieve comparably low FAR scores. Among them, when $\gamma =0$, the loss function becomes weighted BCE loss, which performs slightly better than the others. From Figure 4, it can be seen that $\gamma =1$ with BCEloss results in a higher MDR compared to other settings. Figure 4 displays F1 scores across different settings, clearly showing that focal Loss outperforms BCEloss. When $\gamma =1$ (orange curve) and $\alpha =0.3$, the model achieves the highest F1 score. These values are chosen for focal Loss, and the final model is trained using this configuration, denoted as eMD${‌}_{att}^{*}$.

4.5.4. Comparison with Baselines

The performance of eMD and eMD${‌}_{att}$ models is evaluated in Table 7, achieving improvements of up to 2.73% in F1-Score over the baseline Wav2vec-MLP. The comparison includes baseline models: Neural Network (NN), Long Short-Term Memory (LSTM), and Wav2vec-MLP. The NN struggles with a high missed detection rate, while LSTM shows improvement but remains weak in detecting speaker changes. Wav2vec-MLP provides a stronger baseline with higher precision, though it still misses some detections.

eMD model, tested in audio-only and multi-modal modes, improves recall compared to Wav2vec-MLP, but its false alarm rate increases. The multi-modal setup of eMD offers a slight recall boost over the audio-only version, with similar false alarm rates, suggesting limited additional benefit. eMD${‌}_{att}$ model, incorporating attention, further enhances recall in both audio-only and multi-modal modes, reducing missed detections despite a higher false alarm rate. The multi-modal eMD${‌}_{att}$ configuration performs best, achieving the lowest missed detection rate (14.15%) and highest recall (85.85%) among all models, with an F1-Score of 73.39%, surpassing Wav2vec-MLP’s overall score.

Compared to baselines, eMD${‌}_{att}$ multi-modal approach stands out, leveraging attention and multi-modal data to outperform Wav2vec-MLP, with notable improvements in recall (its is similar to hit rate, an important metric for SCD, because it measures how well the model captures actual speaker change points) and reduced missed detections.

4.5.5. Comparison with Previous Work

Table 8 compares the proposed approach with previous multi-modal methods and a pitch-based approach. However, a direct comparison is not possible due to differences in experimental settings. For instance, Zhao et al. [17] use a custom definition of precision and recall, making the task easier. Other studies apply forgiveness collars when identifying change points. Despite these differences, Table 8 shows that the proposed approach performs competitively and achieves better results than previous multi-modal methods.

5. Limitations

Despite the effectiveness of our approach, there are several limitations that could impact performance. First, we rely on a hard alignment between text and audio, which may introduce errors when transcription and audio segmentation are not perfectly synchronized. This can lead to misaligned speaker change points, affecting downstream processing. Second, the pre-trained Bert model used in our approach does not inherently contain speaker information. As a result, the textual representation lacks explicit speaker cues, which could otherwise improve the detection of speaker transitions.

A more effective approach would involve a joint model that simultaneously performs automatic speech recognition and speaker change detection, reducing alignment errors. Alternatively, a stacked model architecture, where ASR is followed by SCD in a sequential manner, could improve robustness by leveraging intermediate speech representations. These models can be integrated with audio pre-trained models to further improve performance. Future research should investigate joint modeling of automatic speech recognition and speaker change detection to improve alignments between text and audios or use hidden embeddings from pre-trained ASR model to improve SCD performance.

6. Conclusions

In this work, we proposed a multi-modal Speaker Change Detection (SCD) approach that integrates audio and text information to improve detection. The proposed method leverages pre-trained large-scale models for feature extraction and incorporates a self-attention mechanism to refine contextual representations. The features are combined and passed through a fully connected classification network with layer normalization and dropout to make the model more stable. The model also uses focal loss to handle class imbalance of the data.

We tested the model on a speaker change detection task using audio-text pairs from a multi-talker meeting dataset. The experiments included comparing multi-modal and single-modal models, checking the effect of fine-tuning pre-trained models, and testing the impact of self-attention layers. The results show important findings. The multi-modal approach works better than single-modality models, which shows that audio and text together help detect speaker changes. Fine-tuning pre-trained models makes the model much better than freezing them. Updating the pre-trained model’s parameters during training gives a 21% improvement in the multi-modal setting. Adding a self-attention layer improves feature learning, which boosts performance by about 2% compared to a model without it. Despite these improvements, the approach has certain limitations. The reliance on hard alignment between ASR-generated text and audio segments, and the text information is used as symbolic first, then turn it to embeddings which lack speaker-specific information, reducing their effectiveness in capturing speaker transitions based solely on textual cues. To address these limitations, future research should explore jointly performing automatic speech recognition and speaker change detection to reduce alignment errors. Alternatively, a stacked ASR-to-SCD pipeline could leverage intermediate ASR representations for better speaker change detection.