Test-Time Adaptation for Personal Voice Activity Detection: VAD-Gated Test-Time Training and Speaker Embedding Adaptation

1. Introduction

Voice-based human–machine interaction has become a central interface in modern smart devices, creating sustained demand for robust and efficient speech processing front-ends. Voice Activity Detection (VAD), which classifies incoming audio frames as speech or non-speech, is a fundamental preprocessing stage for downstream systems such as automatic speech recognition and voice assistants [1,2]. Beyond filtering out non-speech regions, VAD also plays an important role in reducing power consumption and managing computational resources on resource-constrained devices.

Conventional VAD, however, cannot determine whether detected speech belongs to a specific target speaker—a limitation that becomes critical in multi-speaker environments. Personal Voice Activity Detection (PVAD) was introduced to address this gap, extending the binary VAD formulation into a three-way classification over target speaker speech (tss), non-target speaker speech (ntss), and non-speech (ns) [3]. By conditioning on a target speaker embedding derived from a speaker verification model, PVAD systems can respond selectively to a designated speaker while suppressing competing voices, making them well suited to personal voice assistant applications. A substantial body of work has refined how speaker embeddings are fused with acoustic features [4,5,6,7], culminating most recently in the Flexible Dynamic Encoder RNN (FDE-RNN) [8] and its selective state-space successor, FDE-Mamba [9], which together represent the current state of the art on this benchmark.

Despite these architectural advances, essentially all PVAD systems share a common deployment assumption: once trained, the model is frozen and operates with a fixed, pre-enrolled speaker embedding. The model parameters do not change in response to the audio actually encountered at inference time, and the speaker representation is computed once from enrollment speech and then held constant.

This static design is particularly consequential for PVAD, more so than for ordinary VAD. Because PVAD makes a speaker-conditioned decision, its accuracy depends not only on how well the model separates speech from non-speech, but also on how faithfully the fixed enrollment embedding represents the target speaker as they actually sound in the test utterance, and on whether the acoustic conditions of that utterance resemble those seen during training. Both assumptions are fragile in practice. The acoustic environment at test time—noise, reverberation, recording channel, and device—routinely differs from the training distribution, shifting the feature statistics on which the speaker-discrimination decision is based. A speaker’s voice is itself non-stationary: speaking rate, vocal effort, emotional state, and health vary across sessions, so an embedding enrolled on one occasion may be a poor match for the same speaker on another. The enrollment utterance may also be short, noisy, or recorded under conditions unlike the test signal, yielding an embedding that is biased from the outset. In each of these cases the mismatch falls precisely on the speaker-conditioning pathway that distinguishes PVAD from VAD, and a model that cannot adapt at inference time has no recourse: it must rely entirely on the generalization acquired during offline training, even when the test utterance in front of it carries information that could be used to correct the mismatch. This observation motivates asking whether PVAD can instead adapt to each test utterance as it is processed.

A growing line of research addresses exactly this problem through test-time adaptation, a family of methods in which a trained model adjusts itself to the test distribution using only the unlabeled test signal, without any ground-truth labels. The central difficulty is that adaptation must be driven by a signal that is available at inference, and different methods differ in what self-supervised signal they use and what they update. Entropy minimization, exemplified by TENT [10], updates a small set of model parameters—typically the affine parameters of normalization layers—so as to reduce the entropy of the model’s own predictions, on the premise that confident predictions tend to be correct ones; it has been applied to speech recognition and to speaker adaptation for lip reading [11], usually resetting the adapted parameters after each utterance. Test-Time Training (TTT) takes a different route. Introduced by Sun et al. [12] to combat train–test distribution shift, it pairs the main task with an auxiliary self-supervised task during training; at inference, the self-supervised task is used to fine-tune the model on each test input before the main prediction is made, so that the representation is specialized to the local statistics of that input. A recent reformulation [13] casts TTT as a sequence-modeling layer whose hidden state is itself a small machine-learning model: the hidden state is a weight matrix $\mathbf{W}$, and the update rule is a step of gradient descent on a self-supervised reconstruction loss applied at every position, so that $\mathbf{W}$ keeps learning even as it processes a test sequence. This view is appealing for a streaming, speaker-conditioned task such as PVAD for two reasons. First, the self-supervised reconstruction signal is computed directly from the incoming features, so adaptation is driven by the very utterance whose speaker decision is at stake—exactly the information a frozen model discards. Second, because the adapted state is a weight matrix updated online, it can absorb a distribution shift expressed jointly across many feature dimensions, rather than requiring the shift to be attributed to any single pre-specified factor. TTT has shown benefits in language modeling [13], computer vision [12], and more recently in speech tasks such as speech enhancement [14] and speech editing, where adapting to unseen noise or speaker conditions at inference improves output quality.

To date, however, the behavior of these adaptation mechanisms on PVAD has not been investigated. PVAD differs from previously studied tasks in two respects that make this an open question rather than a foregone conclusion: it is an inherently speaker-dependent task, and its decision depends jointly on the acoustic signal and an externally supplied speaker representation. It is therefore not obvious a priori whether test-time adaptation helps, how large any benefit is, or—given that PVAD has two adaptable surfaces, the internal features and the speaker embedding—which part of the system should adapt. These are empirical questions specific to PVAD, and answering them is the purpose of this study.

In this work we conduct an empirical study of test-time adaptation for PVAD, built on the FDE-Mamba backbone. We investigate two complementary mechanisms. The first, a VAD-gated TTT adapter, instantiates the TTT-Linear formulation [13] within the personalization pathway, adapting an internal weight matrix on the speaker-conditioned feature stream of each utterance. Crucially, this adapter operates on features after speaker-embedding fusion, so the distribution shift it absorbs reflects the combined influence of acoustic conditions, speaking style, and speaker characteristics on the test signal, rather than any single factor in isolation. We tailor the adapter to PVAD with a VAD-probability gate, which restricts adaptation to frames classified as speech, and an exponential moving-average update, which stabilizes the weight trajectory during inference. The second mechanism, TEA (Test-time Embedding Adaptation), performs test-time adaptation of the target speaker embedding: with all model parameters frozen, it updates the d-vector itself through self-supervised objectives, directly targeting the enrollment–test mismatch. Throughout, we treat the study as a measurement exercise: we quantify each component’s contribution independently and in combination, report a recall–precision trade-off that adaptation introduces, and separately isolate the effect of a post-hoc Gaussian smoothing step so that its contribution is not conflated with that of the adaptation mechanisms.

The contributions of this paper are as follows:

• To the best of our knowledge, this is the first work to apply Test-Time Training to PVAD. We adapt the TTT-Linear formulation with a VAD-probability gate and an exponential moving-average update tailored to the speech-dependent, streaming nature of PVAD, and we clarify that the adapter absorbs a composite distribution shift over the speaker-conditioned feature stream rather than adapting the speaker embedding alone.
• We introduce TEA (Test-time Embedding Adaptation), a test-time speaker embedding adaptation scheme for PVAD. While entropy-minimization-based test-time adaptation has been studied for other speech tasks, we are, to our knowledge, the first to apply it to PVAD and, distinctively, to adapt the target speaker d-vector itself rather than the model’s internal parameters.
• We present a systematic empirical study that quantifies the independent and combined effect of each mechanism, analyzes the recall–precision trade-off they induce, isolates the contribution of post-hoc Gaussian smoothing, and shows that the observed behavior holds across both an LSTM-based (FDE-RNN) and a Mamba-based (FDE-Mamba) backbone. We report the gains as consistent but modest, and discuss the limitations that this implies for adaptive PVAD.

The remainder of this paper is organized as follows. Section 2 reviews PVAD and test-time adaptation. Section 3 describes the FDE-Mamba backbone and the two adaptation mechanisms. Section 4 details the experimental setup, and Section 5 reports the main results against the FDE-Mamba backbone, an ablation of each component, the computational cost, and a generalization check on the FDE-RNN backbone, followed by a discussion of limitations. Section 6 concludes.

2. Related Work

2.1. Personal Voice Activity Detection

Personal Voice Activity Detection extends conventional VAD by incorporating target speaker information, enabling the system to distinguish target speaker speech (tss) from non-target speaker speech (ntss) and non-speech (ns). The seminal work of Ding et al. [3] introduced a speaker-conditioned framework that fuses a d-vector speaker embedding [15] with acoustic features for ternary frame-level classification. Personal VAD 2.0 [4] subsequently reduced model complexity for on-device speech recognition while maintaining competitive performance.

Later work explored richer strategies for integrating speaker and acoustic information. AS-pVAD [6] proposed a frame-wise attentive score loss to sharpen target-speaker detection; COIN-AT-PVAD [16] introduced conditional intermediate attention within a stacked encoder–decoder; Makishima et al. [17] studied enrollment-free training to remove the dependency on explicit enrollment utterances; and SE-PVAD [18] replaced log-Mel features with a learnable extractor for improved input discrimination. Most recently, FDE-RNN [8] proposed a flexible dynamic architecture that decouples VAD and PVAD into a shared Dynamic Encoder RNN backbone and a detachable personalization module, using a gating-based skipping mechanism to bypass redundant computation on non-speech frames. FDE-Mamba [9] subsequently replaced the LSTM components of FDE-RNN with selective state-space (Mamba) [19] blocks, improving accuracy and reducing the real-time factor while preserving the dynamic gating design.

A common thread across these systems is a doubly static deployment assumption: the speaker embedding is computed once from enrollment speech and then held fixed, and the model itself is frozen after training. The first assumption leaves the system unable to correct a mismatch between the enrollment embedding and the target speaker as they actually sound at test time; the second leaves it unable to adapt to the acoustic conditions of the test utterance—noise, reverberation, channel, or speaking style—when these differ from the training distribution. Robustness to such adverse conditions has been pursued primarily through training-time interventions, such as multi-condition training or self-supervised pretraining [20], which must anticipate the test distribution in advance. The complementary question of whether a trained PVAD model can instead be adapted at inference time, using the test signal itself—adapting both its internal representation to the acoustic conditions of each utterance and its speaker representation to the target—has received little attention, and it is this question that the present study addresses.

2.2. Test-Time Adaptation and Test-Time Training

Test-time adaptation denotes a family of methods that adjust a trained model to the test distribution using only the unlabeled test data encountered at inference, without ground-truth labels. Methods in this family are distinguished by two design choices: the self-supervised signal that drives adaptation, and the subset of quantities that are updated. We organize the relevant literature along these axes.

TTT as a distribution-shift remedy. Test-Time Training was introduced by Sun et al. [12] to combat train–test distribution shift. The model has a Y-shaped architecture: a shared feature encoder with parameters ${\theta }_e$ feeds two branches—a main-task head ${\theta }_m$ and a self-supervised auxiliary head ${\theta }_s$. During training, all three parameter sets are optimized jointly. At inference, for each individual test input the auxiliary head provides a self-supervised loss ${\mathcal{L}}_s$ (computed without labels), and a few gradient steps on ${\mathcal{L}}_s$ update only the shared encoder ${\theta }_e$—the two heads stay fixed—before the main prediction is read out. Because the adaptation is performed per input and then discarded, TTT treats each test example as a one-sample learning problem, trading additional inference computation for robustness to shifts unseen during training. Subsequent work extended this principle to a range of domains, including 3D reconstruction and medical imaging, and explored which subset of parameters to adapt—from normalization statistics to low-rank or bias-only updates—in order to control the cost and stability of the inner-loop update.

TTT as a sequence-modeling layer. A more recent line of work [13] reinterprets TTT not as an add-on adaptation procedure but as a sequence-modeling layer in its own right. The key idea is to make the layer’s hidden state a small machine-learning model whose parameters are a weight matrix $\mathbf{W}$, and to define the recurrence as a step of self-supervised learning: at each position t, given an input feature (denoted ${\tilde{\mathbf{F}}}_t$ to match the notation we use later for the speaker-conditioned stream), $\mathbf{W}$ is updated by gradient descent on a reconstruction loss between a corrupted and a clean projection of ${\tilde{\mathbf{F}}}_t$, and the layer output ${\mathbf{z}}_t$ is ${\tilde{\mathbf{F}}}_t$ mapped through the updated $\mathbf{W}$. The authors instantiate this as TTT-Linear and TTT-MLP, where $\mathbf{W}$ parameterizes a linear map and a two-layer network, respectively. Viewed this way, the hidden state “learns” on the test sequence with linear inference complexity, and the construction has been shown to compete with attention on long-context language modeling and to transfer to video generation and image classification. Our TTT adapter builds on this formulation; we describe the modifications it requires for PVAD in Section 3.3. Figure 1 contrasts the two formulations; symbols are defined here and, for the components specific to our model, in Section 3.

Entropy minimization and parameter-level adaptation. A parallel and widely used approach is entropy minimization. TENT [10] adapts the affine parameters of normalization layers at test time so as to minimize the entropy of the model’s predictions, on the premise that lower-entropy predictions are more reliable. Later methods refine this with confidence-based sample filtering, sequence-level objectives suited to structured outputs, and moving-average stabilization for continual adaptation [21]. What these methods share is that the adapted quantity is a set of model parameters—most often normalization or affine weights—rather than any input to the model.

Test-time adaptation in speech. Within speech processing, both threads have begun to appear. Dumpala et al. [22] introduced TTT for speech, reporting improvements on several downstream tasks under shifts in speaking style, gender, and age. TTT has since been applied to speech enhancement [14], where a self-supervised auxiliary task lets the model adapt to unseen noise at inference, and to speech editing. On the entropy-minimization side, test-time adaptation has been used for speech recognition and for speaker adaptation in lip reading [11], typically updating a subset of model parameters and resetting them after each utterance.

Two gaps follow from this survey and motivate our study. First, although TTT has been applied to several speech tasks, it has not been applied to PVAD; PVAD’s speaker-dependent, dual-input nature—a decision jointly determined by the acoustic stream and an external speaker embedding— makes its response to TTT an open empirical question rather than a predictable transfer. Second, while test-time adaptation by entropy minimization is itself well established, prior work adapts model parameters; the alternative of adapting the target speaker embedding at test time, an option that exists only for speaker-conditioned tasks such as PVAD, has not to our knowledge been examined. Our study investigates both gaps, adapting the internal feature transformation via TTT (Section 3.3) and the speaker embedding via TEA (Section 3.4).

3. Proposed Method

We study two test-time adaptation mechanisms layered on top of the FDE-Mamba backbone. Section 3.1 briefly recaps the backbone. Section 3.3 introduces the VAD-gated TTT adapter, an in-model mechanism that adapts an internal weight matrix on the speaker-conditioned feature stream of each utterance. Section 3.4 introduces TEA, an inference-time procedure that adapts the target speaker embedding while keeping all model parameters frozen. Section 3.5 clarifies that the two mechanisms act at distinct representation levels, and Section 3.6 describes the post-hoc Gaussian smoothing step that we treat as a separate, clearly delineated component.

Notation. We use bold lowercase for vectors and bold uppercase for matrices. A test utterance is a sequence of T frames. The acoustic input is ${\mathbf{x}}_{1:T}$ with per-frame feature ${\mathbf{x}}_t\in {\mathbb{R}}^F$, where F is the input feature dimension; $\mathbf{e}\in {\mathbb{R}}^E$ is the E-dimensional target speaker embedding (a d-vector). The backbone hidden dimension is D, so the latent and speaker-conditioned features ${\mathbf{h}}_t,{\mathbf{F}}_t,{\tilde{\mathbf{F}}}_t$ all lie in ${\mathbb{R}}^D$, and the TTT weight matrix $\mathbf{W}\in {\mathbb{R}}^{D\times D}$. Frame-level outputs are the VAD logits ${\mathbf{v}}_t\in {\mathbb{R}}^2$ (non-speech vs. speech) and the PVAD logits ${\mathbf{p}}_t\in {\mathbb{R}}^3$ (ns, ntss, tss). The subscript $\left[\mathrm{s}\right]$ selects the speech component of a softmax output, ⊙ denotes the element-wise (Hadamard) product, ${(·)}^{\top }$ the matrix transpose, and $\parallel ·\parallel$ the Euclidean norm. A parenthesized superscript ${(·)}^{\left(k\right)}$ indexes an inner-loop iteration. Table 1 summarizes the symbols.

Figure 2 contrasts the two mechanisms at a conceptual level before we detail them. They differ in what they adapt and when: TTT updates a weight matrix inside the model as it processes each sequence, whereas TEA keeps the model frozen and updates the speaker embedding fed into it, once per utterance.

3.1. FDE-Mamba Backbone

The FDE-Mamba backbone [9], whose flowchart is depicted in Figure 3, follows the two-stage design of FDE-RNN [8], comprising a Dynamic Encoder for VAD and a detachable Personalization module for PVAD, with all three recurrent components instantiated as Mamba [19] blocks. Given a sequence of acoustic features ${\mathbf{x}}_{1:T}$ with ${\mathbf{x}}_t\in {\mathbb{R}}^F$ and a target speaker embedding $\mathbf{e}\in {\mathbb{R}}^E$, the backbone produces frame-level VAD logits ${\mathbf{v}}_t$ and PVAD logits ${\mathbf{p}}_t$. The prediction block first yields a VAD probability $p_{\mathrm{vad}}\left(t\right)=\mathrm{softmax}{\left({\mathbf{v}}_t\right)}_{\left[\mathrm{s}\right]}$, the speech-class entry of the softmax over the VAD logits; the encoder block produces a latent representation ${\mathbf{h}}_t$, and a weighted residual connection combines the encoder output with the raw input according to VAD confidence:

$${\mathbf{F}}_t={\mathbf{h}}_t+\left(1-p_{\mathrm{vad}}\left(t\right)\right){\mathbf{x}}_t.$$

(1)

The fused feature is then modulated by the speaker embedding through a Feature-wise Linear Modulation (FiLM) layer [5]:

$${\tilde{\mathbf{F}}}_t=\gamma \left(\mathbf{e}\right)\odot {\mathbf{F}}_t+\beta \left(\mathbf{e}\right),$$

(2)

where $\gamma \left(\mathbf{e}\right)$ and $\beta \left(\mathbf{e}\right)$ are the scale and shift vectors generated from $\mathbf{e}$, and ⊙ denotes the element-wise product. The speaker-conditioned feature ${\tilde{\mathbf{F}}}_t$ is processed by the back-end Mamba block and a linear classifier to produce the PVAD logits. In the original FDE-Mamba, ${\tilde{\mathbf{F}}}_{1:T}$ passes directly into the back-end block; in this work, we insert the VAD-Gated TTT adapter at exactly this point, so that the adapter operates on the speaker-conditioned stream.

3.2. Overview of the Proposed Modifications

We modify the personalization module of FDE-Mamba at two points, as illustrated in Figure 4. First, within the interaction block, we insert a VAD-gated TTT adapter between the FiLM layer and the Mamba block. This adapter maintains an internal weight matrix $\mathbf{W}$ that is updated at every chunk by a self-supervised reconstruction signal, gated by the VAD probability $g_t$ so that only speech frames drive the adaptation. It adapts the speaker-conditioned feature stream ${\tilde{\mathbf{F}}}_{1:T}$ to the acoustic and speaker conditions of the current test utterance. Second, upstream of the interaction block, we introduce TEA (Test-time Embedding Adaptation), which adapts the target speaker embedding ${\mathbf{e}}_{\mathrm{target}}$ itself before it enters the FiLM generator. With all model parameters frozen, TEA performs a small number of gradient steps on $\mathbf{e}$ using self-supervised objectives computed from the model’s output, directly targeting the enrollment–test mismatch.

The two modifications thus act at different points in the data flow: the TTT adapter operates inside the model on the feature representation, while TEA operates on the input embedding. Their conceptual contrast is shown in Figure 2; Figure 4 shows their concrete placement within the personalization module. Additionally, a Gaussian smoothing step (Section 3.6) is applied as model-agnostic post-processing on the output probabilities; it is unrelated to the adaptation mechanisms and is reported separately. Section 3.3 and Section 3.4 detail each adaptation mechanism in turn.

3.3. VAD-Gated Test-Time Training Adapter

Background: TTT-Linear. The TTT-Linear layer [13] treats its hidden state as a linear model whose weight matrix $\mathbf{W}$ is updated by a step of self-supervised learning at each position. For an input token ${\mathbf{u}}_t\in {\mathbb{R}}^D$, the weight is updated by gradient descent on a reconstruction loss $\ell (\mathbf{W};{\mathbf{u}}_t)={\parallel \mathbf{W}{\mathbf{u}}_t-{\mathbf{u}}_t\parallel }^2$, which measures how well the linear map $\mathbf{W}$ reconstructs its own input:

$${\mathbf{W}}_t={\mathbf{W}}_{t-1}-\eta {\nabla }_{\mathbf{W}}\ell ({\mathbf{W}}_{t-1};{\mathbf{u}}_t),{\mathbf{z}}_t={\mathbf{W}}_t{\mathbf{u}}_t,$$

(3)

where $\eta$ is the inner-loop learning rate. Because $\mathbf{W}$ continues to be updated on the test sequence, the layer adapts its representation to the local statistics of each input even at inference time. Directly inserting Eq. (3) into the PVAD personalization pathway, however, raises three design questions specific to this task. We address each in turn.

3.3.1. Restricting Adaptation to Speech Frames via a VAD Gate

In a generic sequence model, every position contributes equally to the inner-loop update. In PVAD, however, a large fraction of frames are non-speech, and the reconstruction signal on those frames reflects background noise rather than speaker-relevant structure. Updating $\mathbf{W}$ uniformly across all frames would therefore let non-speech frames steer the adaptation, diluting the speaker-conditioned information the adapter is meant to refine.

We address this by gating the inner-loop update with the VAD probability. Operating on the speaker-conditioned stream ${\tilde{\mathbf{F}}}_{1:T}$ (Eq. (2)), and writing $g_t=p_{\mathrm{vad}}\left(t\right)$ for the (detached) VAD probability at frame t — the speech-class softmax output of the prediction block, the same quantity used in the weighted residual of Eq. (1) — the gated update direction is

$$\Delta {\mathbf{W}}_t={\displaystyle \frac{1}{\sqrt{D}}}\left(\mathbf{W}{\tilde{\mathbf{F}}}_t-{\tilde{\mathbf{F}}}_t\right){\left(g_t{\tilde{\mathbf{F}}}_t\right)}^{\top },$$

(4)

where D is the hidden dimension and the factor $1/\sqrt{D}$ scales the update to keep its magnitude stable across feature dimensions, so that the contribution of frame t to the update is scaled by its probability of being speech. Frames the VAD stage deems non-speech ($g_t\approx 0$) contribute little to $\Delta {\mathbf{W}}_t$, while confident speech frames drive the adaptation. We refer to this as a VAD-gated adapter, since the VAD probability $g_t$ acts as a gate that confines the test-time weight update to frames the model itself judges to be speech. This couples the adapter to the model’s own VAD decision rather than to an external mask, and keeps the adaptation focused on the frames where speaker discrimination matters.

To keep the inner-loop update stable when adapting over long utterances, we additionally clamp the update with a scaled hyperbolic tangent applied element-wise to $\Delta {\mathbf{W}}_t$,

$$\Delta {\mathbf{W}}_t\leftarrow ctanh\left(\Delta {\mathbf{W}}_t/c\right),c=0.1,$$

(5)

which bounds every entry of the update to $(-c,c)$ and limits the influence of outlier frames whose reconstruction error is large. We found this bounded update to be important for stability: without it, $\mathbf{W}$ can accumulate large updates and drift over a long utterance.

3.3.2. Stabilizing the Weight Trajectory with an EMA Update

In TTT-Linear the weight matrix is overwritten at each step (Eq. (3)). Applied unmodified during inference on PVAD, this can let $\mathbf{W}$ drift over a long utterance, since each update is taken on a single, noisy reconstruction signal and there is no outer-loop loss to correct accumulated error. To stabilize the trajectory, we combine the gradient step with an exponential moving average (EMA). The post-step weight applies the (clamped) update of Eq. (5) with the inner-loop rate (the factor of 2 arises from differentiating the squared reconstruction loss $\ell ={\parallel \mathbf{W}{\tilde{\mathbf{F}}}_t-{\tilde{\mathbf{F}}}_t\parallel }^2$):

$${\mathbf{W}}^{\mathrm{step}}=\mathbf{W}-2\eta \Delta {\mathbf{W}}_t.$$

(6)

The retained weight is then a convex combination of the previous and updated matrices,

$$\mathbf{W}\leftarrow \alpha \mathbf{W}+(1-\alpha ){\mathbf{W}}^{\mathrm{step}},$$

(7)

where $\alpha \in [0,1)$ is the EMA coefficient. A larger $\alpha$ retains more of the accumulated state and yields a smoother trajectory; $\alpha =0$ recovers the plain TTT step. The adapter output applied to the feature stream uses the post-step weight,

$${\mathbf{z}}_t={\tilde{\mathbf{F}}}_t+{\mathbf{W}}^{\mathrm{step}}{\tilde{\mathbf{F}}}_t,$$

(8)

i.e., the adapter produces an additive correction to the speaker-conditioned feature rather than replacing it, so that the back-end block always receives the original feature plus a learned, input-dependent residual. The EMA-smoothed weight $\mathbf{W}$ of Eq. (7) is carried to the next chunk as the running adaptation state, while the current chunk’s output uses the post-step weight ${\mathbf{W}}^{\mathrm{step}}$.

3.3.3. Chunked Updates for Streaming Operation

Performing one gradient step per frame (Eq. (3)) is both expensive and unstable on the relatively short, noisy feature stream of a PVAD utterance. We therefore divide the sequence into non-overlapping chunks of length b and perform a single matrix-form update per chunk, accumulating the gated reconstruction signal over the b frames within the chunk before applying Eqs. (6) and (7). The resulting weight $\mathbf{W}$ is carried across chunks within an utterance, providing a running adaptation state, and is reset to its learned initialization ${\mathbf{W}}_0$ at the start of each utterance. The chunk size b trades off adaptation granularity against update stability, and is treated as a hyperparameter in our study (Section 5).

Taken together, Eqs. (4)–(8) define the VAD-gated TTT adapter. We emphasize that the adapter operates on ${\tilde{\mathbf{F}}}_{1:T}$, the feature stream after speaker fusion. The distribution shift it absorbs at test time is therefore a composite of acoustic conditions, speaking style, and speaker characteristics as they are jointly expressed in the speaker-conditioned representation; the adapter does not, and is not intended to, isolate any single one of these factors.

3.4. TEA: Test-Time Embedding Adaptation

The second mechanism, Test-time Embedding Adaptation (TEA), adapts a different part of the system. Rather than modifying any model parameter, it adapts the target speaker embedding $\mathbf{e}$ itself, directly addressing the case where the enrollment embedding is mismatched with the test signal. With all model parameters frozen, the embedding is treated as the only free variable. Starting from the enrolled embedding ${\mathbf{e}}^{\left(0\right)}=\mathbf{e}$, TEA performs K gradient steps on a self-supervised objective ${\mathcal{L}}_{\mathrm{TEA}}$ computed from the test utterance,

$${\mathbf{e}}^{(k+1)}={\mathbf{e}}^{\left(k\right)}-\mu {\nabla }_{\mathbf{e}}{\mathcal{L}}_{\mathrm{TEA}}\left({\mathbf{e}}^{\left(k\right)}\right),k=0,\dots ,K-1,$$

(9)

and then runs a final forward pass with the adapted embedding ${\mathbf{e}}^{\left(K\right)}$ to produce the predictions. Because the procedure adapts only the input embedding and resets it for each utterance, it introduces no persistent change to the model and no additional parameters. In Eq. (9), ${\mathcal{L}}_{\mathrm{TEA}}$ is a self-supervised objective requiring no ground-truth labels; we consider three instantiations:

Entropy. Following the entropy-minimization principle [10], we minimize the uncertainty of the PVAD predictions, restricted to speech frames via the VAD gate $g_t$:

$${\mathcal{L}}_{\mathrm{ent}}={\displaystyle \frac{{\sum }_t{g}_t\left[-{\sum }_c{q}_{t,c}log{q}_{t,c}\right]}{{\sum }_t{g}_t}},$$

(10)

where $q_{t,c}=\mathrm{softmax}{\left({\mathbf{p}}_t\right)}_c$ is the predicted PVAD probability for class c at frame t. Minimizing ${\mathcal{L}}_{\mathrm{ent}}$ sharpens the model’s decisions on speech frames, encouraging an embedding under which the model is confident.

Reconstruction. The second objective reuses the self-supervised signal of the TTT adapter. With the speaker-conditioned feature ${\tilde{\mathbf{F}}}_t$ recomputed from the candidate embedding via Eqs. (1)–(2), the loss is the reconstruction error of the adapter’s initial linear map,

$${\mathcal{L}}_{\mathrm{rec}}={\displaystyle \frac{1}{T}}\sum _t{∥{\mathbf{W}}_0{\tilde{\mathbf{F}}}_t-{\tilde{\mathbf{F}}}_t∥}^2,$$

(11)

which favors an embedding whose induced feature stream is consistent with the representation the adapter was trained to reconstruct.

Joint. The third objective is a weighted combination,

$${\mathcal{L}}_{\mathrm{joint}}={\mathcal{L}}_{\mathrm{ent}}+\lambda {\mathcal{L}}_{\mathrm{rec}},$$

(12)

with $\lambda$ a balancing coefficient, set to $\lambda =0.5$ in our experiments. We compare all three objectives empirically in Section 5.

Although TEA shares the entropy-minimization principle with prior test-time adaptation methods [10], it differs in what is adapted: where those methods update model parameters (typically normalization or affine weights), TEA updates the target speaker embedding. This choice is specific to speaker-conditioned tasks and, to our knowledge, has not previously been studied for PVAD.

3.5. Two Complementary Adaptation Levels

The two mechanisms adapt distinct parts of the system and should not be read as redundant. The VAD-gated TTT adapter modifies an internal weight matrix that transforms the speaker-conditioned feature stream; its effect is a learned, input-dependent residual applied to ${\tilde{\mathbf{F}}}_{1:T}$ within the model. TEA, by contrast, leaves every model weight untouched and moves the speaker embedding $\mathbf{e}$, which enters the model upstream through the FiLM layer (Eq. (2)). One mechanism therefore operates in the feature-transformation space and the other in the speaker-representation space. We avoid characterizing them as cleanly handling separate sources of distribution shift—for instance attributing noise to one and speaker mismatch to the other—because the speaker-conditioned representation entangles these factors; rather, we describe them as acting at different representation levels and evaluate their independent and combined effects directly.

3.6. Gaussian Smoothing Post-Processing

Finally, we consider a post-hoc smoothing step that is independent of the two adaptation mechanisms and of the model. After the per-frame probabilities are produced, a one-dimensional Gaussian filter of standard deviation $\sigma$ is applied along the time axis to both the VAD and PVAD probability sequences prior to thresholding:

$${\widehat{q}}_{t,c}=\sum _{\tau }{\mathcal{K}}_{\sigma }\left(\tau \right)q_{t-\tau ,c},{\mathcal{K}}_{\sigma }\left(\tau \right)\propto exp\left(-\frac{{\tau }^2}{2{\sigma }^2}\right).$$

(13)

This suppresses isolated single-frame errors—brief false detections within non-speech regions and momentary dropouts within speech regions —by enforcing temporal continuity in the probability sequence. We stress that Gaussian smoothing is a generic post-processing operation unrelated to test-time adaptation; we include it because it appears in our evaluation pipeline, and we report its contribution separately (Section 5) so that it is not conflated with the effect of the TTT adapter or TEA.

4. Experimental Setup

4.1. Dataset and Data Preparation

We follow the experimental protocol of FDE-RNN [8] and FDE-Mamba [9] exactly, to ensure direct comparability with prior work. All models are trained and evaluated on the LibriSpeech corpus [23]. To simulate multi-speaker scenarios with speaker turns, we adopt the concatenated-utterance scheme of Ding et al. [3]: for each sample, the number of speakers is drawn from $[1,3]$, non-repeating speakers are randomly selected with one designated as the target, a single utterance is sampled per speaker, and the utterances are shuffled and concatenated. Target speaker embeddings are 256-dimensional d-vectors [15] obtained by passing additional, independently sampled utterances of the target speaker through a pre-trained speaker verification model. Following Yu et al. [16], in 20% of training samples the designated target is replaced by a speaker absent from the concatenated utterance, encouraging robustness to unseen enrollment conditions. Multi-condition training with reverberation is applied, with room impulse responses and additive noise drawn from standard corpora [24]. There is no speaker or utterance overlap between the training and test partitions, and all embeddings are constructed within their respective partitions.

4.2. Evaluation Metrics

Consistent with prior work [3,8], we report: Average Precision (AP) for each category, presented as the combined ns&ntss AP and the tss AP; mean Average Precision (mAP), the average of AP across categories, used as the primary ranking metric; Accuracy, the frame-level classification accuracy; and Recall and Precision for the tss category, to characterize target-speaker detection sensitivity and specificity. To assess inference cost, we additionally report the real-time factor (RTF), peak memory allocation, and kilo-FLOPs per frame.

4.3. Implementation Details

All models are implemented in PyTorch. Input features are 40-dimensional log-Mel filterbank coefficients, and the hidden dimension of all blocks is set to $D=64$, following [9]. We train with a batch size of 64 and an outer-loop learning rate of ${10}^{-3}$. The VAD-gated TTT adapter uses a chunk size of $b=16$ and an EMA coefficient of $\alpha =0.7$; the inner-loop learning rate $\eta$ is a learned scalar (a trainable parameter initialized to 0, updated by backpropagation during outer-loop training). For TEA, the embedding learning rate $\mu$, number of steps K, and loss type (entropy, reconstruction, or joint) are swept as described in Section 5; unless otherwise stated, the Gaussian smoothing standard deviation is $\sigma =5$, applied to both the VAD and PVAD probability sequences. The same adaptation mechanisms are additionally applied to the LSTM-based FDE-RNN backbone to assess cross-architecture generality. Unless otherwise stated, all systems are evaluated with the Gaussian smoothing post-processing described in Section 3.6. We treat this smoothing as a standard, model-agnostic component of the inference pipeline, and focus our analysis on the behavior of the TTT and TEA adaptation mechanisms.

5. Results and Discussion

We first establish the overall effect of the proposed test-time adaptation against the FDE-Mamba backbone (Section 5.1). We then dissect the contribution of each component through an ablation study (Section 5.2), report the computational cost (Section 5.3), and verify that the same mechanisms transfer to the LSTM-based FDE-RNN backbone (Section 5.4). We close with a discussion of a recall–precision trade-off and the limitations of the study (Section 5.5). Throughout, “TTT” denotes the VAD-gated TTT adapter, “TEA” the test-time embedding adaptation, and “Gauss” Gaussian smoothing with $\sigma =5$.

5.1. Main Results

Our primary comparison is against the FDE-Mamba backbone [9], which is the direct, already-published baseline that this work extends. Table 2 reports the FDE-RNN baseline for reference, the FDE-Mamba backbone, and the proposed full system (TTT + TEA + Gauss). The proposed system improves the primary PVAD metric, mAP, from 0.9605 to 0.9641 ($+0.0036$) over FDE-Mamba, with the ns&ntss AP rising from 0.9699 to 0.9740 and precision from 0.881 to 0.899; accuracy improves from 89.87% to 90.19%. We state plainly that the improvement over the FDE-Mamba backbone is modest in magnitude. Its value, in the context of this empirical study, lies in showing that inference-time adaptation—which requires no retraining and no additional labels—yields a consistent gain on a strong, recently published PVAD baseline, and in characterizing precisely where that gain comes from, which we do next. To check that the best-mAP result is not an artifact of the stochasticity in the TEA inner loop, we repeated the evaluation of the proposed system five times with the trained model held fixed; the PVAD mAP was $0.96406\pm 0.00001$ (mean ± std over five runs), confirming that the test-time adaptation procedure is highly stable at inference. We emphasize that this quantifies only the inference-time stochasticity, not the variance across independent training runs (see Section 5.5).

5.2. Ablation Study

To attribute the overall gain to its sources, Table 3 reports every combination of the three components on the FDE-Mamba backbone. We make four observations.

The TTT adapter contributes a small, precision-oriented gain. Adding TTT alone raises mAP from 0.9605 to 0.9623 ($+0.0018$) and precision from 0.881 to 0.900, while accuracy is essentially unchanged. The adapter sharpens speaker discrimination rather than improving overall frame classification.

TEA helps only in combination with TTT. Used on its own (“+ TEA”), TEA does not improve over the backbone—mAP decreases to 0.9564—indicating that the inference-time perturbation of the embedding is not, by itself, beneficial under the current protocol. Added on top of TTT, however, it raises mAP from 0.9623 to 0.9633, and the full combination attains the best mAP of 0.9641. We report this dependence explicitly: the value of TEA in this study is conditional on the presence of the TTT adapter.

Gaussian smoothing is the largest single contributor to accuracy, and is independent of the adaptation mechanisms. Smoothing alone raises accuracy from 89.87% to 90.47%—the largest accuracy improvement of any single component, and larger than that of the TTT adapter—while its effect on mAP is comparatively small ($+0.0013$). Because smoothing is a generic, model-agnostic post-processing step, we report its contribution separately so that it is not conflated with the adaptation mechanisms: the accuracy gains it provides and the precision/mAP gains the adapter provides arise from different mechanisms.

A recall–precision trade-off accompanies adaptation. Across the table, adding TTT raises precision but lowers tss recall (0.870 → 0.849); TEA and smoothing partially restore it (to 0.856), but the full system does not recover the recall of the unadapted backbone. We return to this in Section 5.5.

A sweep over the TEA objective (entropy, reconstruction, joint), learning rate $\mu$, and step count K is reported in Table 4 and Table 5; we select the configuration with the highest mAP for the main results. Two findings stand out. First, TEA is highly sensitive to the learning rate: all three objectives degrade sharply as $\mu$ increases, collapsing by $\mu ={10}^{-1}$ (the entropy objective falls to 0.526). A small learning rate ($\mu ={10}^{-3}$) is essential, with the reconstruction and joint objectives performing best there (mAP 0.9641). Second, more adaptation steps do not help: mAP peaks at $K=2$ and declines thereafter, dropping to 0.9567 at $K=10$ (Table 5). Both findings indicate that TEA is useful only as a small, carefully bounded correction; over-adaptation harms performance.

5.3. Computational Cost

Table 6 reports the inference cost. The TTT adapter is inexpensive: it adds about 3% more parameters (145,524 → 150,053) and raises the real-time factor (RTF) only modestly (0.00519 → 0.00708, a $1.4\times$ increase), with no change in FLOPs per frame. TEA, however, is costly. Because it performs K gradient steps on the embedding—each requiring a forward and backward pass through the model—per utterance at inference, both RTF and FLOPs scale with K. At the chosen setting $K=2$, RTF rises to 0.02876 ($5.5\times$ the FDE-Mamba backbone) and FLOPs per frame increase roughly sevenfold (221.9 → 1553.5). We note plainly that this nearly erases the RTF advantage of the Mamba backbone over FDE-RNN (0.02813): the full system with TEA runs at approximately the same speed as the LSTM baseline. The peak memory also grows substantially (43.5 MB → 136 MB), reflecting the activations retained for the backward pass. This cost–benefit profile is unfavorable: TEA at $K=2$ buys a further $+0.0008$ mAP over TTT alone while increasing RTF by roughly $4\times$. The TTT adapter, by contrast, delivers most of the benefit at negligible cost. We return to this trade-off in Section 5.5.

5.4. Generalization to the FDE-RNN Backbone

While the focus of this study is the FDE-Mamba backbone, we additionally verify that the proposed mechanisms are not specific to a selective state-space model by applying the same TTT adapter and TEA to the LSTM-based FDE-RNN. Table 7 shows the same qualitative pattern: TTT raises mAP from 0.9419 to 0.9434, and the full combination reaches 0.9443, with tss AP improving monotonically as components are added. The gains are again modest and of comparable magnitude to those on FDE-Mamba. This consistency across an LSTM and a selective state-space backbone indicates that the contribution of the adaptation mechanisms is largely independent of the underlying sequence model. We present this as a supporting observation rather than a primary result.

5.5. Discussion and Limitations

The recall–precision trade-off. Adding the TTT adapter raises precision (0.881 → 0.900) but lowers tss recall (0.870 → 0.849): the adapter makes the model more conservative in declaring a frame to be target-speaker speech, committing fewer false positives but missing more true target frames. TEA and Gaussian smoothing partially recover recall (to 0.856 in the full system), but the full system still does not match the recall of the unadapted backbone. This trade-off is consistent with the precision-oriented nature of the reconstruction-based adaptation signal, and it is the main reason the accuracy gains are smaller than the AP gains. Applications that prioritize not missing target speech may prefer the unadapted backbone, while those that prioritize suppressing false target detections may benefit from the adapter.

Cost versus benefit of TEA. The two mechanisms differ sharply in cost-effectiveness. The TTT adapter delivers most of the benefit (mAP $0.9605\to 0.9623$) for a $1.4\times$ increase in RTF and no extra FLOPs. TEA, in contrast, adds only a further $+0.0008$ mAP at $K=2$ yet raises RTF by roughly $4\times$ and FLOPs sevenfold (Table 6), because each adaptation step requires a full forward–backward pass. As a result, the complete system runs at about the same speed as the LSTM-based FDE-RNN baseline, largely forfeiting the efficiency advantage of the Mamba backbone. For deployments where inference cost matters, the TTT adapter alone is the more sensible operating point; TEA is justified only when the small additional mAP is worth a severalfold latency increase. We report this plainly rather than presenting the best-mAP system as unconditionally preferable.

Sensitivity of TEA. TEA is also delicate to configure. Table 4 and Table 5 show that performance collapses if the learning rate is raised beyond ${10}^{-3}$ or if more than a few adaptation steps are taken—both symptoms of over-adaptation, in which the embedding drifts away from a useful speaker representation when pushed too hard by the self-supervised objective. The method is therefore beneficial only as a small, tightly bounded correction. This fragility is itself a finding: unconstrained test-time adaptation of the speaker embedding is not robust for PVAD, and careful regularization of the adaptation step is necessary.

Why the gains are modest. The gains from test-time adaptation, while consistent, are small. We attribute this in part to the evaluation protocol, which presents limited distribution shift for either mechanism to absorb. On the acoustic side, although multi-condition training adds reverberation and noise, the train and test distributions are drawn from the same corpus and augmentation pipeline, so the train–test acoustic mismatch that the TTT adapter is designed to correct is mild; on the speaker side, the enrollment embeddings are generally well matched to the test condition, limiting what TEA can recover. Both mechanisms may offer larger benefits under deliberately mismatched conditions—for the TTT adapter, test utterances from unseen acoustic environments, channels, or speaking styles; for TEA, short, noisy, or cross-session enrollment—which the present protocol does not isolate.

Limitations. Three limitations follow. First, although we verified that the proposed system is stable across repeated evaluations of a fixed trained model (Section 5.1), each model configuration was trained only once; given the small effect sizes, repeating the training across several random seeds and reporting the resulting variance would further strengthen the conclusions, and we therefore regard the current gains as suggestive rather than definitive. Second, the benefit of TEA is contingent on the TTT adapter and does not appear in isolation, so its standalone value remains unestablished. Third, the LibriSpeech-based protocol uses concatenated read speech with artificially clean speaker boundaries, well-matched enrollment, and only synthetic acoustic augmentation; evaluation under genuine overlapping speech, real acoustic-environment mismatch, and mismatched enrollment remains important future work, and is the setting in which we would expect test-time adaptation—both the acoustic adaptation of the TTT adapter and the speaker adaptation of TEA—to be most useful.

6. Conclusion and Future Work

We presented an empirical study of test-time adaptation for personal voice activity detection, introducing a VAD-gated Test-Time Training (TTT) adapter and a test-time speaker embedding adaptation scheme (TEA) on top of the FDE-Mamba backbone. To our knowledge, this is the first application of TTT to PVAD, and the first to adapt the target speaker embedding itself at inference time for this task. Across both an LSTM-based and a selective state-space backbone, the two mechanisms produce consistent but modest improvements in mAP and precision, concentrated on speaker-discrimination metrics rather than overall accuracy, at the cost of a small reduction in recall. We further showed that a model-agnostic Gaussian smoothing step accounts for the largest single accuracy gain and should be reported separately from the adaptation mechanisms.

Taken together, these findings provide a measured baseline for adaptive PVAD and clarify the cost–benefit profile of different adaptation choices: the VAD-gated TTT adapter offers low-cost, inference-only gains with minimal architectural changes, while TEA yields additional but smaller improvements at a noticeable increase in computational cost and latency. We also highlighted several limitations of the present study, including the reliance on a single training seed per configuration, the use of a LibriSpeech-based protocol that does not explicitly enforce severe acoustic or enrollment mismatch, and the fact that TEA’s benefit appears most clearly when combined with TTT rather than as a standalone mechanism.

These limitations point directly to several directions for future work. First, extending the evaluation to more challenging and realistic scenarios—such as far-field capture, reverberant and noisy environments, heavily mismatched or short enrollments, and overlapping multi-speaker speech—would better stress the conditions under which test-time adaptation is most likely to help and may reveal larger gains than those observed here. Second, conducting multi-seed training and broader hyperparameter sweeps would allow a more rigorous assessment of the statistical significance and robustness of the reported improvements. Third, it would be valuable to explore alternative adaptation targets and schedules, including cross-utterance or continual test-time adaptation for PVAD, while explicitly addressing stability and catastrophic forgetting, as well as integrating TTT and TEA with stronger pretraining or self-supervised front-ends. We hope that the empirical baseline and analysis provided in this work will serve as a reference point for future research on adaptive PVAD and, more broadly, on test-time adaptation for speaker-conditioned streaming speech tasks.