Robust Multimodal Speech Recovery for Ambient Noise Environments

1. Introduction

Speech recognition converts human speech or speech-related physical signals into linguistic information, and is a key technology for natural human–computer interaction, assistive communication, medical monitoring, smart devices, and privacy-preserving control. Conventional microphone-based automatic speech recognition (ASR) has achieved strong performance, especially with large-scale models such as Whisper [1]. However, acoustic microphones are sensitive to background noise, reverberation, overlapping speakers, long-distance attenuation, and privacy concerns. Therefore, non-acoustic and contactless speech sensing has become an important research direction, particularly for silent speech interfaces, noisy environments, and users with speech impairments [2]. Possible applications include assistive communication for speech-impaired users, silent command input in public spaces, robust interaction in factories or vehicles, medical rehabilitation, and privacy-preserving human–computer interaction. Recent reviews have surveyed biosignal sensors and deep-learning-based speech recognition [3].

Different sensing modalities capture different aspects of speech production. Microphones directly record acoustic pressure waves and preserve phonetic, prosodic, and speaker-timbre information, but they are vulnerable to environmental noise and privacy leakage. Depth cameras capture three-dimensional lip and facial movements. Wang et al. [4] proposed Watch Your Mouth, a depth-sensing silent speech recognition system, showing that depth information can support private speech input without audio. Its advantage is clear spatial representation of articulation, whereas its limitations include occlusion, distance sensitivity, and lack of vocal-fold or timbre information. Radar-based sensing, especially IR-UWB and mmWave radar, detects lip, jaw, and facial micro-motions. Shin and Seo [5] demonstrated contactless silent speech recognition using IR-UWB radar to capture visible articulator motion. Early millimeter-wave sensing demonstrated speech signal acquisition at 94 GHz [6], and micro-Doppler radar has recently been shown feasible for real-time speech detection and characterization [7]. Radar is robust to acoustic noise and does not require visible facial texture, but it has lower spatial resolution and is sensitive to sensor placement.

Since each modality is incomplete, multimodal fusion is promising. Wavoice fused mmWave radar and microphone signals to improve speech recognition in noisy and motion-interference scenarios [8]. Audio-visual speech and gesture recognition using mobile-device sensors further illustrates the benefit of multimodal sensing [9]. Existing fusion methods include early fusion, feature-level fusion, decision-level fusion, and attention-based fusion. Early fusion is simple but sensitive to sampling mismatch; decision-level fusion is robust but loses fine-grained complementarity; attention-based fusion can adaptively weight modalities but increases model complexity. Therefore, feature-level fusion with modality-specific encoders is suitable for heterogeneous modalities such as depth camera and UWB radar. To fairly evaluate the performance of cross-modal fusion schemes for silent speech recognition, this work adopts the comprehensive multimodal dataset released by Ge et al. [10].

Speech recognition maps an audio sequence $\mathbf{X}=(x_1,\dots ,x_T)$ to a text sequence $\mathbf{Y}=(y_1,\dots ,y_U)$, where $T\gg U$ creates a temporal alignment problem. Early DNN-HMM hybrid systems [11] relied on forced alignment and pronunciation dictionaries, but have largely been replaced by end-to-end approaches, including CTC [12], Attention-based Encoder–Decoder models [13], and RNN-Transducer [14]. These methods presuppose a direct acoustic channel; in contrast, our work operates on non-acoustic sensor streams, where the mapping from physical vibrations and articulatory gestures to speech must be discovered without textual or phonetic supervision.

For evaluation, we employ two pretrained models. Whisper [15] provides word-level recognition accuracy as a direct measure of intelligibility. wav2vec 2.0 [16] learns contextualized speech representations through self-supervised learning. We use its embedding space to measure intra-class compactness, inter-class separability, and class-center cosine similarity between real and recovered speech, providing annotation-free estimates of phonetic fidelity that go beyond surface-level recognition accuracy.

Speech mapping learns a function $f:\mathcal{V}\to \mathcal{A}$ that transforms alternative sensing modalities into acoustic representations. Existing approaches can be broadly categorized as supervised, self-supervised, and unsupervised. Supervised methods rely on synchronized paired data and are commonly optimized using L1 or MSE losses. Representative examples include lip-to-speech synthesis systems [17,18], radar-based silent speech recognition and reconstruction-related approaches [19], and command-word-level radar speech recognition systems [20]. Generative vocoders such as MelGAN [21] further improve perceptual quality but may introduce artifacts and training instability.

Self-supervised approaches employ pretrained models to generate discrete or continuous intermediate speech units. VQ-VAE [22] converts acoustic regression into classification through vector quantization, whereas HuBERT [23] generates cluster-based pseudo-labels that have recently been adopted in multimodal and silent-speech systems. Unsupervised methods such as CycleGAN remove paired-data requirements through cycle-consistency learning but generally suffer from weaker content alignment and lower intelligibility.

Beyond learning architectures, the sensing modality fundamentally determines the recoverable speech information. Visual approaches typically employ RGB or depth cameras to capture articulatory motion. Markerless pose estimation can automatically extract speech-articulator landmarks from camera and ultrasound images [24], and end-to-end sentence-level lipreading has been advanced with temporal convolutional networks [25]. Compared with RGB imaging, depth sensing provides three-dimensional geometric information and is less sensitive to illumination variations, enabling more robust characterization of articulatory movements. Radar-based approaches exploit radio-frequency sensing to capture subtle articulatory and physiological motions and can operate under poor lighting or partial occlusion conditions. Doppler radar speech recognition has also been combined with mobile vision transformers [26].

More recently, optical sensing technologies have demonstrated the feasibility of measuring vocal-fold dynamics and tissue vibration patterns [27,28]. Unlike visual observations, laser sensing belongs to a physiological vibration sensing modality, capturing speech-related motions propagated through biological tissues. Consequently, laser measurements may contain physically informative speaker-dependent characteristics that are weakly represented in purely articulatory observations. For laser-speckle-based speech reconstruction, dedicated algorithms have been developed to detect and remove speckle noise [29] and to enhance the reconstructed speech [30]. Laser Doppler vibrometry has been used for remote acoustic sensing and control with retro-reflective membranes [31]. Although optical sensing has shown promise for measuring speech-related physiological vibrations, its application to speech recovery remains largely unexplored compared with visual and radar-based approaches.

Despite substantial progress, several challenges remain: (1) synchronization across heterogeneous modalities; (2) high-frequency information loss in radar and visual sensing; (3) speaker dependence arising from physiological variability; (4) the fidelity–naturalness trade-off between regression and adversarial methods; (5) weak physical association between articulatory observations and speaker timbre; and (6) difficulty in preserving fine-grained spectral dynamics under global reconstruction objectives. These challenges motivate multimodal fusion strategies that integrate complementary observations from visual, radar, and physiological vibration sensing modalities, but existing efforts typically require precise frame-level alignment and linguistic supervision. A paradigm that operates under weak alignment and without text or phoneme labels is yet to be established.

We propose a weakly-aligned multimodal speech reconstruction framework that departs from this supervision-heavy paradigm. Our central premise is that linguistic structure—phonetic form, speaker identity, and content configuration—can emerge autonomously from the physical correspondence between heterogeneous sensor streams and audio, without any textual or phonetic supervision during reconstruction. This premise is consistent with the physical principles of recovering speech from vibrations in radar and laser sensing [32]. The framework fuses laser vibration sensing, depth-camera articulatory capture, and UWB radar micro-motion detection, mapping their complementary physical observations to a log-mel spectrogram through a two-stage, label-free process. The resulting waveform, obtained via Griffin–Lim vocoding, carries intelligible speech in which phonetic, speaker, and content structures are spontaneously organized. The only linguistic labels ever used are corpus-level content categories introduced exclusively as an auxiliary regularisation signal during encoder pretraining; these classification heads are discarded before the regression stage, and no language-level supervision is provided to the speech reconstruction task itself.

Under the leave-two-subjects-out protocol, a calibration split is optionally applied: for each held-out subject, 30% of the trials (stratified by content category) are moved back into the training set. This increases the amount of adaptation data while preserving the unseen-speaker evaluation on the remaining 70%. Cross-validation experiments reveal that the model learns discriminative speaker-and-content representations under closed-set conditions while highlighting the remaining challenge of speaker-independent generalization. The framework is inherently scalable because it requires only naturally synchronised sensor–audio pairs and no manual transcriptions or alignments.

The main contributions of this paper are summarized as follows:

• Weakly-Aligned Multimodal Physical Reconstruction Paradigm. We propose the first speech recovery framework that fuses laser vibrometry, depth-camera articulatory capture, and UWB radar micro-motion sensing under a weakly aligned, text-free setting. It exploits the direct physical correspondence between heterogeneous sensor streams and the acoustic field, requiring only coarse synchronization that tolerates inter-modality delays of approximately 0.7 s, without phoneme boundaries, word alignments, or transcriptions. A two-stage strategy pretrains modality-specific encoders with an auxiliary word-level classification loss as a feature regularizer, then discards the classification heads and performs label-free spectrogram regression. This yields a low-cost, scalable framework where phonetic structure and intelligibility emerge solely from sensor–audio correspondence.
• Two-Stage Training with Auxiliary Regularisation and Perception-Driven Loss. We introduce a decoupled training methodology: modality-specific encoders are first pretrained with a word-level classification objective to learn discriminative content structure, then frozen for text-free and phoneme-free regression. A hybrid loss combines MSE with a mid-range weighting term and a jump-aware constraint, shifting optimization from coarse spectral mapping toward fine-grained formant transitions and energy onsets. This captures both articulatory dynamics and speaker-specific detail, improving intelligibility and naturalness.
• Comprehensive Multi-Faceted Evaluation and Ablation Analysis. We establish a three-dimensional evaluation protocol covering intelligibility (Whisper), centre cosine similarities (wav2vec 2.0 embedding metrics), and perceptual quality (MOS). Ablation on encoder pretraining shows that removing the auxiliary classification structure degrades reconstruction fidelity, confirming that the classification loss improves feature quality and discriminability. Incremental modality-combination experiments, evaluated by Whisper, wav2vec 2.0 metrics, and MOS, reveal the complementary contributions of each sensor modality. Collectively, these analyses show that linguistic intelligibility arises from physically grounded sensor streams under weak alignment and without textual or phonetic supervision.

2. Dataset Description

In this study, we adopt the RVTALL multimodal speech dataset proposed by Ge et al. [10], which was developed for contactless lip reading and acoustic analysis. This dataset characterizes speech production through multiple synchronized sensing modalities, including Ultra-Wideband (UWB) radar, laser-speckle measurements, Kinect-based audio-visual recordings, and lip landmark annotations. Compared with conventional audio-only speech corpora, RVTALL provides complementary acoustic and physical observations of speech-related articulatory and vibrational dynamics. In this work, we focus on the depth-camera, laser-speckle, and UWB radar modalities, and investigate their effectiveness for speech classification and reconstruction.

Figure 1 shows the acquisition setup of the multimodal speech signals, where (a) presents the top-view layout and (b) presents the front-view layout. During data collection, the participant was seated in front of the sensing system and was instructed to pronounce the predefined speech corpus. The depth camera with an integrated microphone was placed in front of the participant to record the acoustic signal and capture mouth-region visual information. The UWB radar sensors were arranged around the participant to measure speech-induced facial and upper-body micro-motion through radio-frequency reflections. The laser-camera system was mounted on a tripod and directed toward the facial or neck surface to capture subtle laser-speckle variations caused by speech-related skin vibration. The fixed relative positions and distances among these devices helped maintain a consistent acquisition geometry across recordings. This configuration enables simultaneous acquisition of acoustic, articulatory, and vibration-related observations, providing complementary multimodal information for speech classification and reconstruction.

The dataset contains recordings from 20 volunteers with different linguistic backgrounds, including Chinese, English, and Arabic speakers, as summarized in Table 1. This participant distribution introduces inter-speaker variability in accent, pronunciation style, articulation pattern, and speech rhythm. Such diversity is important for evaluating whether a model can generalize across speakers rather than only learning subject-specific signal patterns.

The corpus is organized into three linguistic levels: vowels, isolated words, and short sentences. As shown in Table 2, the single-subject speech subset contains 5 vowels, 15 isolated words, and 16 sentences. The isolated words are mainly related to emergency and medical scenarios, such as “ambulance”, “bleed”, “medical”, “rescue”, and “emergency”. This fixed-corpus design allows the same speech content to be compared across participants and sensing modalities, which is useful for word-level classification, cross-modal alignment, and multimodal feature analysis.

Figure 2 presents representative multimodal samples from the dataset, including UWB radar Doppler representations, laser vibration signals, microphone waveforms, and mouth-region visual observations. The UWB components are shown through upper and lower radar views, which describe motion-induced Doppler variations during speech production. The laser signal records skin-surface vibration caused by speech-related facial and throat movement, while the microphone waveform provides the corresponding acoustic reference. The mouth images further illustrate visible articulatory states at different moments of speech. These examples show that the dataset provides heterogeneous but complementary descriptions of the same speech process, covering radio-frequency motion responses, optical vibration measurements, acoustic signals, and visual articulatory information. Such multimodal observations support the analysis of both speech content and physical speech-production dynamics, providing a data basis for subsequent speech classification and reconstruction experiments.

In this work, we use the laser-speckle signal as the primary modality for speech reconstruction, because it directly reflects speech-related surface vibration induced by vocal-fold and facial motion. The microphone signal is used as the acoustic reference and reconstruction target during training and evaluation. To improve the representation of speech content and articulatory dynamics, we further incorporate depth-camera and UWB radar modalities as auxiliary inputs. The depth camera provides visual evidence of mouth movement and lip-shape variation, which is closely related to the vocal-tract configuration during speech production. The UWB radar captures motion-induced radio-frequency responses from facial and upper-body micro-movements, providing additional physical cues that are complementary to the optical vibration signal. Therefore, our dataset usage focuses on a multimodal reconstruction setting in which laser-speckle measurements provide the core vibration information, while depth-camera and UWB radar features enhance content discrimination, temporal alignment, and reconstruction robustness.

3. Method

We propose a weakly-aligned cross-modal speech reconstruction paradigm that departs from all dominant pipelines in silent speech, lip-reading, and radar-based speech recovery. Our framework fuses laser vibrometry, a depth camera, and ultra-wideband radar within a two-stage architecture, aiming to recover the full acoustic waveform from heterogeneous physical sensors under coarse synchronisation without any linguistic supervision. Importantly, no frame-level temporal alignment is enforced across modalities; only source-level pairing (same subject, same content, same experimental trial) is required. The encoders and the shared BiLSTM implicitly resample and align the heterogeneous temporal grids through deep feature projection, avoiding the hardware and software cost of sample-synchronous acquisition. The three sensors are physically motivated by the source-filter nature of speech production, and their complementary views are harnessed along two canonical prior dimensions: speaker identity and linguistic content. Laser vibrometry captures skin vibrations on the cheek that are directly homologous to the acoustic waveform, thus providing a weakened but direct signal of the voice source and speaker timbre. The depth camera records visible articulatory gestures of the lips and jaw, which primarily reflect the vocal-tract filter, yielding a weakened but direct representation of linguistic content. UWB radar detects deeper tissue motion related to subglottal pressure and chest-wall dynamics at two stand-off distances (1.2 m and 2.2 m). The two distance streams are processed by independent encoders whose outputs are concatenated before fusion, contributing both source and filter cues. We collect these physically grounded information streams and verify through reconstruction experiments that the direct associations among the modalities and their complementarity are sufficient to permit intelligible speech recovery even when the sensors are only coarsely synchronised in six-second windows and no phoneme boundaries, word alignments, or transcriptions are provided.

The pipeline consists of two decoupled stages. Stage I pretrains a dedicated encoder for each modality with an auxiliary classification objective that predicts the spoken word identity from the encoded features. The classification loss endows the encoders with a stronger ability to separate content classes while requiring no linguistic labels in the subsequent regression task; the auxiliary classifiers are discarded once pretraining converges. To handle disparate sampling rates and sensor drifts within the coarse window, each modality is resampled through a deep feature projection into a unified temporal grid of 240 frames per six-second window, producing an abstract semantic alignment that resolves temporal mismatch without explicit warping. The encoders are then frozen. Each encoder input is normalised in a modality-specific manner: laser waveforms are standardised by the global training mean and standard deviation; Kinect landmark trajectories are globally normalised per feature dimension over the training set; and UWB range–time maps are instance-normalised per sample to compensate for distance-dependent amplitude variation. These statistics are computed strictly on the training fold to prevent data leakage. Stage II concatenates the latent codes from all modalities and feeds them into a bidirectional LSTM followed by fully connected layers that directly predict a continuous log-Mel spectrogram. The regression network receives only the frozen modality-specific latent representations and accesses no text, phonetic, or speaker labels. Optionally, a learned content embedding (dimension 12 by default) can be concatenated to the fused latent vector as an auxiliary regularisation path; it is indexed by the corpus-level content category and requires no phonetic or word transcript. A Griffin–Lim vocoder subsequently converts the predicted spectrogram into a waveform. This design enforces a deliberate separation: the auxiliary-supervised representation structuring in Stage I creates a content-organised latent space, and the label-free regression in Stage II is thereby freed from simultaneously discovering linguistic structure, concentrating entirely on the physical sensor-to-audio mapping.

Figure 3. Proposed multimodal speech reconstruction and recognition framework (Stage I training and Stage II fine-tuning).

Figure 3. Proposed multimodal speech reconstruction and recognition framework (Stage I training and Stage II fine-tuning).

To train the regression network we design a three-component hybrid loss function that combines global spectral consistency with sensitivity to perceptually important local dynamics. Let $M_{t,f}$ and ${\widehat{M}}_{t,f}$ denote the target and predicted log-Mel spectrogram at time frame t and frequency bin f. The total loss is

$$\mathcal{L}={\lambda }_1{\mathcal{L}}_{\mathrm{MSE}}+{\lambda }_2{\mathcal{L}}_{\mathrm{mid}}+{\lambda }_3{\mathcal{L}}_{\mathrm{jump}},$$

(1)

with ${\lambda }_1=0.6$, ${\lambda }_2=0.3$, ${\lambda }_3=0.1$. The mean squared error term enforces global spectral fidelity,

$${\mathcal{L}}_{\mathrm{MSE}}={\displaystyle \frac{1}{TF}}\sum _{t,f}{\left(M_{t,f}-{\widehat{M}}_{t,f}\right)}^2.$$

(2)

A mid-range dominant loss concentrates the training signal on structurally informative regions by defining a soft weighting function $w\left(e\right)$ that peaks for mid-range errors and decays for both very small and very large errors,

$${\mathcal{L}}_{\mathrm{mid}}={\displaystyle \frac{1}{TF}}\sum _{t,f}{\left({\displaystyle \frac{e_{t,f}}{w\left(e_{t,f}\right)+ϵ}}\right)}^2,e_{t,f}=|M_{t,f}-{\widehat{M}}_{t,f}|.$$

(3)

The jump-aware loss preserves temporal dynamics by penalising frame-to-frame spectral differences, employing soft gating coefficients $g_t$ derived from the energy of the target difference and a directional penalty $r_t$ that activates when both target and prediction exhibit significant spectral movement,

$${\mathcal{L}}_{\mathrm{jump}}={\displaystyle \frac{{\sum }_{t=2}^T{d}_t(1+r_t)g_t}{{\sum }_{t=2}^T{g}_t+ϵ}},$$

(4)

where $d_t={\displaystyle \frac{1}{F}}{\sum }_f{(\Delta M_{t,f}-\Delta {\widehat{M}}_{t,f})}^2$ measures the magnitude discrepancy of the temporal difference $\Delta M_{t,f}=M_{t,f}-M_{t-1,f}$. This composite objective forces the model to reproduce both the static spectral envelope and the temporal evolution of the log-Mel representation.

Reconstructed waveforms are evaluated for semantic intelligibility, content and speaker preservation, and perceptual quality. Because no text appears during reconstruction training, any recovered words constitute emergent output-level linguistic intelligibility. A Whisper model is fine-tuned on the reconstructed training waveforms using the standard sequence-to-sequence cross-entropy objective,

$${\mathcal{L}}_{\mathrm{Whisper}}=-\sum _i\log P(y_i\mid y_{<i},X_{\mathrm{mel}}),$$

(5)

and transcription accuracy, expressed as word error rate, is measured on held-out speakers. Speaker and content preservation are quantified by extracting embeddings from a frozen Wav2Vec 2.0 model. For a given factor (content or speaker), let $C_k^{\mathrm{orig}}$ and $C_k^{\mathrm{recon}}$ denote the average normalised embedding of all samples belonging to class k in the original and reconstructed speech, respectively. The preservation of that factor is measured by the mean cosine similarity across classes,

$$\mathrm{Sim}={\displaystyle \frac{1}{K}}{\displaystyle \sum _{k=1}^K}{\displaystyle \frac{C_k^{\mathrm{orig}}·C_k^{\mathrm{recon}}}{\parallel C_k^{\mathrm{orig}}\parallel \parallel C_k^{\mathrm{recon}}\parallel }}.$$

(6)

Mean Opinion Score is collected from native listeners on a four-point scale ranging from completely unintelligible to high intelligibility and naturalness, including personal speaking style.

To examine how factors are encoded in the regression network, we perform linear probing on the BiLSTM hidden states. Linear logistic probes are trained to predict content identity, and probe accuracy indicates how explicitly each dimension is encoded. This probing analysis is employed specifically to validate the efficacy of the pretrained encoders: a linear probe trained directly on the frozen encoder outputs serves as an upper-bound estimate of the content-discriminative information captured by the modality-specific encoders, which can then be compared against the end-to-end model trained from scratch without auxiliary pretraining.

4. Results and Discussion

The quantitative results reported below combine the core pipeline outputs with post-hoc diagnostic analyses (linear probing, subjective listening tests, and wav2vec 2.0 embedding metrics) that were performed on the saved checkpoints and generated audio. The corresponding evaluation scripts are not included in the training repository for brevity. We evaluate the proposed framework under a strict leave-two-subjects-out protocol: for each of ten folds, all samples from two held-out speakers form the test set while the remaining speakers constitute the training set. A ladder-like improvement is consistently observed as more sensing modalities are fused.

The encoder ablation results are illustrated in Figure 5. The end-to-end model trained from scratch without auxiliary pretraining exhibits severe confusion across content classes, with an overall accuracy of 33.75%. In contrast, the full pipeline with pretrained encoder achieves 53.08% overall accuracy, demonstrating that the auxiliary classification loss equips the encoder with a discriminative residual structure that separates content categories. These results confirm that the pretrained encoder captures strong content-discriminative features and that the BiLSTM is essential for preserving this structured information during spectrogram regression.

Figure 4. Subject 19 – Real and predicted Mel frequency spectrum and sound waveforms.

Figure 4. Subject 19 – Real and predicted Mel frequency spectrum and sound waveforms.

Figure 5. Confusion matrices for end-to-end and pretrained encoder approaches.

Figure 5. Confusion matrices for end-to-end and pretrained encoder approaches.

Semantic intelligibility is assessed by fine-tuning Whisper on reconstructed training waveforms and testing on held-out speakers. Figure 6 and Figure 7 show the per-sample accuracy at word and sentence levels, respectively, while Figure 6 and Figure 7 summarise the accuracy distributions across modality combinations. The original audio reaches 99.5% accuracy after the same adaptation, verifying the fidelity of the ASR evaluator. At the word level, accuracy rises from 65.4% for laser alone to 81.7% for the full laser+depth+UWB combination, demonstrating that each additional modality contributes complementary articulatory information.

Sentence-level accuracy improves from 20.3% (laser) to 51.1% (laser+depth). The full three-modality system achieves 46.5%, a slight drop compared with laser+depth. This does not indicate a failure of UWB. UWB enriches the signal with high-frequency speaker-specific detail; for short words this extra detail improves intelligibility, while for long sentences the fixed-capacity BiLSTM must simultaneously preserve fine speaker traits and varied linguistic content, making the optimisation harder. The accompanying gains in joint content–speaker embedding similarity and MOS confirm that the trade-off is beneficial.

Content–speaker preservation is measured via Wav2Vec 2.0 centre cosine similarity. Figure 8 and Figure 9 show the per-sample cosine similarity at word and sentence levels, respectively, while Figure 8 and Figure 9 summarise the similarity distributions across modality combinations. Word-level joint content–speaker similarity grows from 0.779 (laser) to 0.847 (full).

Sentence-level similarity with laser alone is only 0.799, reflecting the difficulty of maintaining consistency over long utterances from noisy vibration signals. Adding the depth camera dramatically boosts the similarities (0.920), confirming its denoising and content-locking function. The full laser+depth+UWB system pushes them beyond 0.932, meaning the reconstructed sentences are nearly indistinguishable from the original speech in the Wav2Vec 2.0 embedding space. The simultaneous increase of similarity when UWB is added confirms that the radar stream complements the other two modalities without disrupting the already-learned linguistic structure.

To further validate the robustness of the proposed framework, we compare the multimodal fusion results against an extreme noise baseline. The noise audio is generated by severely degrading the original speech with a composite cocktail of babble noise (simulating multiple simultaneous speakers), additive white Gaussian noise, room reverberation (RT₆₀=1.2 s), band-pass filtering, dynamic range compression, hard clipping, and random impulse bursts, resulting in a highly negative SNR of $-8$ dB—an acoustic environment so hostile that human voice is effectively masked. As shown in Figure 6 and Figure 7, the tri-modal fusion system achieves a mean word-level accuracy of $81.7\%$ and sentence-level accuracy of $46.5\%$, whereas the noise-corrupted audio drops to $69.0\%$ and $23.1\%$, respectively. The margin is even more pronounced in the semantic embedding space: the mean Wav2Vec 2.0 cosine similarity for the fused reconstruction reaches $0.847$ at the word level and $0.932$ at the sentence level, far exceeding the noise baseline of $0.607$ and $0.605$ (Figure 8 and Figure 9). These gaps confirm that the physical sensor streams—laser vibrometry, depth-camera articulatory capture, and UWB radar—are intrinsically immune to the acoustic interference that cripples microphone recordings. Because the modalities measure tissue vibration, facial geometry, and physiological micro-motion rather than air-pressure waves, they continue to lock onto the target speaker even when ambient noise completely masks the audible voice. Consequently, the framework not only recovers intelligible speech under conditions where conventional microphones fail, but also preserves speaker identity and content fidelity in multi-speaker, high-reverberation environments representative of crowded public spaces.

Subjective listening tests (MOS) are summarised in Table 3 and Table 4. Single-laser speech is practically unusable (word average 1.00, sentence average 1.18) due to the severe noise inherent in the laser signal. The full laser+depth+UWB system reaches 3.080 for words and 2.674 for sentences, both well above the intelligibility threshold. The largest jump occurs when the depth camera is combined with the laser signal, and UWB further improves naturalness. Crucially, the sentence-level MOS of the three-modality system (2.674) is noticeably higher than that of laser+depth (2.097), confirming that the small Whisper sentence-level dip does not reflect a genuine perceptual loss; instead, the richer spectral detail brought by UWB enhances naturalness even if it challenges a data-limited ASR evaluator.

To examine whether the reconstruction quality is affected by the speaker’s linguistic background, we further break down the word- and sentence-level accuracy and cosine similarity by the native language of the held-out speakers (Chinese, Arabic, and English). Figure 10 and Figure 11 present the corresponding radar charts. At the word level, the accuracy profiles are largely consistent across the three language groups, with minor variations on specific words such as “ambulance” and “sanitize.” At the sentence level, English speakers generally achieve higher accuracy than Chinese and Arabic speakers, possibly due to the smaller number of English sentences in the corpus or clearer articulation. For cosine similarity, the trends are similar: the three language groups exhibit comparable similarity scores at the word level, while at the sentence level English again shows a slight advantage. These observations suggest that the proposed framework generalizes across linguistic backgrounds without systematic degradation for any particular language, although the limited number of speakers per language prevents strong conclusions. The overall high similarity and accuracy across all groups confirm that the physically grounded reconstruction is not biased toward a specific language.

Figure 4 compares the real and reconstructed mel-spectrograms and waveforms for subject 19 across five representative words. The predicted mel-spectrograms successfully recover the harmonic structure and temporal envelope of the original speech, although some high-frequency detail is smoothed. The corresponding waveforms preserve the overall amplitude contour and periodicity, confirming that the multimodal fusion framework captures the essential acoustic characteristics necessary for intelligible speech reconstruction.

The consistent ladder-like improvement from laser to laser+depth to laser+depth+UWB across all metrics confirms that each sensor covers distinct, partially overlapping regions of the acoustic–speaker–content manifold. Laser alone provides the acoustic foundation and speaker information, but without clean content constraints it yields unintelligible speech. Depth camera adds the missing articulatory content, drastically improving intelligibility, while also denoising the laser-derived speaker stream. UWB radar contributes additional physiological information that further refines speaker identity and spectral dynamics. The small sentence-level Whisper accuracy dip from 51.1% to 46.5% when UWB is added is therefore not a degradation but a consequence of the richer speaker detail; the simultaneous MOS increase from 2.097 to 2.674 confirms that the small Whisper sentence-level dip does not reflect a genuine perceptual loss.

Most prior work relies on strong alignment, textual or phonetic supervision, and unimodal visual or radar inputs. LipNet, AV-HuBERT, and LipSound2 operate on video and require aligned transcriptions or frame-level audio-visual correspondence. Radar ASR uses radar with strong text labels, while laser speech recovery reviews physical principles without a full synthesis evaluation. In contrast, our framework uses weakly aligned laser, depth, and UWB streams with auxiliary labels that are discarded after pretraining, and outputs continuous speech evaluated by both ASR and human listeners. The resulting Whisper WER of 18.3% and word MOS of 3.080 are not directly comparable to the other benchmarks due to differing tasks and data, but they illustrate the viability of reconstructing intelligible speech from silent, non-contact multimodal sensors without textual or phonetic supervision.

Taken together, these results demonstrate that linguistic intelligibility, content–speaker preservation, and perceptual quality obtain from accurate continuous spectrogram reconstruction based on physically complementary sensor streams under weak alignment, without any textual or phonetic supervision. The framework realises a paradigm in which language arises as a by-product of physical regression, validated by human listening, ASR compatibility, and representational consistency.

5. Conclusion and Future Work

This paper presented a weakly aligned multimodal speech reconstruction framework that fuses laser vibrometry, a depth camera, and ultra-wideband radar without any textual or phonetic supervision. The two-stage training paradigm, in which modality-specific encoders are first regularised by an auxiliary word-level classification task and then frozen for label-free spectrogram regression, proves essential for extracting discriminative features and enabling intelligible speech recovery. Comprehensive experimental results demonstrate the complementary nature of the three physically grounded sensors: the laser stream provides speaker-specific source information, the depth camera captures articulatory content, and the UWB radar contributes additional physiological detail that refines both source and filter representations. Under a strict leave-two-subjects-out protocol, the full laser+depth+UWB system achieves a word-level Whisper accuracy of 81.7%, a word MOS of 3.08, and a sentence MOS of 2.67, confirming that the reconstructed speech is intelligible to both automatic recognisers and human listeners, confirming that linguistic intelligibility emerges from the physically aligned multimodal observations without explicit linguistic supervision.

Future work will pursue several directions to improve the reconstruction quality and extend the applicability of the paradigm. First, the log-Mel spectrogram representation will be replaced with a modern neural audio codec, which is expected to produce more natural waveforms and better preserve fine phonetic detail. Second, the Griffin–Lim vocoder will be substituted with an adversarial generative network to enhance perceptual quality. Third, we will evaluate the framework on larger, more diverse, and multilingual datasets to systematically investigate the generalisation boundaries of the emergent representational structure and to move toward continuous, unconstrained speech scenarios. Finally, exploring lightweight on-device implementations could pave the way for practical assistive communication and privacy-preserving speech interfaces.