Impact of face masks on acoustic parameters for Forensic Speaker Recognition

1. Introduction

Forensic speaker recognition (aka forensic speaker comparison) aims to compare one or several recordings of a questioned speaker with one or several recordings of a suspect, and has been receiving a lot of interest given that it plays a key role in criminal investigations, delivering important conclusions to the Justice System. It differs from regular speaker recognition in several ways, including short utterances, background noises, low voice quality, among others technical aspects. No less important is the approach itself, especially the level of robustness, accuracy and reproducibility given that the conclusions have a direct impact in solving crimes [1,2,3].

Three main approaches can be distinguished: auditory-phonetic based on the auditory examination of recordings by trained phoneticians or linguists; acoustic-phonetic methods involving the measurement of various acoustic parameters such as fundamental frequency and its variation, formant frequencies and speech tempo; automatic and semiautomatic speaker recognition in which the central processing stages (feature extraction, feature modelling, similarity scoring and likelihood ratio computation) operates fully or partial automatically [4,5,6,7,8].

The COVID-19 pandemic has led to a dramatic increase in the use of protective face masks, given their mandatory use in almost every daily situation, preventing the virus from spreading. However, the use of face masks has also had an impact on speech processing technologies as they act as voice barriers or filters, affecting not only the acoustic properties of the signal but also the speech patterns, introducing both consequential and adaptative changes in the human voice [9,10].

Although the COVID pandemic was declared finished, forensic laboratories continues to receive requests for analysis of investigations relating to the pandemic period once the time of justice always refer to activities that took place previously. For example, contracts made using identity theft with a telecommunications or insurances companies during the pandemic are now being investigated. Also, the number of citizens who have kept the habit of wearing a protective mask, especially if we look at airports, is not negligible. On the other hand, this study contributes to better knowledge of the impact of using masks as disguise voice technique.

The present study focuses on the impact of face masks in seven acoustic parameters used in acoustic-phonetic approach for speaker recognition. A subsequent study on the effect of the presence of protection masks in forensic automatic speaker recognition (FASR) systems is being performed and will be presented in a near future. Both studies make use of the Forensic Multilingual Voices Database (FMVD), developed under the “Competency, Education, Research, Testing, Accreditation, and Innovation in Forensic Science” (CERTAIN-FORS) project, funded by the European Union (EU) and coordinated by European Network of Forensic Science Institutes (ENFSI).

2. Related Work

Analyses have been conducted on sustained vowels produced with and without a face mask. Some studies detected no acoustic differences in terms of fundamental frequency (F₀), jitter, shimmer, and harmonics-to-noise ratio (HNR), first and second formants (F₁ and F₂), between samples obtained from individuals not wearing protection masks and wearing surgical masks [11,12,13,14]. Joshi et al. also did not observe differences between the cloth and KN95 type’s masks [14]. However, Gojayev et al. found significant differences in HNR and shimmer when FFP3 masks were used [13]. Lin et al. observed a higher sound pressure level (SPL), a small decrease in jitter and shimmer, and an evident decrease in F₃ when wearing clinical masks [15]. Georgiou found that F₂ was altered by the effects of both cotton and surgical face masks [16].

Considering acoustic analysis on continuous speech, Magee et al. [17] found that power distribution in frequency, measures of timing, and spectral tilt were significantly impacted by wearing an N95-type mask; cepstral and harmonics-to-noise ratios remained unchanged for all masks (surgical, N95, and cloth). Nguyen et al. [18] found significant attenuation of mean spectral level in the 1–8 kHz region and no significant change at 0–1 kHz when wearing face masks (surgical and KN95). Knowles and Badh [19] studied the effect of surgical and KN95-type masks in different speech styles (normal, loud, and clear), obtaining results in line with the previous studies and consistent across all three speech styles.

Nguyen et al. [20] also investigated the impact of face masks on the acoustic features of fricatives, observing significant lower root mean square amplitude and /f/ center of gravity when wearing a N95-type mask compared with non-mask conditions. These results were consistent with previous studies performed by Fecher and Watt [21] and Saigusa [22].

It is important to mention that all results from the above-discussed studies were obtained with small datasets.

Latoszek et al. performed a systematic review and meta-analysis of the impact of protective face coverings on acoustic markers in voice [23]. The authors found nine eligible studies, resulting in 422 participants, and the results from the meta-analysis did not show significant differences between the absence and presence of masks.

To evaluate the effects of face masks on speech production between Mandarin Chinese and English and their implications for forensic speaker identification, Geng et al. [24] performed a cross-linguistic study. Voice samples were collected from thirty volunteers who were Mandarin native speakers and fluent in English as a second language. Each participant gave two text-reading samples: one in Mandarin without a mask, one in Mandarin wearing a surgical mask, one in English without a mask, and one in English wearing a surgical mask. The results of acoustic analysis showed that mask speech exhibited higher F₀, intensity, HNR, and lower jitter and shimmer than no mask speech for Mandarin, whereas higher HNR, lower jitter, and shimmer were observed for English mask speech.

3. Materials and Methods

The FMVD was developed with the collaboration of several Forensic Institutes, members of the ENFSI Forensic Speech and Audio Analysis Working Group (FSAAWG), representing nine countries: Lithuania (LT), Croatia (HR), Romania (RO), Türkiye (TR), Ukraine (UA), Portugal (PT), Georgia (KA), Hungary (HU), and Spain (ES).

All samples were collected from volunteers who signed an informed consent statement designed for the development and distribution of the FMVD among ENFSI FSAAWG members. Each person was recorded reading a selected text without a mask (NM), wearing a surgical mask (SU), and wearing an FFP2 mask (FP). A small number of samples in Russian (RU), Polish (PL), and German (DE) languages were also recorded, given the existence of native speakers in the collaborating Forensic Institutes.

Samples from mobile (MB) communications in Lithuanian, Croatian, Portuguese and Spanish, and landline (LL) communications in Lithuanian, Polish and Russian were also collected, however, in some cases in a small number.

The project provided the masks and the text, controlling as possible the sampling process. An excerpt from “The Little Prince,” written by Antoine de Saint-Exupéry, was selected, avoiding translation issues once this book is already translated in almost every language. Recording device models used in each of the participating countries and the respective sampling rates of the obtained voice samples are presented in Table 1. All recordings were made in WAV format, mono signal, resolution of 16 bits, indoors, and approximately 2.5 minutes long. Acoustic conditions could not be normalized once the collection process took place in several audio laboratories from different countries.

Table 2 shows the distribution of the 860 volunteers by language, sex, and recording channel: microphone or recorder, mobile (MB), and landline (LL) communications. Given their reduced size, subsets with less than 10 samples were left out of the presenting study, namely voice samples in German and Polish languages; Spanish, Croatian, and Lithuanian mobile recordings; and landline communications in Russian.

Recordings were pre-processed using a Matlab tool developed for removing silences from voice samples according to a background noise threshold. The same threshold was applied to all the samples from each subset of data per language and recording channel, resulting in over 116 hours of recordings for the study. Figure 1 presents an example of the silence removal pre-processing process.

Seven acoustic parameters were chosen to be studied: F₀, difference between 1st and 2nd harmonics (H₁-H₂), intensity, speech rate, HNR, jitter, and shimmer, as they are used in speaker recognition, particularly on the acoustic-phonetic approach [5,6,8,24,25,26,27,28]. Their computation was performed using Praat (version 6.3.06).

Statistical analysis was conducted using Minitab Statistical Software (version 21.4.2). Since it was found that the requirements for carrying out an analysis of variance (ANOVA) were not satisfied, namely the normality of the residuals, for each language and sex, a linear mixed effects model was applied to analyse statistical differences on the above parameters between with and without each face mask type, using subjects as a random factor. A similar approach was used by by Geng et al. [24].

4. Results

Observation of the spectrograms showed that the face masks work as a low-pass filter at approximately 1 kHz (Figure 2). This observation is in accordance with previous studies findings [17,18,29].

Figure 3, Figure 4 and Figure 5 shows the average and standard error values of the seven acoustic parameters obtained from the samples collected using a microphone or recorder, by language, sex and mask type. Results for the same quantities obtained from mobile and landline recordings are presented in Figure 6, Figure 7 and Figure 8.

Table 3 summarizes the results of the linear mixed effects model on the acoustic parameters, by sex and mask type, for all languages recorded with a microphone or recorder.

For the samples collected with a microphone or recorder, preliminary observations show that the protection masks have almost no effect on the smallest subsets of data, namely RU and ES. Given the overall results for the rest of the dataset, this observation indicates the need to take more samples to better assess the impact of protective masks on acoustic parameters in these languages.

The presence of a face mask was significant for F₀ in almost all men, regardless of the spoken language; only HU males wearing SU masks were unaffected. F₀ in females wearing SU masks speaking in PT, LT, and HR was significantly affected, as well as in PT and HU females wearing FFP2 masks. The impact of wearing face masks translates into an increase in F₀.

Face masks were statistically significant for shimmer in TR, PT, and HU speakers and UA males. SU masks revealed a significant impact on shimmer in both genders speaking RO and females speaking KA.

The overall effect translates to a decrease in shimmer average value with the exception of SU masks in RO speakers in which an increase in shimmer’s value was observed.

HNR was significantly affected by the presence of masks in both genders speaking LT, TR, PT, and HU. FFP2 masks showed a significant impact on HNR in HR speakers of both genders and in RO males. SU masks affected the same parameter in females speaking KA. The presence of face masks caused an increase in the HNR average value except for KA females, for which a decrease in HNR was observed.

A significant effect on speech rate was observed in RO speakers, TR females, and UA males. SU masks affected females speaking in LT, UA, and KA, as well as TR males’ speakers. FFP2 masks showed a significant effect on both males and females’ PT speakers. The effect of face masks is characterized by a decrease in speech rate, apart from PT speakers, where an increase in speech rate was detected.

The presence of protection masks for jitter was statistically significant in KA females, leading to an increase in its value, and in TR and HU male speakers wearing an FFP2 mask, for which a decrease in jitter value was observed.

Intensity was significantly affected by the presence of masks in UA males, KA females, and LT of both genders. FFP2 masks showed an impact on intensity in females speaking RO, UA, and HU, as well as in both males and females’ HK speakers. SU masks revealed an impact on intensity only in PT male speakers. Intensity increased with the presence of protection masks except in KA females’ speakers, which presented a decrease in its value when wearing an FFP2 mask.

A significant effect on H₁-H₂ was observed in UA females. FFP2 masks affected females speaking in HR and RO, as well as LT and PT males’ speakers. SU masks showed a significant effect on females’ HU speakers. The effect of face masks is characterized by an increase in H₁-H₂ for male speakers and a decrease in its value for females, apart from HU female speakers, where an increase in H₁-H₂ was detected.

Table 4 summarizes the results of the linear mixed effects model on the acoustic parameters, by sex and mask type, for Spanish and Portuguese mobile communications samples, and for Lithuanian landline recordings.

The results of the linear mixed effects model applied to the samples obtained via mobile communications (Table 4) revealed, for PT speakers, a significant effect on F₀, HHR, and shimmer in both genders, on intensity for males and on H₁-H₂ for females’ speakers. SU masks revealed a significant impact on jitter in both genders. Speech rate was affected in females wearing FFP2. The effects were translated into an increase in F₀, H₁-H₂, intensity, and HNR average values and a decrease in speech rate, jitter, and shimmer.

The analysis performed on mobile recordings collected in ES language showed a significant impact on intensity for male speakers and females wearing FFP2 masks. FFP2 masks significantly affected speech rate, intensity, and shimmer for males and jitter for females. Both mask types revealed an impact on HNR and H₁-H₂ for female speakers.

Recordings via landline communications were studied only for LT speakers, and the results revealed a significant effect on intensity in female speakers wearing both masks and males wearing a FFP2 mask. F₀ was affected in males (Table 4). These effects led to an increase in both parameters’ values.

5. Discussion and Conclusions

This study aims to investigate the impact of two types of protection masks on seven acoustic parameters used in the acoustic-phonetic approach for forensic speaker recognition by sex and language.

The results showed that F₀, H₁-H₂, intensity, speech rate, HNR, jitter, and shimmer were affected by the presence of face masks, denoting different behaviours depending on mask type, sex, language and recording channel.

An increase in F₀, particularly in males, was observed in the presence of face masks of both types. Also, an overall increase in HNR was detected, as was an increase in intensity, more evident when wearing FFP2 masks. A decrease was denoted in shimmer value, as well as in speech rate, for which females wearing SU masks were more affected. The effect of face masks on H₁-H₂ was characterize by an increase of its value for males and a decrease for female speakers. Jitter was the less affected parameter, still denoting changes in the presence of FFP2 masks for both genders.

These findings support that, in the presence of face masks, which act as a low pass acoustic filter, speakers tend to perform phonetic and acoustic adjustments to compensate for the filter effect, trying to improve speech intelligibility [10,24,30,31].

The results for recordings via mobile communications show that F₀, HNR, and shimmer were affected in a similar way to the samples collected with a microphone or recorder. On the other hand, intensity was more affected by the presence of masks in mobile recordings; speech rate decreased in mobile communications and increased in microphone recordings; jitter was significantly affected in mobile communications when in microphone recordings a small impact was observed; and the impact on H₁-H₂ revealed a different behaviour from the one observed in microphone recordings, increasing its value for females and decreasing for male speakers. These comparisons were only possible for ES and PT voice recordings.

When comparing the obtained results for landline and microphone or recorder samples, different responses were also observed. Only F₀ and intensity were significantly affected in landline communications, while in samples recorded with a microphone or recorder, only jitter and shimmer showed no significant impact in the presence of protective masks. These comparisons were only possible for LT voice recordings.

Given the observed impact of face masks on the studied acoustic parameters, care must be taken when performing acoustic-phonetic forensic speaker recognition examinations.

7. Future work

• Increase the number of samples in languages with smaller datasets.
• Collect more samples from mobile and landline communications to assess if the impact of face masks is also language dependent in these channels.
• Add more languages to the FMVD.
• Study the impact of face masks in more acoustic parameters.
• Finish the ongoing study about the impact of face masks on the FASR approach.
• Development of a mask detector tool from speech signals