Sustainable Soundscape Monitoring of Modified Psycho-Acoustic Annoyance Model With Edge Computing for 5G IoT Systems

1. Introduction

The concept of Soundscape inside the Smart City has taken a main role in the urban environment. According to ISO 12913-1:2014, it is referred as the "acoustic environment as perceived or experienced and/or understood by a person or people, in context" [1,2]. In this way, soundscape means a paradigm change from noise control policies towards a new multidisciplinary approach as it involves not only physical measurements, but also subjective assessment coming from humans and social sciences which establish the focus on how people actually experience an acoustic environment in context [3]. The taxonomy of sounds in the soundscape description, which is defined in ISO 12913-2:2018 [2], can be highly advantageous when implementing deep learning methods for automatic sound classification, as stated in a study by Lopez et al. [4].

Several authors have taken different perspectives on soundscape monitoring and psycho-acoustic annoyance, as evidenced by studies by Mont et al. [5], Castrillon et al. [6], and Yang et al. [7]. These studies have explored the use of various parameters based on psycho-acoustic models, with a focus on Loudness [8], denoted as L, as the most relevant parameter. Other parameters such as Sharpness, Roughness, and Fluctuation Strength [9,10], denoted as S, R and F respectively, have also been considered. Moreover, in [11], authors offered a thoughtful study of different researches using psycho-acoustic metrics. More recently, alternative variations of the equivalent sound pressure level [12,13,14] have been proposed to describe the soundscape in different environments.

In the traditional vision, the soundscape description is focused on measurements for mappings which are usually based on the Equivalent Sound Level (named $L_{eq}$). However, $L_{eq}$ is not enough for this description when evaluating subjective annoyance or pleasantness, since similar values for $L_{eq}$ can lead to different feelings of the noise, perceived by different people, failing to provide information related to the environmental noise annoyance [15] and their psycho-acoustic properties. This is due to the information from $L_{eq}$ regarding the frequency characteristics. In addition, there are many sources of noise with low levels of $L_{eq}$ that produce an uncomfortable annoyance and even worse than the ones with high values of $L_{eq}$, for instance, an isolated tone from a mechanic vibration. To define metrics based on the human hearing system, different studies and techniques have been carried out and different methods have been defined in order to estimate the subjective annoyance (such as Zwicker’s [16] model or Moore’s [17] model). Nevertheless, all of them require high computational costs due to the complexity of the analysis and required signal processing.

In the case of the psycho-acoustic parameters, the subjective response produced by tonal sound is more annoying [18] and some authors proposed penalty parameters. In [19], the authors proposed a modification of the psycho-acoustic annoyance model by Zwicker’s in order to weigh the effect of tonal sounds in the subjective evaluation. The purpose in the current work is to modify the basic implementation established in [10] by adding the Tonality as a weighting parameter. Moreover, a validation of this advanced implementation is carried out by several measurement campaigns in different sound environments and comparing with results of the basic model, i.e. without Tonality.

This paper is organized as follows. After the introduction in which the motivation and aim of this work is presented, the methodology and materials used in the proposed system setup is described in Section 2, together with the Zwicker’s model and the modifications introduced in this work. In Section 3, the implementation of this advanced model is explained using two different approaches. Then, the use of these approaches in different environments is studied in Section 4, discussing the obtained results. Last section, concludes the study summarizing the discussion and focusing the main outputs.

2. Materials and Methods

The development of the algorithms involved in the Psycho-acoustic Annoyance (PA) model, based on the Zwicker’s model, was mainly focused in [10]. In this work, a modification of the basic implementation of this model is created, in order to introduce Tonality as a parameter, that can be involved with higher nuisance environments.

2.1. Zwicker’s model and Tonality

As previously mentioned, the Zwicker’s annoyance model [16] traditionally proposes the evaluation of four parameters (i.e. N, S, R and F) to determine the $PA$ as a non-linear combination of the aforementioned parameters.

1) Loudness (N): This magnitude describes the perceptual value of perceived loudness by the human ear. It is measured in $Sones$ on a linear scale and represents the subjective loudness of a sound. It does not take into account any perceptual distinction between "pleasant" or "annoying" sounds. The calculation procedure is standardised in the DIN45631/A1 [20] or ISO 532-1:2017 [8] standards.

2) Sharpness (S): This metric assesses the level of unpleasant sensory perception that a sound produces in humans, and considers the high-frequency components of the sound. It is often viewed as a representation of the centre of the spectrum and is measured in $Acums$ on a linear scale. Standardization for this metric is provided by DIN 45692 [21].

In our implementation, it is used the method defined by Zwicker and Fastl which is the one on which the standard is based [10].

3) Roughness (R): This metric describes the perceived fluctuation of a sound even when L or $L_{eq,T}$ remain unchanged. It analyses the effects with different degrees of frequency modulations (around 70 Hz) in each critical band (CB). The basic unit of R is $Asper$. In our implementation, the model used for R is the one defined by Jourdes [22] based on the optimised model of Daniel [23].

4) Fluctuation Strength (F): Describes how strongly or weakly sounds fluctuate. It depends on the frequency and depth of loudness fluctuations, around 4 Hz in each equivalent rectangular bandwidth (ERB). It is measured in $Vacils$.

The traditional Zwicker’s model takes into account the S contribution as long as it is greater than 1.75 $acums$. Equations (1) describe the determination of psycho-acoustic annoyance according to this model. In case equation (1b) has a S lower than 1.75 $acums$, $w_S$ is 0. Also, in the case of N, it takes into account the contribution of the 5^th percentile ($N_5$), which indicates, after sorting the data from lowest to highest, the value of N below which the 5^th percentile of loudness observations is found.

$$PA=N_5·(1+\sqrt{w_S^2+w_F^2})$$

(1a)

where

$$w_S=\frac{3·(S-1.75)}{4}·log(N_5+10)$$

(1b)

$$w_F=\frac{2.18}{N_5^{0.4}}·(0.4F+0.6R)$$

(1c)

The modified Zwicker model, shown in [19], introduces a new term in the equation (1a) that takes into account the tonal component of the evaluated sounds. This term is seen in the equation (2a), where the term $w_T$ in the equation (2b) takes into account the tonality (T). This component extraction has been implemented using the Aures tonality model [24], from Terhardt’s algorithm [25] for the extraction of the pitch of complex tonal signals.

$$PA=N_5·(1+\sqrt{w_S^2+w_F^2+w_T^2})$$

(2a)

where

$$w_T=\frac{6.41}{N_5^{0.52}}·T$$

(2b)

5) Tonality (T): According to [26], for the application of the Aures tonality model, after the spectrum analysis, the calculations are divided into two ways. One is the calculation based on the tonal components which involves the calculation of the excess sound pressure level (SPL) for each tonal component and its tonal weighting function, and the other is the calculation of the total loudness of the whole spectrum and the loudness of the noise spectrum, where noise spectrum means the spectrum that removes the tonal components or narrowband components. Finally, the tonal loudness weighting is obtained, and the tonal weighting is calculated for all components with positive SPL excess levels. It is important to note that the Aures tonality model considers the level of dependence of the human auditory system through N [27]. It also shows a way to extract a single value combining the tonal weighting and the tonal weighting of loudness.

2.2. Study of JNDs in sound quality metrics

Just Noticeable Difference (JND) is an important concept in psycho-acoustics. JNDs refer to the smallest change in a stimulus that a person can detect. This can be related to physical properties of sound such as loudness, pitch, and timbre, as well as more subjective qualities like attractiveness and emotion.

JNDs are important in understanding the way people perceive sound. They are used to measure how sensitive people are to changes in sound, which is valuable for applications such as designing audio equipment, creating soundtracks for movies, or understanding how different types of music affect people.

The magnitude of a JND is determined by several factors, including the intensity of the sound, the duration of the sound, and the context of the sound. For example, a small change in a sound may be much more noticeable when it is played at a higher volume or when it is in a noisy environment. JNDs can also be used to compare different people’s sensitivity to sound. People with lower JND are more sensitive to sound changes, while people with higher JND require greater sound changes in order to perceive differences. In this work, attending to the definition of JND provided by Ernst H. Weber [28], the JND is a statistical quantity, rather than an exact one. So, from trial to trial, the difference that a given person notices will vary somewhat, and it is, therefore necessary to conduct many trials in order to determine the threshold. For the purpose of this research, this statistical nature can be considered similar to the computation of a standard deviation, taking into account a reference sample instead of the mean value of the set of samples.

According to the study exposed in [29], the JNDs for the main psycho-acoustic metrics for a refrigerator are 0.5 $Sones$ for loudness ($\Delta N$), 0.08 $Acum$ for sharpness ($\Delta S$), 0.04 $Asper$ for roughness ($\Delta R$), and 0.012 $Vacil$ for fluctuation strength ($\Delta F$). These values were obtained after doing auditory experiments which were designed to determine the difference between the reference and comparing with the stimulus in terms of each sound quality characteristic. The duration of each pair of experiments was 10.5 seconds (5.0 s reference stimuli + 0.5 s silence + 5.0 s comparison stimuli). They had 40 participants, between 20 and 35 years old and with normal hearing. The stimuli were presented with a headphone system in a semi-anechoic chamber.

As the Zwicker’s PA is a complex magnitude composed by all the previous metrics, we can get the JND by deriving the composed metric (in this case PA) and propagating the error with the related metrics, shown in equation (3), of the JNDs of the previous metrics within equation (1a).

$$\Delta f=\frac{\partial f}{\partial x}\Delta x+\frac{\partial f}{\partial y}\Delta y+...$$

(3)

Considering equation 3 and taking into account the values from [29], we obtain,

$$\Delta PA=\frac{\partial PA}{\partial N}\Delta N+\frac{\partial PA}{\partial S}\Delta S+\frac{\partial PA}{\partial R}\Delta R+\frac{\partial PA}{\partial F}\Delta F$$

(4)

Where the different partial derivatives are

$$\frac{\partial PA}{\partial N}=1+\sqrt{11881·\frac{{(\frac{2F}{5}+\frac{3R}{5})}^2}{2500·N^{4/5}}+1.268·{10}^{30}\frac{{ln(N+10))}^2·{(\frac{3S}{4}-\frac{21}{16})}^2}{6.721·{10}^{30}}}-\frac{N·\frac{11881·{(\frac{2F}{5}+\frac{3R}{5})}^2}{3125·N^{9/5}}-\frac{2.535·{10}^{30}·ln(N+10)·{(\frac{3S}{4}-\frac{21}{16})}^2}{6.721·{10}^{30}·(N+10)}}{2·\sqrt{\frac{11881·{(\frac{2F}{5}+\frac{3R}{5})}^2}{2500·N^{4/5}}+\frac{1.268·{10}^{30}·{\left(ln(N+10)\right)}^2·{(\frac{3S}{4}-\frac{21}{16})}^2}{6.721·{10}^{30}}}}$$

(5)

$$\frac{\partial PA}{\partial S}=\frac{6.338·{10}^{30}·N·{\left(ln(N+10)\right)}^2·(\frac{9S}{8}-\frac{63}{32})}{6.721·{10}^{30}·\sqrt{\frac{11881·{(\frac{2F}{5}+\frac{3R}{5})}^2}{2500·N^{4/5}}+\frac{1.268·{10}^{30}·{\left(ln(N+10)\right)}^2·{(\frac{3S}{4}-\frac{21}{16})}^2}{6.721·{10}^{30}}}}$$

(6)

$$\frac{\partial PA}{\partial R}=\frac{11881·N^{1/5}·(\frac{12F}{25}+\frac{18R}{25})}{5000·\sqrt{\frac{11881·{(\frac{2F}{5}+\frac{3R}{5})}^2}{2500·N^{4/5}}+\frac{1.268·{10}^{30}·{\left(ln(N+10)\right)}^2·{(\frac{3S}{4}-\frac{21}{16})}^2}{6.721·{10}^{30}}}}$$

(7)

$$\frac{\partial PA}{\partial F}=\frac{11881·N^{1/5}·(\frac{8F}{25}+\frac{12R}{25})}{5000·\sqrt{\frac{11881·{(\frac{2F}{5}+\frac{3R}{5})}^2}{2500·N^{4/5}}+\frac{1.268·{10}^{30}·{\left(log(N+10)\right)}^2·{(\frac{3S}{4}-\frac{21}{16})}^2}{6.721·{10}^{30}}}}$$

(8)

According to equation (4) the JND of Zwicker’s psycho-acoustic annoyance depends on N, S, R and F, but the most complex value is N and it will have a major contribution to JND.

3. Architecture implementation

The methodology used in this work to design the system followed a double strategy: a) acquisition and sample of the sound signal, together with the computation of the psycho-acoustic metrics is performed in the Edge node of the network, and b) acquisition and sample of the sound signal in an external device, e.g. implementation in a Fipy node [30], and then transmission of the audio file to the Edge node through the Message Queuing Telemetry Transport (MQTT) protocol for the computation of the metrics.

The different algorithms of the psycho-acoustic metrics N, S, R, F and T have been implemented in Matlab, from [31,32]. Similarly, these algorithms have been integrated in Simulink models that allow their conversion to C/C++ code using the Matlab "coder" tool and their direct installation in Single Board Computer (SBC) type platforms. In our case, it has been integrated on a Raspberry Pi 4B+ with 4GB of RAM (RPi4). This platform acts as the Edge node of the system, which allows calculating the psycho-acoustic metrics from the recorded audios.

Figure 1 (considering Fipy with a microphone and sending the audio to the Edge) and Figure 2 (integrating the microphone in the Edge) show the Simulink schematic of the complete algorithm that determines the different psycho-acoustic metrics, as well as the modified psycho-acoustic annoyance [19], taking into account the two strategies explained in this section, both options have been implemented.

In Figure 1, the option with Fipy (MCU+modems) from Pycom, where an ESP32 MCU node with various wireless connectivity options (i.e. Wifi, BLE, Lora, Sigfox and LTE-M/NB-IoT) connected to an INMP441 MEMS microphone, samples audio, performs windowing and sends the audio to the Rpi4, which allows the calculation of the different metrics, is shown. Figure 2 shows the option in which the RPi4 itself, which acts as an Edge, has a USB microphone connected.

In both options, the output, which is an array with different metrics, is sent to an online Cloud. In this work, Thingspeak platform [33] is used, with MQTT protocol for data communication, because of its quick integration in Matlab.

3.1. Calibration procedure

The calibration is done using a file with sinusoidal tone at 1kHz in the scheme of Figure 1. The result of the loudness with this file is N 40 sones, measuring a sound pressure level of 40 dB (as it is stated in the definition of loudness). Also, the other parameters are fixed accordingly to its definition in the Matlab-generated code.

4. Results and discussion

4.1. Application of the system to compare models’ results

After the calibration procedure, the values of the psycho-acoustic parameter values of a pneumatic hammer from a digital recording1 were measured during 17 minutes. This sound source was rendered using a Behringer MS16 loudspeaker and sound was captured with the RPi4, sending values in the cloud every 1s.

The produced PA by this sound source has been calculated using both the basic Zwicker model, Eq. 1, and the modified model with tonality, Eq. 2. Figure 3 shows the values calculated with these two models (Zwicker’s PA and modified PA with tonality), showing that the appearance of tonal sounds in a noisy environment produces greater subjective annoyance. The calculation of the annoyance model with the tonality was launched, and it can be seen in this figure with increased values. It is also observed that this noise, although annoying at certain levels, is not very tonal so a large percentage of the measurements of the two models coincide.

4.2. Application of the system to a specific site: UCAM Campus

In the application of the proposed system to a real scenario, a set of Single Board Computers (SBC), in particular Raspberry Pi 4B+, was used to deploy a wireless acoustic sensor network across the campus of the Universidad Católica de Murcia (UCAM) located in Murcia, Spain. Figure 4 shows the location of the 7 nodes distributed in three different soundscapes, both indoors, a canteen and a monastery cloister, and outdoors, a square called pergolas.

Figure 5 shows the results of a 10s measurement (one shot) in the three devices located at the canteen (Figure 5a–c). Moreover, results from a long recording (700s) in an intermediate location within the canteen are shown in Figure 5d. This zone is one of the busiest areas of the campus, therefore it is observed that the psycho-acoustic annoyance is higher at specific moments due to a higher concentration of people.

Also, Figure 6 shows the plots of in 10s measurement two devices located at the cloister of the monastery of the UCAM (Figure 6a,b). The low annoyance is one of the main characteristics in this location, as it is shown in the long recording (630s) (Figure 6c).

Third environment, pergolas square, is an outdoor zone that can be both crowded and quite place depending in the period of time, where staffs and students can use in break times. Figure 7 shows the psycho-acoustic annoyance of a 10s measurement of two deployed devices in this square (Figure 7a,b). At the moment when the measurement was done, the outdoor environment was quite calm, the nuisance is not too high, as it is shown in the long recording (640s) (Figure 7c).

Figure 8 shows the relationship between SPL, in dBA, and the psycho-acoustic annoyance in the three locations on the UCAM campus, the canteen (Figure 8a), the cloister (Figure 8b) and the outdoor pergolas (Figure 8c). The three cases show a quasi-linear relationship.

4.3. JND in the UCAM Campus

With the values obtained in the error propagation considered in section 2.2, the application of $\Delta PA$ produces a range of JNDs for the Zwicker’s psycho-acoustic annoyance in the canteen of the campus with an average value for $\Delta PA$ of 0.3553 and a variance value of 0.1012, which determines a mean percentage of 4.2% of relative variation for the PA ($\Delta PA/PA$) to have perceptual a change in this environment. In the cloister area of the campus the mean value for the PA JND is 0.7489 and the variance is 0.0044 , which determines a mean percentage of 38.7% of relative variation for the PA ($\Delta PA/PA$) to have perceptual a change in this environment. This value means that silent places can have a higher variation in the perception of annoyance. And in the case of the outdoor pergolas the PA JND has a mean value of 0.5008 and a variation value of 0.0289, which determines a percentage of 6.6% of relative variation for the PA ($\Delta PA/PA$) to have perceptual a change in this environment.

5. Conclusions

This paper presents different implementation options for a modified psycho-acoustic annoyance algorithm and compares it with Zwicker’s psycho-acoustic annoyance, introducing a term that weights the tonality of sounds. The implementation of these algorithms was done in different Raspberry Pi 4B+ and applied to different locations in the UCAM Campus in Murcia (Spain). According to reference [29], the JNDs of PA values computed from sound quality metrics obtained in these measurements are kept in a range between 4% and 38% depending on the environment. According to our experience in this study silent places can have a higher variation in the perception of annoyance.

The model exported to an SBC (RPi4B) has been applied to measure noise generated by a recorded pneumatic hammer and different locations in the campus. It has been observed that from the values calculated with the two models, the occurrence of tonal sounds in a noisy environment produces greater subjective annoyance. The MATLAB implementation to integrate in the RPI4B+ with the codegen command ave been have published in https://github.com/jausegar/urbauramon/tree/master/URPAA_Rasp_simulink h.

Both implementations, the one from Figure 1 and the one from Figure 2, showed a good performance in computing all the metrics in real-time, but the offloading in the first case allowed a better performance as the computing load in the sampling and windowing was done in the external node.

For future work, it is intended to extend this study to a larger number of noisy environments to describe other soundscapes for classification purposes.