Danger Detection Through Audio Captioning and Classification Fusion

1. Introduction

Sound is a constant presence in our everyday lives, from the alarm that wakes us in the morning, to the car horn when stuck in traffic, to the laughter we share with friends. People rely on it continuously to understand their surroundings. Hearing conveys rich information about the environment, often without any deliberate effort, and it also serves more focused purposes such as drawing attention and raising alarms, highlighting the need for protection. Yet the same sound is tied to different memories and perceptions for each individual, so its meaning is never fixed. Sound is, in this sense, inherently ambiguous, and this ambiguity makes certainty especially critical when building a danger recognition system, where a false alarm or a missed detection both carry real consequences. The definition of danger adopted in this work is grounded in the framework established by the International Organization for Standardization (ISO). ISO 7731:2003 [1] defines auditory danger signals as sound signals designed to warn people of hazardous situations in public and work areas, requiring them to be clear, distinct, and easily recognizable. However, the standard covers exclusively intentionally designed alarm signals and does not extend to naturally occurring dangerous acoustic events, such as glass breaking, gunshots or human distress sounds like screaming or crying, that arise in the context of violent or otherwise dangerous incidents. For this reason, the definition of dangerous sounds adopted in this work extends beyond the scope of ISO 7731:2003 to include such naturally occurring acoustic events, following definitions used in the acoustic surveillance literature [2,3,4].

Computer audition is the field concerned with giving machines the ability to interpret sound and has matured from task specific models toward general purpose systems that can describe an acoustic environment rather than merely label it [5]. This capability is central to detecting hazardous events and audio is an attractive complement to vision-based surveillance. Microphones capture events from every direction rather than within a fixed field of view, acoustic waves reflect around occluding objects, and audio remains robust to lighting conditions while being comparatively inexpensive to acquire and process in real time [6,7]. However, recognizing an isolated acoustic event is not equivalent to detecting danger. A single sound such as a scream or a loud bang can occur in both threatening and entirely benign contexts, which has motivated a shift from classifying individual events toward estimating the risk conveyed by the acoustic scene as a whole [8].

We focus our search in violence detection, a field investigated mostly by analyzing audio and video [9,10,11]. This is a good motive that shows that violence and subsequently also danger can be more effectively addressed as a multimodal problem, meaning that using audio and video, and in our case audio and language, can benefit the detection of crucial danger events more effectively [12,13].

Several danger detection systems, across both audio and video modalities, still treat the problem as multiclass classification, assigning a fixed behavioral label to an input without any semantic reasoning about the surrounding context. Such approaches have been seen in audio-based systems such as Omarov and Altayeva’s 8-category sound classifier [2], as well as video-based systems such as Wei et al.’s 5-category campus behavior detector [10]. Other systems that do incorporate language rely on zero-shot predictions, converting audio directly into words without generating a more detailed description of the scene [14]. Still others attempt to compensate by requiring the appearance of multiple sequential danger events before flagging a real incident [15]. Work in the image captioning domain suggests a more effective alternative. In the image domain, danger is rarely a property of a single isolated event, but of the broader scene in which that event occurs [8]. Capturing this requires semantically rich descriptions of the full acoustic scene, rather than sparse labels or isolated keywords, so this is the gap our work addresses.

One of the basic aspects for systems that detect danger is the need for fast detection, so the proposed systems must focus on fast predictions that rarely miss danger, while not producing many false alarms. Also, based on the research of Zhu-Zhou et al. [16] that focuses on real-time systems for acoustic violence detection, the need for low computational cost is of high importance. Moreover, our uncertainty fusion rule follows a broader pattern in resource conscious audio classification, where an inexpensive first-stage decision resolves clear cut cases and a more capable classifier is reserved for ambiguous ones. Zhu-Zhou et al. [16] apply this principle using a lightweight linear detector to screen segments before forwarding uncertain cases to a deep classifier, with the screening threshold derived empirically from the distribution of training scores rather than fixed in advance. In our current research paper we adopt the same logic.

Considering all the aforementioned, in this study we propose an acoustic-semantic framework as depicted in Figure 1. In this design we first train a convolutional neural network (CNN) for binary danger/no-danger classification using audio classification datasets (ESC-50, UrbanSound8K and AudioSet). Following the acoustic classifier, textual descriptions are extracted for each audio clip using a pretrained audio captioning model called CoNeTTE. A custom danger lexicon combined with SBERT semantic similarity scoring is applied to these captions to identify danger indicative content, producing a labeled caption dataset. A DistilBERT model is then fine-tuned on this caption dataset for danger classification. The predictions of the acoustic classifier (CNN) and the semantic classifier (DistilBERT) are combined through a rule-based, uncertainty fusion strategy. This fusion design approach achieves a danger class recall of 0.82 and F1-score of 0.67 on UrbanSound8K, improving over the standalone CNN classifier (recall 0.70) while preserving a transparent, human readable decision trail.

We further evaluate the fusion rule on two external, previously unseen datasets (XD-Violence and DESED) sourced from different acoustic domains than our training data, to assess the generalization of the fusion strategy beyond the conditions it was designed on. The system is demonstrated on a dataset repurposed from general sound classification corpora (ESC-50, UrbanSound8K, AudioSet) rather than a corpus built for danger detection. It is therefore presented as a proof of concept for an approach intended to be applied to dedicated danger datasets as they become available.

Within this context, this work addresses two research questions. RQ1: can potential danger be detected from audio alone, and what makes audio an appropriate and practical modality for the task? RQ2: does automatic audio captioning, used as an intermediate semantic representation, improve danger detection, and in what way? Across both, the guiding objective is to maximize recall of the danger class while keeping precision as high as possible, that is, to detect danger whenever it occurs while raising as few false alarms as possible.

To address these questions, this study delivers three main contributions. First, we introduce a hybrid, uncertainty-driven fusion framework that combines direct raw-waveform acoustic classification with a semantic natural language processing branch, offering a novel approach to environmental danger detection. Second, by utilizing automatically generated audio captions as an intermediate representation, our pipeline provides an implicit, human-readable justification for its decisions, directly addressing the interpretability gap of traditional “black-box” deep learning models. Third, we design and evaluate a resource-conscious sequential activation strategy that reserves the computationally heavier semantic analysis exclusively for ambiguous cases, maintaining a low computational profile. The robustness and boundary limitations of this framework are extensively validated on both internal test sets and two entirely unseen external datasets representing distinct acoustic environments.

The paper is structured as follows. In Section 2, related work on audio-based danger detection is referenced, in Section 3, the methodology, dataset preparation, architecture, and technical details of the proposed method are introduced, in Section 4 the experimental results are presented, in Section 5 the results are discussed and analyzed, and finally, in Section 6, the research is summarized and concluded, presenting future work and the new directions it opens up.

2. Related Work

The earliest work on audio-based danger detection relied on classical machine learning. Souto et al. [17] approached domestic violence detection as acoustic scene classification into violent and non-violent scenes, extracting handcrafted features such as MFCCs and training conventional classifiers including Support Vector Machines (SVM), k-Nearest Neighbors (k-NN), and Random Forests. In a broader study of classical audio signal processing methods (Gaussian mixture models, hidden Markov models and non-negative matrix factorization) Purwins et al. [18] found that deep learning models outperformed these approaches, provided that sufficient training data were available.

This turn toward deep learning brought substantial gains in classification accuracy. Momynkulov et al. [6] proposed a system for the fast detection and recognition of dangerous sounds in urban environments, advocating audio recordings over video on the grounds that acoustic analysis requires cheaper equipment, lower network demands, and less computation while still supporting real-time monitoring. Durães et al. [7] evaluated deep learning models for violence detection in audio using mel-spectrograms of speech data as input, demonstrating that simple models perform very well, achieving classification accuracy close to that of far larger models, with the added benefit of applicability to human–machine monitoring and interaction systems. Bakhshi et al. [19] corroborated this, showing that the best simple model achieved 8% higher classification accuracy than the previous state-of-the-art model on the same dataset, underscoring the feasibility of detecting violence from audio alone when resources are limited.

A persistent limitation of CNN-based approaches is their difficulty in modeling the temporal relations in sequential data that are valuable for extracting information from audio [20]. To address this, Omarov and Altayeva [2] combined a one-dimensional CNN (1D-CNN) for spatial feature extraction with a Bidirectional Long Short Term Memory (BiLSTM) network for temporal modeling. Their system recognizes eight categories of dangerous sound (gunshot, explosion, screaming, crying, glass breaking, fire, emergency alarms, and weapons) with high accuracy and strong generalization across similar sound classes. Zhu-Zhou et al. [21] approached the same problem through transfer learning, showing that pretrained networks such as YAMNet capture distinct patterns that transfer effectively to violence detection, saving computational resources and training time while maintaining classification accuracy. Convolutional models have also been specialized to individual danger classes. Doungpaisan and Khunarsa [20] conducted a comparative study of CNN architectures and time–frequency representations for gunshot classification and for emergency vehicle sirens, both Gowri Shankar et al. [22] and Tran and Tsai [23] tackled siren sound classification with CNN-based architectures [24,25]. Tran and Tsai, in particular, proposed SirenNet, a two-stream CNN combining WaveNet for raw waveform processing through a 1D-CNN with MLNet for spectrogram processing through a 2D-CNN. They identified key limitations of prior work, like small, country-specific datasets, reliance on handcrafted features, inadequate evaluation of environmental noise effects, and a lack of raw waveform processing and addressed them substantially through experiments on UrbanSound8K and ESC-50, the same datasets adopted in the present work.

Danger and violence have also been addressed as multimodal problems drawing on video and audio/visual cues [26,27]. Rendón-Segador et al. [9] proposed ViolenceNet, which couples a three dimensional DenseNet with a Multi Head Self Attention layer and a bidirectional convolutional LSTM to encode spatio-temporal features and classify a video as violent or not. Wei et al. [10] targeted campus surveillance with a Spatio-Temporal Fusion–Temporal Difference Network for detecting multiple categories of abnormal behavior under a pronounced long-tailed class distribution. Bridging the two modalities, Lou et al. [11] fused auditory and visual information through an autoencoder-mapping model that projects both streams into a shared semantic subspace using a CNN-LSTM feature extractor, reporting that the multimodal method outperforms single modality baselines by exploiting the complementarity between channels.

Beyond isolated acoustic events classification, more recent work has investigated the detection of compound incidents involving multiple events and richer semantic interpretation. Saradopoulos et al. [15] proposed a real-time system for critical incidents in smart cities, using a network of edge microphones (ESP32 platforms) that transmit compressed audio to Raspberry Pi 5 devices hosting an Audio Spectrogram Transformer trained on AudioSet. The system exports detected events as JSON and applies a two stage rule in which a primary sound category (e.g., a vehicle sound) must be followed by a secondary category (e.g., glass breaking or a collision) within a defined time window before an incident is logged, achieving precision 0.882, recall 1.0, and an F1-score of 0.89 on a 1 hour and 40 minute recording.

The most semantically rich approach examined is that of Potamitis [14], who combined sound event detection, speech recognition, sentiment analysis, and Large Language Model (LLM) reasoning to convert an audio stream into interpretable life threatening risk estimates expressed in natural language, producing a binary risk decision every two seconds together with a short textual justification. This work is conceptually the closest to ours in its use of text as an intermediate representation for reasoning about danger. The key difference is that Potamitis maps audio directly to words through zero-shot models, whereas our approach generates full natural language descriptions with CoNeTTE and then performs danger classification through a fine-tuned DistilBERT. In parallel, a line of research applies captioning to the visual domain [28,29]. Cheoi et al. [8] proposed a surveillance system that fine-tunes BLIP-2 on a custom dataset to caption surveillance video in terms of the relations among objects, actions and spatial elements, then uses BERT [30] to interpret the generated captions and derive a risk score on a scale from 1 to 7. Although this system targets visual rather than acoustic data, its architecture directly mirrors and is the principal inspiration for the audio pipeline proposed in the current work. A related unsupervised direction is explored by Ogura et al. [31], who paired unsupervised anomalous sound detection with difference captioning. Using CLAP as the base model, the system captions both normal operation and faulty samples and then employs GPT-4 to compare the two captions and produce a caption explaining their differences, yielding interpretable descriptions of detected machine faults, evaluated on a dataset derived from the DCASE 2020 challenge, without requiring language training data for fault sounds.

A related line of work pursues resource conscious classification, in which an inexpensive first stage resolves clear cases and a more capable model is reserved for ambiguous ones. Zhu-Zhou et al. [16] proposed a cascade strategy for acoustic violence detection in computationally constrained scenarios, where a lightweight linear detector screens incoming segments and forwards only the uncertain ones to a more expensive deep classifier. Their forwarding threshold is not fixed in advance but derived empirically from the distribution of training scores, keeping computation low while preserving detection accuracy.

The most comprehensive overview of the field is provided by Figueroa-Flores et al. [3], who analyzed twenty studies on artificial intelligence techniques for violence detection in audio. Their review identifies CNNs as the dominant architecture and mel-spectrograms and MFCCs as the dominant feature representations, with transfer learning combined with data augmentation as the key strategies for adapting pretrained models to danger detection. It further documents emerging trends, like LSTM-based temporal modeling (25% of studies), transformer-based approaches (25%) and multimodal fusion strategies (30%), while noting that no single solution fits all settings and that the standardization of datasets, evaluation metrics and core practices remains an open problem. These observations motivate the interpretable, fusion based design we present, which pairs a lightweight acoustic classifier with a semantic classifier operating over natural language descriptions of the acoustic scene.

In summary, while the literature demonstrates significant progress in deep-learning-based danger detection, a clear tension remains between computational efficiency and semantic interpretability. Lightweight acoustic classifiers are well-suited for real-time edge deployment but often suffer from high false-alarm rates due to their inability to reason about context. Conversely, semantically rich approaches that leverage natural language or multimodal fusion offer superior contextual understanding but incur a high computational cost that limits practical, resource-constrained implementation. The framework proposed in this study directly bridges this gap by introducing an uncertainty-driven sequential pipeline. By using a fast, raw-waveform acoustic classifier to resolve clear-cut cases and selectively invoking a semantic captioning and analysis branch only for ambiguous signals, we achieve both high interpretive depth and computational viability for practical surveillance systems.

3. Materials and Methods

The proposed approach follows a rationale similar to the caption-based systems reviewed above, but exploits both acoustic and linguistic classification [32,33,34], aiming not merely at recognizing a dangerous event as an isolated occurrence but at semantically analyzing the entire acoustic scene in which that event occurs. The system is organized as two parallel decision flows over the same audio input, combined at the end by a rule based fusion stage. In the acoustic flow, a convolutional neural network (CNN) [35] maps the audio directly to a danger probability. In the semantic flow, a pretrained captioning model [36], CoNeTTE, converts each clip into a natural language caption, and a fine-tuned DistilBERT model maps that caption to a danger probability. Constructing these two flows involves six preparatory steps: (i) building a training dataset from three established audio-classification datasets (ESC-50, UrbanSound8K, and AudioSet); (ii) training the acoustic CNN classifier; (iii) generating captions with CoNeTTE, (iv) labeling them with a lexicon-based SBERT danger scorer to create the caption dataset; and (v) fine-tuning DistilBERT on that labeled dataset. Finally, (vi) the outputs of the acoustic and semantic classifiers are combined through a set of decision fusion rules, from which the best performing configuration is selected as the final system.

3.1. Audio Dataset

The training dataset was constructed following the danger class selection strategy of Omarov and Altayeva [2], who assembled a dangerous sound dataset by selecting hazard related classes from ESC-50 [37], and extended here with additional classes drawn from other corpora. Eight danger classes were taken from ESC-50 (siren, chainsaw, glass breaking, coughing, sneezing, crying baby, car horn, and dog) and augmented with two indoor relevant classes: gun_shot from UrbanSound8K [38] and screaming from AudioSet [39]. This yields ten danger classes. All remaining classes are mapped to the non-danger category, framing the task as binary classification between danger and no-danger sounds. Because ESC-50 and UrbanSound8K both draw recordings from Freesound, an overlap check based on source metadata and audio signal comparison was performed before merging, removing 240 duplicate recordings from UrbanSound8K (99 car_horn, 91 dog_bark, 50 siren) to avoid performance overestimation. To preserve balance with the 40 sample per class structure of ESC-50, 40 gun_shot clips were randomly drawn from UrbanSound8K. The AudioSet screaming class yielded only 18 usable clips, owing to its loose labeling and the frequent unavailability of the source YouTube videos. The retrieved 10-second AudioSet clips were split into two 5-second halves, the higher energy half was retained, and all screaming clips were manually verified by listening, prioritizing quality over quantity. The final training set (Table 1) contains 2058 clips, 378 danger (18.4%) and 1680 no-danger (81.6%), an imbalance that reflects the rarity of genuine danger events in real indoor monitoring. After overlap removal and exclusion of the 40 gun_shot clips used in training, the remaining 8419 UrbanSound8K clips (with gun_shot, siren, car_horn, and dog_bark treated as danger, giving 2419 danger and 6000 no-danger) were held out as an independent test set for both classifiers.

The ten danger classes were selected by combining three sources of evidence: the auditory danger-signal framework of ISO 7731:2003 [1], prior work on hazardous sound detection [2], and a small majority-vote labeling procedure following the panel based ground truth strategy of Dimoulas et al [40]. Five listeners independently judged whether each candidate sound should be treated as a potential danger, and a class was retained as danger when at least three of the five agreed, prioritizing recall of genuinely hazardous events over the exclusion of ambiguous ones.

3.2. Audio Classifier

The acoustic classifier builds on the one-dimensional CNN architectures [41] proposed by Vrysis et al. [42] and Stamatiadou et al. [43], which apply successive Convolutional–Pooling–Dropout (CPD) blocks to audio event classification [44]. The classifier operates directly on the raw waveform [45,46,47,48]. Each 5 second clip, sampled at 22.05 kHz (110,250 samples), passes through three CPD blocks with 32, 64, and 96 filters respectively (kernel size 8), followed by global average pooling and two fully connected layers, with ReLU activations throughout, 25% dropout, and a sigmoid output for binary classification (~72k trainable parameters). Four variants of this raw waveform model were compared, differing only in the stride/pooling configuration, which trades temporal detail against output compactness: LV-18 (stride 1, pooling 8) preserves the most detail, LV-24 (stride 2, pooling 4) balances detail and reduction, LV-42 (stride 4, pooling 2) favors feature compression, and LV-81 (stride 8, pooling 1) minimizes the representation for speed. Figure 2 illustrates this structure for the stride 1 / pooling 8 configuration. Because stride × pooling equals 8 in every variant, all four share the same between-block output shapes and differ only in the internal placement of the down sampling.

Raw waveform input was adopted deliberately. As a conventional alternative, a feature-based CNN operating on a 32-dimensional descriptor per clip (13 MFCCs, 12 Chroma, and 7 Spectral Contrast coefficients, time-averaged over the clip and standardized using training-set statistics) was also evaluated, and it proved competitive. However, such a model performs a comparatively heavy, high dimensional analysis, stacking many handcrafted features beneath successive convolutional layers, whereas raw waveform 1D-CNNs learn directly from the signal, dispense with a separate feature extraction stage, and are lighter and better suited to the real-time, on device operation that danger detection systems require [6,16]. The study therefore centers on the raw waveform models and retains the feature based CNN only as a comparison point.

3.2.1. Training Procedure

The principal challenge in training is the class imbalance [49], with danger clips forming only 18.4% of the data. Two cost function strategies were compared. Focal loss [50,51] with α = 0.75 and γ = 2, a modification of binary cross-entropy that downweighs easily classified examples and focuses learning on hard ones (α raising the weight of the positive danger class, γ controlling the decay), and weighted binary cross entropy with fixed class weights of 4.18 for danger and 1.0 for no-danger, derived from the training set class ratio. All models used the Adam optimizer and a batch size of 32, and were trained in Google Colab on an NVIDIA T4 GPU. The dataset was split 60%/20%/20% into training, validation, and test subsets using stratified sampling on the original source class, and this split was held fixed across all runs to allow direct comparison. To counter the scarcity of the screaming class, data augmentation (noise addition, time shifting, pitch shifting, and time-stretching) was applied to the training subset only [52]. Rather than a fixed number of epochs, two checkpoints were retained per run, the model with the lowest validation loss and the one with the highest validation AUC. The four raw waveform architectures, together with the feature-based CNN retained for comparison, were each trained under both loss functions, yielding ten trained models and twenty checkpoints in total.

A complementary experiment used 5-fold cross validation to assess generalization more thoroughly, with each ESC-50 sample retaining its official fold assignment and the gun_shot and screaming clips distributed proportionally across folds (8 and 4/4/4/3/3 respectively). In this setting, proportional undersampling of the majority class was applied at a 1:1.25 danger to no danger ratio, 240 danger and 300 no-danger clips per run (540 total), retaining roughly 7–8 samples from each of the 42 non-danger categories, while screaming was augmented to 24 samples within the training folds only. Using focal loss (α = 0.75, γ = 2), each architecture was trained on every fold, giving 25 runs and 50 checkpoints, with final performance reported as the mean and standard deviation across folds.

3.3. Semantic Classifier

The semantic branch converts each clip into a natural language caption and then reasons about danger over that caption. It comprises three steps: caption generation with CoNeTTE, rule based semantic danger scoring that also labels the caption dataset, and fine-tuning of a DistilBERT classifier on the labeled captions.

3.3.1. Caption Generation with CoNeTTE

Once acoustic classification was complete, the pretrained CoNeTTE model [53] was applied to all 2058 clips to generate the natural language captions that form the caption dataset. CoNeTTE was used with no fine-tuning or retraining, following similar caption based approaches in the literature [14,31]. It was chosen for several reasons. Its demonstrated ability to generalize to unseen datasets, aided by task embedding inputs that indicate the source dataset, its training exclusively on environmental and everyday acoustic scenes [54] rather than speech or music, matching this work’s domain and its treatment of speech, which it reports as the presence of speech rather than transcribing its content. Captions were generated in the Clotho configuration [55] with a length of 3–20 words, and manual inspection confirmed that Clotho produced more descriptive captions, richer in content and in the spatio-temporal relationships of events than the AudioCaps configuration (for example, Clotho rendered a siren clip as “a police car with a siren is blaring and a vehicle is driving by”, whereas AudioCaps produced the terser “a police car siren is blaring”) [56], consistent with the original CoNeTTE report [57]. A qualitative comparison of the two configurations is provided in Table 2.

3.3.2. Semantic Danger Scoring

Interposing CoNeTTE raises two needs. Assessing how faithfully its captions reflect the original audio class and exploiting the semantic cues in language that the acoustic model may miss (for instance, that a distant siren, a single dog bark, or gunshots from a television do not indicate real danger). This stage therefore has two goals: to label the caption dataset for DistilBERT fine-tuning, and to provide a first rule based semantic classification of danger. Scoring uses the sentence transformers library with the all-MiniLM-L6-v2 model, a Sentence-BERT variant [58,59]. Rather than exact matching, which cannot capture paraphrase, each caption and each lexicon keyword is encoded into a normalized embedding and compared by cosine similarity, a caption being taken to match a keyword when their similarity exceeds a preset threshold:

Similarity = cosine_similarity(E(caption), E(lexicon_term_i)),
where E is the embedding.

(1)

A manually constructed danger lexicon, stored as JSON, defines four scoring categories. The first assigns a base danger weight to each of the ten danger classes, set empirically by severity in domestic settings following analogous work [8]: dog and sneezing score 1; car horn and coughing score 2; crying baby, siren, screaming, and chainsaw score 3; and glass breaking and gunshot score 4. The hierarchical assignment of these base weights is grounded in a heuristic prioritization of threat level, physical safety, and urgency within indoor and residential environments. Acute, unambiguous indicators of physical violence or structural compromise, such as gunshots or glass breaking, are assigned the maximum weight of 4. Conversely, common physiological or environmental sounds that frequently occur in benign contexts, such as sneezing or a dog barking, are assigned the minimum weight of 1. These lower weighted events only trigger a danger classification if they are accompanied by strong intensity or proximity modifiers (e.g., ‘loudly’ or ‘in the foreground’). This weighted scaling ensures that the semantic scorer remains highly sensitive to immediate emergencies while successfully suppressing false alarms from routine household activity.

The second category captures intensity and duration: terms such as “loud”, “repeatedly”, “continuously”, and “rapidly” add one point, while neutral terms such as “once”, “passing by”, and “steady pace” leave the score unchanged. The third category holds context and distance modifiers: proximity terms (“near”, “foreground”, “close”) raise the score by up to two points, whereas distance terms (“far”, “background”, “in the distance”) lower it by up to one. The fourth category applies a −2 penalty when the caption indicates playback through a television, radio, recording, or other device, preventing non environmental sounds from being misclassified as dangerous. The contribution of each matched term is its similarity times its lexicon weight:

Contribution_i = Similarity_i × LexiconWeight_i.

(2)

The total is the sum of an event score (from the danger class matches), a modifier score (from intensity, duration, and context, clipped to [−3, +3] to avoid extremes), and a media score:

DangerScore = EventScore + ModifierScore + MediaScore.

(3)

with a caption labeled potential danger when DangerScore > 0. Three threshold profiles were defined, low (class 0.35, event 0.38, modifier 0.35), medium (0.40, 0.42, 0.38), and strict (0.45, 0.48, 0.42), producing three candidate datasets. In parallel, a second JSON of keywords covering all 50 ESC-50 classes, the 10 UrbanSound8K classes, and the AudioSet screaming class is used to compute a class alignment score, identifying the most semantically similar true class for each caption and quantifying how well the caption matches the clip’s real label, an added quality indicator that flags classes where captioning is least reliable. Each record in the resulting dataset stores the filename, original class, caption, binary danger label, total danger score, best aligned class with its similarity, and the alignment score. The resulting dataset manifests are publicly available (see Data Availability Statement). The low threshold dataset was selected for DistilBERT fine-tuning, with the justification given in the Results. Table 3 shows representative scorer outputs.

3.3.3. Fine- Tuning and Training DistilBERT

The labeled caption dataset was used to fine-tune a DistilBERT model for binary classification. It takes a CoNeTTE caption as input and outputs a danger / no-danger prediction, forming the semantic classifier. The pretrained distilbert-base-uncased model [60] was imported from Hugging Face and adapted for sequence classification by replacing its output layer with a two class head. Six experiments were run, crossing two independent factors, the data split strategy and the training loss. Two split strategies were compared, both at 60%/20%/20%. The first, a random split, stratifies only on the danger label so that each subset mirrors the overall class distribution. The second, an alignment stratified split, stratifies jointly on the danger label and the alignment score of Section 3.3.2 (value 1 for a caption that matches its true class, 0 otherwise), ensuring that captions of differing quality are distributed proportionally across subsets rather than, for example, evaluating the model only on well described samples. For each split, three loss functions were compared: standard cross-entropy (equal class weights, simple but prone to majority bias), weighted cross-entropy (per-class weights from the training distribution, directly offsetting imbalance) and focal loss (γ = 2, additionally downweighing easy examples to focus on ambiguous captions). All experiments shared a learning rate of 2×10⁻⁵, a training batch size of 8 and evaluation batch size of 16, and a maximum of 8 epochs, selecting each epoch’s checkpoint by danger recall on the validation set, prioritizing recall, since the system’s goal is to minimize missed danger events even at the cost of additional false positives. The best performing of the six models was carried forward to the fusion stage.

3.4. Decision Fusion

The final stage combines the outputs of the acoustic CNN classifier and the semantic DistilBERT classifier. Twelve fusion rules were evaluated on the held out UrbanSound8K test set (after overlap removal and re-inclusion of the 40 gun_shot clips used during training). Table 4 defines the rules. Two act as baselines, cnn_only and distilbert_only, which use a single model’s probability; the remaining ten combine the two. Two evidential rules apply Dempster–Shafer combination [61,62].

Following the cascade approach of Zhu-Zhou et al. [16], the fusion thresholds are not fixed in advance but derived empirically from the model’s own score distribution. Specifically, the threshold-dependent rules use two data driven thresholds derived from the medians of the averaged false-negative and false-positive probabilities on UrbanSound8K. During model comparison these were computed per loss family (0.35–0.60 for focal loss models, 0.27–0.66 for weighted cross-entropy models) to rank candidate configurations on a common basis, with the near boundary rescue zone spanning 0.35–0.50 and 0.27–0.50 respectively and the cascade rules using the corresponding low threshold. Once the final model was selected, the thresholds were recomputed on that model’s own probability distribution, giving an activation zone of 0.17–0.60 for the deployed system, calibrating the hand-off to the exact classifier in use rather than to a family average. For the CNNs trained under the 5-fold undersampling scheme, only the custom_uncertain rule was applied, with an activation zone of 0.43–0.59 set from the mean median across all folds and models on UrbanSound8K.

Two fusion topologies were implemented (Figure 3): a parallel scheme, in which both classifiers run on every clip and their outputs are combined, and an uncertainty correction scheme, in which the semantic classifier is invoked only when the CNN is uncertain, the latter offering a computational advantage, since the semantic branch need not run when the acoustic model is already confident. The best performing configuration, the LV-18 weighted-cross-entropy acoustic classifier combined with the stratified weighted-cross-entropy DistilBERT classifier through the custom_uncertain rule, was selected as the final system and carried forward to external evaluation. The acoustic classifier trained under 5-fold cross-validation with undersampling was retained as a complementary configuration for assessing robustness.

4. Experimental Results

4.1. Acoustic Classifier

Model selection prioritized recall of the danger class, so as to miss as few genuine danger events as possible. On the internal test set (412 clips), the LV-18 model trained with weighted cross-entropy achieved the highest danger class recall (0.86–0.87 across the two checkpoints) while maintaining a competitive danger class F1 (0.64) (Table 5). This choice was confirmed on the full, held out UrbanSound8K set (8419 clips), where LV-18 with weighted cross-entropy and the validation AUC checkpoint produced the highest danger class recall of any model (0.70), at 81.8% overall accuracy (Table 6).

The feature-based CNN, retained as a conventional comparison point, proved competitive but did not surpass the raw waveform models on the selection criterion. Its best danger class recall on UrbanSound8K reached 0.64, below the 0.70 of the selected LV-18 model, and it offered no compensating advantage in precision or macro F1. Given that it also performs a heavier, high dimensional feature analysis less suited to real-time deployment, the raw waveform LV-18 configuration with weighted cross-entropy was carried forward as the acoustic classifier.

4.2. Semantic Danger Scoring

The lexicon based SBERT scorer was evaluated across the three threshold profiles on the full 2058 clip caption dataset (Table 6). The low threshold (0.35) maximized danger class recall at 0.92, at the cost of precision (0.52), for a danger class F1 of 0.66; the medium and strict profiles traded recall for precision (recall 0.86 and 0.62, precision 0.69 and 0.87). Consistent with the recall first objective, the scorer’s role being to label captions without discarding genuine danger, the low threshold dataset was selected for DistilBERT fine-tuning. A per class fidelity check, using the class alignment keywords to compare each caption against its true label, found all original classes correctly aligned except sneezing, which the scorer associated with coughing, reflecting the acoustic proximity of the two.

Table 7. Sentence-BERT danger-scoring results for the three threshold profiles (2058 clips). Selected profile highlighted.

Threshold	TN	FP	FN	TP	Accuracy	Danger Prec.	Danger Rec.	Danger F1	Macro-F1
Low (0.35)	1352	328	29	349	82.65%	0.52	0.92	0.66	0.77
Medium (0.40)	1532	148	53	325	90.23%	0.69	0.86	0.76	0.85
Strict (0.45)	1644	36	144	234	91.25%	0.87	0.62	0.72	0.83

Table 7. Sentence-BERT danger-scoring results for the three threshold profiles (2058 clips). Selected profile highlighted.

Threshold	TN	FP	FN	TP	Accuracy	Danger Prec.	Danger Rec.	Danger F1	Macro-F1
Low (0.35)	1352	328	29	349	82.65%	0.52	0.92	0.66	0.77
Medium (0.40)	1532	148	53	325	90.23%	0.69	0.86	0.76	0.85
Strict (0.45)	1644	36	144	234	91.25%	0.87	0.62	0.72	0.83

4.3. Semantic Classifier

Across the six DistilBERT configurations, all models performed strongly, reaching danger class recall of 0.92–0.94 on the caption test set with balanced precision (Table 8), and on the UrbanSound8K captions producing roughly one false alarm for every sixteen correctly detected danger sounds. The configuration carried into the final system, DistilBERT fine-tuned on the alignment stratified split with weighted cross-entropy, combined high danger recall with strong precision, making it well suited to correcting the acoustic classifier within the fusion stage.

4.4. Decision Fusion

The twelve fusion rules were evaluated on UrbanSound8K for every acoustic-semantic model pairing. The final system combines the LV-18 weighted cross-entropy acoustic classifier with the stratified weighted cross-entropy DistilBERT classifier through the custom_uncertain rule, which forwards a clip to the semantic classifier only when the acoustic probability falls within the uncertainty zone and otherwise accepts the acoustic decision. For this deployed model, the uncertainty zone was calibrated to the selected classifier’s own probability distribution on UrbanSound8K, giving an activation range of 0.17–0.60 (Section 3.4).

This configuration raised danger class recall to 0.82 (precision 0.56, F1 0.67), a substantial improvement over the standalone acoustic classifier (recall 0.70, precision 0.45, F1 0.55) and approaching the standalone semantic classifier (recall 0.86, precision 0.63, F1 0.73), while activating the semantic branch for only 3276 of 8419 clips (≈39%) (Table 9). In confusion matrix terms, fusion recovered 1994 of the 2419 danger clips (TP 1994, FN 425) against 1572 false alarms, reducing missed danger from the acoustic classifier’s 727 to 425 at a moderate cost in additional false positives. This is the central empirical result of the system: the semantic classifier, invoked only in the acoustic model’s uncertainty zone, improves danger recall while roughly 61% of clips are still resolved by the fast acoustic classifier alone.

A complementary experiment applied the same rule to an acoustic classifier trained under 5-fold cross-validation with majority class undersampling, paired with a DistilBERT classifier, to assess robustness under a different training regime (Table 10). Across folds it reached a mean danger class recall of 0.84 (± 6.7%), precision 0.52, F1 0.64, and macro-F1 0.70, with per-fold recall ranging from 0.74 to 0.91. The best balanced fold achieved recall 0.83 at 61% precision, roughly one false alarm per 2.5 correctly detected danger sounds. These results confirm that the recall gains of the uncertainty correction rule are stable across training regimes, although the fixed split configuration above is the one carried forward to external evaluation.

4.5. Generalization to External Datasets

The final system was evaluated on two external datasets not seen during training, following the false alarm focused protocol of Potamitis [14] (Table 11). Both were adjusted to match this work’s danger definition. From DESED, the dog class was excluded [63], since dog is treated as a potential danger class here and its detection would not constitute a false alarm. From XD-Violence, the non_violence class was excluded [64] to yield a pure danger set, taking into consideration that the non_violence class may contain various audio events that are not specified so evaluating the proposed model on them does not provide meaningful results. After these adjustments, the evaluation sets comprised 610 DESED clips (all no-danger) and 1383 XD-Violence clips (all danger). Because the external clips differ in length from the fixed 5-second training clips, all files were processed through 5-second windows with 50% overlap, and the window level predictions were aggregated into a clip level decision under a Multiple Instance Learning (MIL) scheme. Following the standard MIL formulation, aggregation used max pooling, so that a clip is labeled potential danger whenever at least one of its windows is, in keeping with the system’s recall-oriented objective.

DESED, containing only indoor non-danger sounds, measures false alarm behavior directly. The final fusion system correctly identified 449 of the 610 no-danger clips (no-danger recall 0.74, F1 0.85), misclassifying 161 as danger, a false-positive rate of 26%, roughly one false alarm for every three correct decisions, with the semantic branch triggered on 272 clips. As on the internal data, the standalone semantic classifier was the most conservative on this set (recall 0.88, only 74 false alarms) and the standalone acoustic classifier the least (recall 0.69), with fusion between the two. This ordering reflects the complementary roles of the two classifiers, the semantic branch tempering the acoustic model’s tendency to over flag.

On XD-Violence, whose danger content is defined largely by its visual channel, the audio only system reached a danger class recall of 0.69, detecting 952 of the 1383 danger clips (danger class F1 0.82), with the semantic branch triggered on 112 clips. Rather than a failure of the method, this quantifies a known boundary. For danger that is primarily visual, the acoustic modality alone is insufficient, confirming that the proposed system is best positioned as a complement to, not a replacement for, audio-visual approaches. A qualitative review of the false negatives in the XD-Violence dataset reveals two primary causes for the missed detections. First, in several clips the danger is entirely visual, accompanied only by low-level ambient noise or silent movement that lacks any distinct acoustic signature (e.g., physical confrontations, thefts). Second, many clips from cinematic sources contain background music or synthetic sound effects that mask genuine acoustic danger cues (e.g., screaming or impacts), causing both the CNN and CoNeTTE to misinterpret the scene. This highlights the inherent physical boundaries of audio-only monitoring in highly noisy or media-rich environments.

4.6. Answers to the Research Questions

With respect to the two research questions posed earlier for RQ1 the results indicate that audio alone is a viable and practical basis for danger detection. Sound is almost always present, regardless of lighting, line of sight, or occlusion, so an event a camera would miss can still be heard, and it is far less resource intensive than video, requiring less data to capture, process and transmit, which suits real-time and on-device monitoring. Empirically, the system detected danger from audio with a danger class recall of 0.82 and F1 of 0.67 on UrbanSound8K, and on realistic indoor no-danger audio (DESED) its false alarm rate was acceptable. Its low recall on XD-Violence, a dataset whose danger is defined largely by the visual channel, marks the boundary of the modality rather than a failure of the method. It is specifically a visual danger dataset for which audio only is insufficient. Audio is therefore good practice for hazards with an acoustic signature, particularly indoor and residential monitoring, and is best positioned as a complement to visual approaches where danger is purely visual. For RQ2, captioning as an intermediate semantic stage is useful and improves detection. The semantic branch helps recognize danger indicative acoustic events without missing genuine danger. It raised danger class recall over the standalone acoustic classifier (0.70 → 0.82), though it does not by itself improve precision. The custom_uncertain rule strikes a favorable balance between recall and false alarm rate while confining the costly semantic branch to roughly 39% of clips. Beyond accuracy, the intermediate caption makes each decision interpretable. Meaning that the system emits a human readable description alongside its prediction, an implicit explanation that conventional black-box classifiers lack.

Further research is nonetheless needed, above all on data. The system was validated on a dataset repurposed from general sound classification rather than one built for danger detection. Thus, a dedicated, appropriately designed danger dataset is required to establish the approach fully. Following the incremental repository building strategy of Kotsakis and Dimoulas [65], the captioning pipeline itself can be used to progressively accumulate such a dataset, on which a dedicated model can then be trained.

5. Discussion

The proposed system was built and evaluated as two independent, parallel flows, an acoustic CNN classifier and a semantic branch (CoNeTTE captioning followed by DistilBERT), but the selected uncertainty correction rule (custom_uncertain) defines a sequential runtime for the deployed system. The full CoNeTTE–DistilBERT path is activated only when the CNN’s danger probability falls within its uncertainty zone, while clips classified with high confidence as danger or no-danger are decided by the acoustic classifier alone. Examined separately, the two classifiers play complementary roles. The acoustic CNN is fast and has comparatively high danger recall, but produces a large number of false alarms, whereas the semantic classifier is the more precise of the two. Fusing them through the uncertainty correction rule improves the recall of the potential danger class relative to the standalone CNN while keeping false alarms in check, and it does so while invoking the semantic branch for only roughly 39% of the inputs.

The choice of a sequential activation rule was driven by the requirements of a practical system to classify danger quickly and as reliably as possible, missing as few danger events as possible while producing as few false alarms as possible. This dual objective follows the emphasis of Momynkulov et al. [6] on rapid danger detection and of Saradopoulos et al. [15] on reducing false alarms for a usable real-world system, and it is oriented toward the fast, accurate detection of potential dangers in indoor settings such as homes, workplaces, and public buildings. The computational profile of the system supports this orientation. The acoustic CNN stage has very low computational requirements, and is the lightweight component of the pipeline, consistent with the efficiency reported for small acoustic CNNs in the literature [66]. The heaviest stage is caption generation with CoNeTTE, which pairs a ConvNeXt encoder [67] with an autoregressive Transformer decoder [68]. Although CoNeTTE attains competitive captioning performance with four to forty times fewer parameters than earlier captioning models [53], its autoregressive decoding still makes it the most demanding component when activated. The fine-tuned DistilBERT classifier is comparatively lightweight [60]. Overall, the pipeline is designed to run on commodity hardware without specialized accelerators, since the heaviest component is invoked only selectively. It is precisely the sequential activation of the semantic branch that keeps this profile practical, since the heaviest stage does not run on the clips the acoustic model already resolves with confidence. Whereas the continuous, real-time systems of Saradopoulos et al. [15] and Potamitis [14] prioritize low response latency at the expense of semantic depth, the present approach uses audio captioning together with an adapted DistilBERT text classifier to achieve a richer semantic understanding of the acoustic environment and greater specialization in predicting dangerous events.

6. Conclusions and Future Work

This work introduced a methodology for the automatic prediction of potential danger from audio that departs from the dominant approach of direct acoustic classification by incorporating natural language as an intermediate semantic representation. Rather than mapping an audio signal directly onto a binary danger label, the proposed system generates a natural language description of the acoustic scene through the pretrained CoNeTTE captioning model and then applies NLP based semantic analysis, via a fine-tuned DistilBERT classifier, to estimate the level of danger conveyed by that description. Its central contribution is the integration of automated audio captioning into a danger detection pipeline, which addresses the interpretability gap that characterizes conventional deep learning classifiers. Because these models emit a danger prediction without an understandable justification, trust in their decisions is limited. By using natural language as an intermediate stage, the system provides an implicit explanation of each prediction in the form of a descriptive caption, making the decision process more transparent and auditable for human users. Empirically, combining the acoustic and semantic classifiers through the uncertainty correction rule improved the recall of the danger class over the standalone acoustic classifier while keeping the additional cost of the semantic branch confined to the minority of clips that require it. A natural next step is to apply this pipeline iteratively. The present system was validated on a repurposed dataset not originally designed for danger detection, and the same captioning-based process can be used to progressively accumulate a large, task-specific danger dataset. Once such a dataset exists, a dedicated model can be trained on it, following the incremental repository building strategy of Kotsakis and Dimoulas [65], in which a modest initial dataset is iteratively augmented and later reserved for more sophisticated models. This makes the proposed pipeline a recommended route to the danger detection problem once appropriately designed data are available.

The implementation is a step toward interpretable, semantically enriched danger detection systems and opens several directions for future work. The most significant limitation evident in the literature is the gap between training conditions and real deployment conditions, which stems from the nature of the available datasets. Most studies rely on simulated, staged, or acoustically clean data that fail to capture the complexity of genuine dangerous incidents [3], so models trained on them are likely to struggle to generalize to the noisy, unpredictable environments in which they are most needed. Future efforts should prioritize the collection of more realistic and diverse datasets, spanning heterogeneous recording conditions, varied demographics, and a broader range of dangerous acoustic events, while the standardization of danger and violence definitions across studies, together with systematic reporting of architectures, hyperparameters, and evaluation metrics, remains an open challenge [3]. A particularly promising dataset structure, in the spirit of Ogura et al. [31], would place dangerous events in the positive class and their acoustically similar counterparts in the negative class. Such a dataset would benefit both classifier training and the construction of more targeted natural language datasets for captioning.

On the captioning side, the available datasets and evaluation metrics remain insufficient for systems that are genuinely useful in human–machine interaction. As Xu et al. [69] argue, the very notion of a useful audio description needs to be reconsidered. A description should not be limited to identifying individual sound events but should also convey the environmental context in which they occur and the temporal relationships among them. Descriptions of this kind would help close the interpretability gap in deep learning classification by allowing models to accompany their predictions with a linguistic justification, a capability whose absence has been a significant obstacle to trust in danger prediction applications, since existing systems produce labels that are not further exploited [14]. A further direction concerns the fusion stage itself. Rather than combining the outputs of the individual classifiers through hand designed rules, a machine learning based fusion model could be trained, given datasets containing real dangerous events and false alarms, to predict whether a particular combination of acoustic and textual outputs corresponds to genuine danger, so that decisions would rest on patterns learned from data rather than manually specified thresholds.

Finally, audio remains complementary to visual information. Audio-only danger detection systems such as the one proposed here can operate alongside more demanding multimodal systems that combine sound and image, enhancing the interpretability of their decisions and better supporting the understanding of language and human speech [14]. Because the ultimate goal of such models is deployment in real systems, research into their performance, computational requirements and into model design guided by those constraints is warranted, as is careful attention to user data. Feedback loops that collect data from users are valuable, but only if the protection of personal data is ensured first. More broadly, and in line with the most recent literature [5], the field of audio related tasks is trending toward unification and away from isolated, problem specific solutions. As with large language models, a shift toward unified audio models appears likely, and future implementations should be structured with that trajectory in mind.