Advancing Dolphin Acoustic Monitoring: A Comprehensive Whistle Classification Framework

1. Introduction

Dolphins have an exceptional sonar system to adapt to their living environment. In their sonar system, there are three major acoustic signal types [1,2]. The first one is clicks or tickings, which are high-frequency, narrow pulse signals used for underwater navigation, localization, foraging, and obstacle avoidance. They have remarkable adaptive and anti-jamming abilities. The second type is whistles, which are a communication signal. The fundamental frequency of the communication signals (whistles) is mainly ranged from 500 Hz to 24 kHz, and the signal length is usually between 0.1 and 2 seconds; for those in the pregnancy period, the signal length might be longer than 3 seconds. They serve various functions, including population communication, emotional expression, long-distance communication, and individual identification [3]. The third type is burst pulses (distress signals), which usually show up when dolphins are tense, such as when they are angry, fearful, frustrated, or under acute stress. These sounds can come off like rough barks, drawn-out howls, or rapid trills. They feature shorter pulse intervals and lower intensity compared to echolocation clicks. These signals are usually challenging to characterize, which makes the research on burst pulses difficult [4]. Hence, current studies on dolphin acoustic signals are mainly on echolocation and communication signals.

The Indo-Pacific bottlenose dolphin (Tursiops aduncus) is a medium-sized cetacean in the family Delphinidae. Compared to the common bottlenose dolphin, it is typically slimmer with more speckling on the belly. It usually inhabits warm coastal and shelf waters across the Indian Ocean and western Pacific [5]. It is brilliant and adaptable, it feeds on fish and cephalopods, and is known for cooperative hunting and frequent interactions with boats. In China, this species is identified as a nationally protected Class II animal. Extensive efforts have been devoted to studying their bioacoustic signals for detection and conservation.

Current public marine mammal acoustic datasets [6,7,8] are constrained by limited sample sizes, low signal-to-noise ratios, variable recording conditions, and ambiguous annotations. Consequently, models trained on these data often yield suboptimal performance in detection and recognition tasks. Furthermore, a systematic framework for training and evaluating different dolphin call detection models is lacking.

To address these issues, this study constructs a high-quality whistle signal dataset for the Tursiops aduncus. The acoustic signals are recorded in a quiet and controlled environment and subsequently annotated by human experts. We further develop a framework to comprehensively train and evaluate deep convolutional neural networks for robust whistle classification. Specifically, we explore the performance of various CNN architectures using different feature representations as input. To enhance model generalizability under complex, realistic conditions, we simulate underwater whistle sound propagation using Bellhop, a widely used underwater acoustic raytracing model and augment our dataset with in situ oceanic noise. The resulting synthetic signals are used for both training and evaluation, thereby improving the robustness of the classification models.

The remainder of the paper is organized as follows: Section II reviews research on marine-mammal sound recognition; Section III describes the collection and annotation of Tursiops aduncus signal records from Chimelong; Section IV reports experimental setup and classification results; and Section V concludes with a summary and directions for future work.

2. Literature Reviews

2.1. Dolphin Whistle Signal Types

The study of dolphin species classification and recognition algorithms typically begins with analyzing communication signal characteristics. Matthews et al. surveyed tonal calls across cetacean species, reporting summary statistics for acoustic parameters including start, end, minimum, maximum, and center frequencies, as well as call duration and the number of inflection points [9]. They also included information about the recordings, including location, encounter/group counts, and recording length. McCowan proposed a quantitative contour similarity based method to discriminate the whistle types [10]. Janick and Slater investigated the hypothesis that signature whistles serve to maintain group cohesion rather than being solely stress-induced [11]. The authors compared the whistle types of four captive bottlenose dolphins when they were together versus alone, observing whistle copying behavior in different experimental pools. They also studied individual differences among the dolphins based on this phenomenon. Beeman used a signal analysis system with fast Fourier transforms to analyze audio spectrograms and categorized communication signals into six types, which are ascending, fluctuating, sinusoidal, U-shaped, descending, and residual [12]. However, the production context of these signals and dolphin types remained unclear. Hawkins and Gartside conducted another study on classifying Indo-Pacific bottlenose dolphin whistles into five categories, including sinusoidal, ascending, descending, level, and concave [13]. They demonstrated that whistle types convey specific information to companions related to the caller’s behavior or situation (behavioral context), establishing a connection between whistle types and behaviors. Azevedo et al. categorized the communication signals of Atlantic spotted dolphins into six types, namely, ascending, descending, ascending-descending, descending-ascending, smooth, and multiple [14]. They measured nine acoustic parameters of each whistle’s fundamental components (e.g., starting frequency, ending frequency, and frequency at 1/4 duration) and concluded that dolphins alter whistle structures based on behavioral states, confirming the communication function of whistles. Rui-chao et al. also defined dolphin whistles into six categories, which are constant, upsweep, downsweep, concave, convex, and sine [15]. Dolphin whistles vary with frequency over time, and individuals can use "signature whistles" to communicate their identity. Harley demonstrated that dolphins can generalize from trained signature whistles to previously unheard whistle instances produced by the same individuals, indicating that whistle classification is driven by contour-based representations rather than by specific acoustic parameters or voice cues [16].

2.2. Acoustic Signal Classification

Early signal classification methods relied heavily on traditional statistical models to extract whistle segments, but a key limitation of these statistical models was the incomplete detection of whistle signals, resulting in compromised classification results. Douglas and Gillespie addressed this by aggregating statistics over many fragmented whistle signals, which yielded a classification accuracy of over 94% [17]. Researchers combined cepstral coefficients and the Gaussian mixture model to detect and recognize different bioacoustic signals, namely, background noise, communication signals, burst pulse signals, and combined signals, and identify species [18,19]. Yang Wuyi et al. proposed a method for classifying broad-snouted dolphin communication signals using syntactic patterns [20]. This method involves extracting the trajectory curves of the fundamental frequency of dolphin communication signals over time, followed by identifying the primitive sequences of fundamental frequency changes. Based on the categorization criteria for these signals, the grammar that generates the primitive sequences for each category is then summarized. With these established syntactic patterns, they can extract the primitive sequence features and classify them to achieve automatic classification of dolphin communication signals, advancing marine mammal acoustic research.

In recent years, Convolutional Neural Networks (CNNs), renowned for their success in computer vision, have also proven highly effective for audio processing tasks like audio tagging and sound event detection, demonstrating versatile applicability [21]. Gao et al. utilized deep neural networks to classify and identify echolocation signals and pulse noise from three typical marine mammal species [22]. Their results showed that the fully connected network surpassed the spectral energy algorithm, achieving a 30% higher accuracy. Zhang Yu et al. applied CNN for the classification of multi-species echolocation click signals [23]. To predict species labels, they applied majority voting and Maximum A Posteriori (MAP) methods across m consecutive clicks, finding that classification accuracy improved as m increased. While deep learning methods excel at human speech recognition when trained on large datasets, their application to dolphin vocalizations remains challenging due to the limited size of available samples.

2.3. Underwater Acoustic Propagation Modeling

To evaluate and enhance model robustness in complex marine environments, it is essential to understand underwater signal propagation and simulate the acoustic communication channel. Research on underwater acoustic propagation modeling theory began in the 1960s [24], initially focusing on ray theory and normal mode theory for horizontally invariant environments. To handle complex, horizontally varying acoustic propagation, parabolic equation (PE) theory and coupled normal mode theory emerged in the 1970s. Propagation models include ray models, normal mode models, parabolic equation models, multipath expansion models, fast field models, and hybrid algorithms [25]. Notably, BELLHOP [26], a widely-used ray model particularly suitable for shallow water and high-frequency scenarios, employs Gaussian beam tracing for computing ray paths and acoustic fields in horizontally non-uniform environments. RAY [27], another prominent ray model, evaluates the impact of seabed parameters such as compressional and shear wave speeds, attenuation, and density on broadband signal propagation, making it highly valuable in detailed acoustic environmental studies. KRAKEN [28], a frequently used normal mode model, efficiently computes underwater acoustic fields by solving eigenvalue problems under horizontally invariant or mildly varying conditions, ideal for low-frequency propagation modeling. PE methods effectively address horizontally varying scenarios, but their computational load increases significantly at higher frequencies or with complex seabed interactions, limiting their application. Consequently, underwater acoustic propagation models must be carefully selected based on specific scenarios, such as signal frequency, environmental complexity, and computational efficiency, to ensure accurate representation of underwater acoustic channels.

3. Methodology

In this section, we describe the data collection and annotation procedures, the deep learning classification methods, and the data simulation process.

3.1. Data Collection and Annotation

The raw data were collected at the dolphin aquarium of Chimelong Ocean Kingdom in Zhuhai, China. Two self-contained hydrophones (SoundTrap 300 HF) were used, each enclosed in a custom acrylic housing to prevent dolphins from accessing the devices. As shown in Figure 1, the cylindrical housing has two sections: the upper section holds adjustable suction-cup mounts for the hydrophone and an underwater camera, and the lower section contains lead weights to ensure stability underwater. The hydrophones were deployed at two locations in the pool, as shown in Figure 2: one at the center to capture signals from all directions, and another in a corner, positioned safely out of the dolphins’ reach. The hydrophones record signals at a sampling frequency of 144kHz from five Tursiops aduncus—two adults and three infants—in a single large pool.

The raw dataset contains approximately 72 hours of recordings. Since the signals were recorded in a large pool with low ambient noise, the whistle signals have relatively high signal-to-noise ratios. In this study, we manually annotated 19.5 hours of recordings following the category definitions of Xue Rui-chao et al. [15], with the addition of a new class termed double concave. The seven whistle types are defined as:

• Constant: nearly flat contour with frequency variation under 1 kHz across the time span of the signal.
• Upsweep: the fundamental frequency increases over time.
• Downsweep: the fundamental frequency decreases over time.
• Concave: the fundamental frequency first decreases, then increases over time.
• Convex: the fundamental frequency first increases, then decreases over time.
• Sine: sinusoidal-like contour.
• Double concave: two consecutive concave contours concatenated together.

All these seven whistle samples are visualized in Figure 3.

The annotated dataset includes the start and end times of each whistle in the raw recordings, along with its call type. In total, 3913 samples were labeled, and the dataset statistics are summarized in Table 1. The classes are imbalanced—for example, the convex type has 912 samples, whereas the constant type has only 256. Overall, the whistles span a frequency range of approximately 3–40 kHz.

3.2. Deep Learning Classification Methods

For whistle classification, we employed five widely used CNN network architectures [21], including MobileNet, Xception, ResNet, ResNeXt and SE-ResNeXt. The descriptions of the implemented models are detailed as follows.

• MobileNet [29]: a lightweight CNN that uses depthwise separable convolutions to reduce computation complexity while maintaining strong performance, which is suitable for mobile applications.
• Xception [30]: the “Extreme Inception” model that decouples spatial and channel-wise convolutions for more efficient feature extraction.
• ResNet (Residual Network) [31]: introduces residual connections (shortcuts) to enable effective training of very deep networks.
• ResNeXt [32]: extends ResNet with grouped convolutions and a cardinality parameter, capturing a broader range of feature interactions.
• SE-ResNeXt [33]: combines ResNeXt with Squeeze-and-Excitation blocks to model channel dependencies and enhance feature recalibration.

In the computer vision field, input data typically consists of three 2-dimensional channels (red, green, and blue). In contrast, our mono-channel acoustic recordings are 1-dimensional time-series signals, making it difficult to apply standard vision-based CNN architectures directly. To ensure compatibility with these CNN models, we explore two types of feature representations for the whistle signals, namely, the spectrogram-based representations and the waveform-based representations.

For the spectrogram-based representations, the audio signal is first transformed into either a 2D log-scaled Mel spectrogram (Log-Mel) or a 2D Mel-frequency cepstral coefficient (MFCC) sequence. Both features are widely used in speech and sound recognition because they reflect human auditory perception. MFCCs characterize the short-term spectral envelope of the signal, while Log-Mel spectrograms provide a detailed time–frequency representation. To match the 3-channel input format of common CNNs, we then compute the first- and second-order delta coefficients for each feature map. These deltas capture temporal changes in the features, allowing the 3-channel representation to encode both spectral structure and its evolution over time.

For the 1D waveform-based input, the CNN architectures are adapted to accommodate raw audio directly. Specifically, a multi-scale stacking module is introduced to learn a 2D representation for the downstream 2D-convolutional layers, as shown in Table 2. Unlike the Log-Mel or MFCC-based method, this module requires no explicit frequency-domain transformations. Instead, the convolutional blocks implicitly learn the spectral features, providing an end-to-end solution for the classification task.

We evaluate the models using mean Average Precision (mAP), a standard metric in information retrieval and object detection. It measures the average precision of a model across multiple classes. For each class, a precision–recall curve is computed, where precision (P) is defined as true positive (TP) divided by the sum of TP and false positive (FP) and recall (R) is the ratio of true positive detections to the total number of ground truth positives. The average precision (AP) is calculated as the area under the interpolated precision–recall curve, and mAP is obtained by averaging the AP values across all classes or queries.

3.3. Simulated Marine Whistle Signals

To further evaluate and enhance the robustness of the whistle classification models, we simulate whistle propagation through a marine acoustic channel and add in-situ ambient noise at controlled SNR levels for both training and testing.

Specifically, we use BELLHOP to compute the impulse response of the marine acoustic channel between the source and receiver. The simulated signals are then generated by convolving the whistle signal with this channel response:

$$r\left(t\right)=s\left(t\right)\ast h\left(t\right)$$

(1)

where r(t) is the received signal from the receiver in simulation, s(t) is the source signal, and h(t) is channel impulse response. To simplify the simulation model, it is assumed that the channel is linear and time invariant during the vocalization. It is worth noting that the underwater acoustic channel is complex, with multiple propagation paths generating many arrival pulses, most of which have very small amplitudes and minimal impact. For simplicity, pulses below 1% of the maximum amplitude are discarded. Additionally, because the impulse response is very weak, it is scaled by 100 to mimic hydrophone gain and ensure a signal intensity comparable to the original. Cross-correlation analysis is then used to determine the offset introduced by convolution, ensuring that the simulated whistle signal is centered within the segment.

Ambient noise is assumed to be additive and is directly added to the signal:

$$y\left(t\right)=x\left(t\right)+n\left(t\right)$$

(2)

where y(t) denotes the resultant noisy signal, x(t) denotes the clean whistle signal, and n(t) is a noise signal randomly sampled from the ocean noise recordings in the SHIPSEAR dataset [34].

4. Experiment and Results

In this section, we will explain the experiment setup and then present the results.

4.1. Input Data Pre-Processing and Experiment Set Up

The whistle signals are cropped into 0.75-second segments. For signals longer than 0.75 seconds, the central 0.75-second segment of the original recording is used. For shorter signals, the segment is extended by including additional portions from the original recording at both ends to reach the target length. As shown in Figure 4, this corresponds to roughly the 85th percentile, ensuring that most whistle contours are complete.

While the raw audio is recorded at 144 kHz, the fundamental frequencies of dolphin whistles lie primarily below 20 kHz. We therefore downsample the data to 44.1 kHz using the Kaiser algorithm. The downsampled signals are then transformed into Log-Mel and MFCC spectrograms, and their first-order (delta) and second-order (accelerate) coefficients are added as additional channels. This results in three input representations: Log-Mel + delta + accelerate, MFCC + delta + accelerate, and raw waveform data, as illustrated in Figure 5.

We use 64 Mel filter banks for the Log-Mel and MFCC transformation. The frame length is 80 ms, and the frameshift is 10 ms. In such a case, the shape of the MFCC/Log-mel would be $64\times 76$. Additionally, we use a window size of 9 to calculate the delta and acceleration coefficients. This value is often used as a reasonable compromise between capturing sufficient temporal context and avoiding over-smoothing. It has been found to work well empirically in many audio processing tasks.

To evaluate model performance, we perform 5-fold cross-validation. The dataset is divided into five stratified folds. In each of five iterations, four folds are used for training, while the remaining fold is split in half for validation and testing, resulting in an 8:1:1 ratio for training, validation, and test sets. The CNN models described in the Methodology section are trained from scratch for 50 epochs using cross-entropy loss and the stochastic gradient descent (SGD) optimizer. The model achieving the highest validation accuracy is retained for subsequent testing.

In the BELLHOP simulation, standard environmental settings are used, including seabed topography, acoustic parameters, and the sound velocity profile. A source and receiver location are selected to compute the acoustic channel impulse response. Both the source and receiver are positioned 100 meters underwater, consistent with typical dolphin activity and hydrophone deployment. Figure 6 illustrates the locations of the source and receiver, along with the surrounding underwater terrain.

Figure 7 shows spectrograms of an example original signal (left), its corresponding simulated signal (middle), and the difference between them (right).

The models are trained on three types of datasets: (1) original training data (org), (2) simulated training data (sim), and (3) a combination of original and simulated data (all). They are evaluated on corresponding test sets: (1) original (org), (2) simulated (sim), and (3) combined (all), with varying SNR levels to assess model generalizability.

4.2. Results

Our analysis begins with an evaluation of all input feature representations across all models using the clean datasets. The results are summarized in Figure ?? and Table 3. All three input representations, namely, Log-Mel (logmel), MFCC (mfcc), and the raw waveforms (wave), yield strong performance, with every model achieving an mAP above 0.80. Although the waveform representation performs slightly below the MFCC and Log-Mel inputs across most architectures, the gap is modest. For example, the largest mAP difference between waveform and Log-Mel inputs is approximately 0.04 on the Xception model. Several factors may account for the performance advantage of spectrogram-based features. First, converting the waveform to a time-frequency representation can reduce the burden on the network to learn low-level spectral decomposition. Second, MFCC and Log-Mel features emphasize perceptually salient components, which may improve the training efficiency under limited data. In contrast, waveform-based models must learn these transformations implicitly, which often requires larger datasets or deeper architectures. In general, MFCC and Log-Mel inputs perform similarly. The highest mAP is achieved by the Xception architecture with the Log-Mel input, while the MFCC input lags behind by only 0.01, indicating that both representations are highly effective for whistle classification.

In Table 3, we examine the class level AP values. For each model, Log-Mel and MFCC demonstrate generally comparable performance across all categories. Interestingly, within the ResNet architecture, MFCC lags behind Log-Mel by roughly 0.05 across the CV, SIN, and UP categories. And the waveform input in general doesn’t perform well on CV and SIN classes. This degradation is likely influenced in part by the imbalanced class distribution.

We further examine the confusion matrices of the three input feature representations on the first fold of the best-performing Xception model. A large number of CV samples are misclassified as UP, leading to a reduced average precision for the CV class. This confusion is likely driven by the short duration of certain CV signals, causing models to interpret their rapid, transient frequency changes as resembling UP patterns. In contrast, the DW class—despite its limited sample size—shows more distinctive characteristics, as reflected in its consistently high AP scores. This observation indirectly supports the validity of the newly defined DW category, which is clearly separable from the six previously established classes.

Figure 8. Confusion matrices of the Xception model on first-fold data using three input representations: Log-Mel (left), MFCC (middle), and Wave (right).

Figure 8. Confusion matrices of the Xception model on first-fold data using three input representations: Log-Mel (left), MFCC (middle), and Wave (right).

We further investigate fine-tuning ImageNet-pretrained CNN models [21] for the dolphin whistle classification task. The training dataset, methodology, model architectures, and hyperparameters are kept identical to those used for models trained from scratch. The resulting performance is summarized in Table 4 and illustrated in Figure 9.

As shown in Figure 9, the ResNet-family models benefit from pretraining and fine-tuning, whereas models with fewer trainable parameters, such as MobileNet and Xception, show no improvement. This advantage is likely due to the residual connections and larger model capacity in ResNet architectures, which facilitate more effective parameter updates and adaptation to new domains. It is also worth noting that the fine-tuned SE-ResNeXt performs poorly with MFCC inputs. This may be because MFCC processing generates a compact set of largely uncorrelated coefficients, whereas the pre-trained SE modules are designed to capture channel correlations important for visual recognition. Since these learned channel relationships are not present in MFCC features, fine-tuning on them results in degraded performance. Conversely, starting from scratch to capture these MFCC features might yield better results than the original spectral features. Log-Mel features more closely resemble image-like representations, allowing the fine-tuned model to better leverage prior knowledge and achieve superior results. In contrast, models fine-tuned on waveform inputs show smaller performance gains, as waveform data fundamentally differ from the two spectral representations.

We then assess model robustness on simulated datasets. The mAP results for Log-Mel, MFCC, and waveform inputs are presented in Table 5, Table 6, and Table 7, respectively. Models trained on the original data with all three input types show only a slight decrease in mAP (0.01–0.02) when evaluated on simulated data, indicating that they can still accurately classify the simulated signals. These results also suggest that, despite visual similarity on spectrograms, the simulated data introduce slightly more confusion in classification than the original recordings. When tested on signals with different SNR levels, models perform comparably to the clean dataset at high (40 dB, 30 dB) and moderate (20 dB) signal-to-noise ratio (SNR) levels. At lower SNR (10 dB), performance declines moderately but remains acceptable. When the SNR drops to 0 dB, i.e., noise intensity matches the signal, model performance decreases sharply, as the noise overwhelms the signal. Overall, all five models maintain adequate classification accuracy at SNRs of 10 dB or higher, suggesting potential applicability to real-world data. For lower-quality signals, further processing, such as signal enhancement, would be needed, though this is beyond the scope of the current study.

Finally, we explore the potential of using simulated data for data augmentation. Models are trained using both the original training data and the corresponding simulated signals, and tested under the same conditions as described above. The results, shown in Table 5, Table 6 and Table 7, indicate that incorporating simulated data generally improves mAP scores. The augmented training also enhances model robustness to noise, particularly at 0 dB SNR. Overall, these experiments demonstrate that the proposed data simulation process effectively improves model generalizability.

5. Conclusions and Future Work

This work presents a novel Tursiops aduncus whistles dataset, consisting of 3913 manually annotated signals across seven whistle types. Using this dataset, we develop a comprehensive framework to train and evaluate various CNN architectures and input feature representations. The experiments demonstrate that both the training-from-scratch model and the fine-tuned ImageNet-pretrained models achieve strong performance, with mAP consistently above 0.8. Xception performs best from scratch using Log-Mel and MFCC features, while pretrained ResNet-family models deliver comparable or slightly better accuracy. To assess the robustness of the models in a complex environment, we introduce simulated data generated with the Bellhop acoustic channel and added real marine noise from the SHIPSEAR dataset. The models maintain reliable performance on simulated data and remain effective under moderate noise. Training with simulated data further improves accuracy and noise robustness, demonstrating its value as a practical augmentation strategy for whistle signal classification. Future work will focus on efficient dolphin signal detection methods and lightweight classification models suitable for deployment on resource-limited devices.