Preprint
Article

This version is not peer-reviewed.

Asynchronous Acoustic–Image Classification for Unattended Ground Sensors with Domain Adaptation and Bidirectional Prototype Distillation

A peer-reviewed version of this preprint was published in:
Applied Sciences 2026, 16(19), 9568. https://doi.org/10.3390/app16199568

Submitted:

08 August 2026

Posted:

11 August 2026

You are already at the latest version

Abstract
Acoustic and image sensors in unattended ground sensor systems often operate with different sampling rates, trigger mechanisms, and fields of view, so their observations are asynchronous and lack instance-level correspondence. In addition, web images used to supplement scarce field annotations exhibit a substantial domain shift from target-domain images. This paper proposes a three-stream joint learning framework that integrates reliability-weighted image-domain adaptation, reliability-aware dynamic bidirectional prototype distillation, and balanced complementary modality dropout. A lightweight shared backbone processes labeled acoustic spectra, labeled mixed-source images, and unlabeled target-domain images. Target pseudo-label reliability is estimated by a main classifier and two auxiliary classifiers and is used to weight marginal and class-conditional distribution alignment. Acoustic and image class prototypes transfer category-level knowledge without constructing artificial instance pairs, while direction-specific distillation weights are dynamically assigned according to classification reliability, prototype availability, and prototype stability. Experiments on 148,038 acoustic spectra and 167,754 images yield acoustic, image, and overall accuracies of 92.02\%, 95.26\%, and 93.79\%, respectively, exceeding the best corresponding baseline results by 1.49, 1.24, and 2.47 percentage points. The deployment model contains 123.40 K parameters and requires 0.47 MiB in FP32, supporting resource-constrained field deployment.
Keywords: 
;  ;  ;  ;  ;  ;  

1. Introduction

Unattended ground sensor (UGS) systems provide persistent, covert, and distributed monitoring for moving ground targets. Acoustic sensors offer low-cost, wide-area, and all-weather observations, whereas image sensors provide visually interpretable appearance cues with strong discriminative value. Combining these heterogeneous sensing channels can improve target classification reliability in field environments [1,2]. However, practical acoustic–image UGS systems differ substantially from conventional synchronized audio–visual datasets. Acoustic and image sensors commonly use different sampling rates, trigger rules, sensing ranges, and fields of view. Their observations therefore cannot be assumed to form temporally aligned or instance-level paired samples.
The image branch also faces a severe data limitation. Although field videos can provide UGS image frames, consecutive frames are highly correlated and cover only a limited range of scenes, target instances, and environmental conditions. Public web images can expand the labeled training set, but the resulting source-side images differ from target UGS images in viewpoint, scale, illumination, background, and acquisition conditions. Such dataset bias and distribution shift can substantially reduce generalization performance [3,4,5]. Consequently, the learning problem involves two coupled challenges: image-domain adaptation from labeled mixed-source images to unlabeled UGS images, and cross-modal knowledge transfer between non-paired acoustic and image samples.
Existing domain adaptation methods reduce source–target discrepancy by statistical distribution matching or adversarial learning [6,7,8,9]. These methods are effective for image-domain transfer but do not explicitly exploit category-level acoustic–image complementarity. Conversely, multimodal alignment and cross-modal distillation methods can associate heterogeneous representations [10,11,12], but most assume paired or weakly paired observations and rarely address simultaneous image-domain shift. Moreover, fixed and symmetric knowledge transfer can propagate unreliable information when the two modalities converge at different rates or contain unequal class evidence.
To address these limitations, this paper develops an asynchronous acoustic–image joint learning framework that learns from three independently sampled data streams: labeled acoustic spectra, labeled mixed-source images, and unlabeled target-domain UGS images. The method combines reliability-weighted image-domain alignment with category-level bidirectional prototype distillation and a balanced complementary modality dropout strategy. The main contributions are as follows:
  • A three-stream shared learning architecture is formulated for asynchronous and non-paired acoustic–image observations. It jointly handles labeled acoustic data, labeled mixed-source images, and unlabeled target images without constructing instance-level acoustic–image pairs.
  • A reliability-weighted two-stage image-domain adaptation mechanism is introduced. Multi-classifier consensus estimates target pseudo-label reliability, which weights both marginal distribution alignment and class-conditional alignment.
  • A reliability-aware dynamic bidirectional prototype distillation mechanism is proposed. Direction-specific transfer strengths are determined from teacher classification reliability, prototype availability, and prototype stability, reducing negative transfer caused by fixed symmetric distillation.
  • A balanced complementary modality dropout mechanism regularizes the shared backbone while preserving equal marginal retention probabilities for both modalities and preventing simultaneous removal of the two labeled streams. The complete method attains 93.79% overall accuracy with a compact 123.40 K-parameter deployment network.
The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 describes the datasets and proposed method. Section 4 presents the experimental settings and results. Section 5 discusses the findings and limitations, and Section 6 concludes the paper.

3. Materials and Methods

3.1. Problem Formulation

Let the labeled acoustic dataset be
D A = { ( x i A , y i A ) } i = 1 N A ,
where x i A ∈ R 1 × 32 × 32 is an acoustic power spectral density image and y i A ∈ { 1 , … , C } . The labeled mixed-source image domain is
D I s = { ( x i I s , y i I s ) } i = 1 N s ,
which contains web images and a small subset of labeled UGS images. The unlabeled target domain is
D I t = { x j I t } j = 1 N t .
The three datasets share the same category set but contain no temporal or instance-level correspondence. In each iteration, mini-batches are independently sampled from
P ( B A , B I s , B I t ) = P A ( B A ) P I s ( B I s ) P I t ( B I t ) ,
which prevents the model from relying on artificial acoustic–image pairings.
The task is to learn a shared feature extractor and classifier that correctly classify acoustic samples and target-domain UGS images. During inference, each acoustic or image sample is processed independently; simultaneous availability of both modalities is not required.

3.2. Datasets and Preprocessing

3.2.1. Acoustic Data

The acoustic data contain pedestrian, tracked-vehicle, and wheeled-vehicle observations collected by a UGS acoustic array. To maintain a consistent single-node input configuration, only the first acoustic channel is used. Each waveform is divided into one-second windows containing 1024 samples. A short-time Fourier transform is applied to obtain a power spectral density representation [28], which is cropped, normalized, and resized to a 1 × 32 × 32 grayscale spectrum.
The acoustic dataset is divided using stratified group splitting with an approximate ratio of 8:1:1. Windows from the same original recording are assigned to only one subset to prevent near-duplicate leakage. Table 1 summarizes the resulting 148,038 acoustic samples.

3.2.2. Image Data

UGS images are extracted from field videos collected at target distances of approximately 20–200 m during the day and at night, with grass, gravel, cement, and hard soil as backgrounds. Figure 1 shows representative deployment environments and field-collected observations, illustrating the long sensing distances, complex backgrounds, varying illumination conditions, and small apparent target sizes encountered in practical UGS operation. Target bounding boxes are annotated and cropped from 13,012 original frames with a resolution of 1920 × 1080 . The cropped images are resized with aspect-ratio-preserving padding and converted to 1 × 32 × 32 grayscale images. The UGS subset contains 47,706 target crops: 15,746 pedestrian, 16,681 tracked-vehicle, and 15,279 wheeled-vehicle images.
To improve source-side diversity, 120,048 labeled web images are added, including 39,640 pedestrian, 40,240 tracked-vehicle, and 40,168 wheeled-vehicle samples. The final image dataset contains 167,754 samples. A small labeled UGS subset is combined with the web images to form the labeled mixed-source domain I s ; the remaining UGS training images form the unlabeled target domain I t . UGS videos are grouped before splitting so that frames from one video do not cross training, validation, and test subsets. Table 2 gives the detailed distribution.
Figure 2 illustrates the heterogeneous acoustic and image inputs and their shared category-level, rather than instance-level, correspondence.

3.3. Overview of the Proposed Framework

Figure 3 presents the complete framework. Three streams independently process acoustic spectra A, labeled mixed-source images I s , and unlabeled target images I t . They share convolutional and classifier parameters but use modality-dependent batch normalization: the acoustic stream has independent batch-normalization statistics, whereas the two image streams share image-domain statistics. Related domain-specific batch-normalization designs separate low-level domain statistics while sharing the remaining network parameters [29]. This design retains a common semantic space while avoiding direct mixing of acoustic and image feature distributions at normalization layers.
The framework contains three interacting mechanisms. First, reliability-weighted maximum mean discrepancy and class-wise conditional MMD align source and target image distributions. Second, acoustic and image class-prototype memories establish category-level cross-modal transfer. Third, balanced complementary modality dropout regularizes the labeled acoustic and image residual paths. The unlabeled target stream remains active to preserve stable domain alignment.
Figure 3. Overall architecture of the proposed asynchronous acoustic–image joint learning framework. Acoustic, labeled mixed-source image, and target-image samples are independently drawn and processed by a shared lightweight backbone.
Figure 3. Overall architecture of the proposed asynchronous acoustic–image joint learning framework. Acoustic, labeled mixed-source image, and target-image samples are independently drawn and processed by a shared lightweight backbone.
Preprints 227477 g003

3.4. Shared Lightweight Backbone

The shared feature extractor follows an inverted-residual design inspired by MobileNetV2 [30]. All inputs have one channel and a spatial size of 32 × 32 . A stem convolution is followed by ten inverted-residual blocks with an expansion factor of two. The base channel width is eight, and the final embedding dimension is d = 128 . Table 3 lists the architecture. Shortcut connections are used only when the stride is one and the input and output channel dimensions match.
Let G ( · ; θ ) denote the shared feature extractor and H ( · ; ϕ ) the main classifier. The high-level embeddings and logits are
f m = G ( x m ; θ ) , z m = H ( f m ; ϕ ) , m ∈ { A , I s , I t } .
The acoustic and image mini-batches contribute separately to the same parameter update, enabling cross-modal regularization through shared weights without assuming synchronized inputs.

3.5. Reliability-Weighted Image-Domain Adaptation

3.5.1. Multi-Classifier Target Reliability

Reliable pseudo-labels are required for class-conditional image alignment. In addition to the main classifier, the framework uses a two-layer multilayer perceptron (MLP) classifier and a graph convolutional network (GCN) classifier [31]. The GCN operates on a graph constructed from source and target image features. For a target sample j, the ensemble probability is
p ¯ j = 1 3 p j main + p j MLP + p j GCN .
The pseudo-label and confidence are
y ^ j I t = arg max c p ¯ j , c , q j = max c p ¯ j , c .
Classifier agreement is defined as
a j = 1 3 I ( y ^ j main = y ^ j MLP ) + I ( y ^ j main = y ^ j GCN ) + I ( y ^ j MLP = y ^ j GCN ) .
The final reliability score combines confidence and agreement:
r j = q j γ ρ + ( 1 − ρ ) a j ,
where γ = 2.0 emphasizes confident predictions and ρ = 0.25 retains a minimum contribution from confidence when classifier agreement is incomplete.

3.5.2. Reliability-Weighted Marginal Alignment

Let u i I s and u j I t denote B10-layer image features. Source weights are uniform, α i s = 1 / n s , whereas target weights are normalized reliability scores,
α j t = r j ∑ k = 1 n t r k + ε .
The reliability-weighted MMD loss is
L RW - MMD = ∑ i , i ′ α i s α i ′ s k ( u i I s , u i ′ I s ) + ∑ j , j ′ α j t α j ′ t k ( u j I t , u j ′ I t ) − 2 ∑ i , j α i s α j t k ( u i I s , u j I t ) ,
where k ( · , · ) is a multi-kernel Gaussian function. Five bandwidths { b / 4 , b / 2 , b , 2 b , 4 b } are used around the batch-dependent base bandwidth b. Reliability weighting limits the influence of uncertain target samples during early training.

3.5.3. Class-Wise Conditional Alignment

Marginal alignment does not ensure that samples of the same class overlap across domains. A class-wise conditional MMD loss is therefore computed using source labels and high-confidence target pseudo-labels. Target sample j participates only when q j ≥ τ , with τ = 0.8 . For class c, let S c and T c denote the source and selected target index sets. The class is valid only if both sets contain at least n min = 2 samples. Its discrepancy is
D c = 1 | S c | ∑ i ∈ S c φ ( u i I s ) − 1 | T c | ∑ j ∈ T c φ ( u j I t ) H 2 .
The class weight is the average target confidence,
ω c = 1 | T c | ∑ j ∈ T c q j ,
and the class-wise conditional loss is
L CW - CMMD = ∑ c ∈ C v ω c D c ∑ c ∈ C v ω c + ε ,
where C v is the set of valid classes. The complete image-domain adaptation loss is
L DA = λ MMD L RW - MMD + λ CMMD L CW - CMMD .

3.6. Reliability-Aware Dynamic Bidirectional Prototype Distillation

Figure 4 provides an overview of the bidirectional prototype-distillation mechanism and the category-level supervision exchanged between the two modalities.

3.6.1. Modality-Specific Prototype Memories

The acoustic and labeled image streams maintain separate class-prototype matrices
P A = [ p 1 A , … , p C A ] ⊤ , P I = [ p 1 I , … , p C I ] ⊤ ,
where each row is an L 2 -normalized class centroid in R d . For modality m ∈ { A , I s } and class c, the current batch centroid p ˜ c m updates the memory using an exponential moving average:
p c m ← Norm m p p c m + ( 1 − m p ) p ˜ c m ,
with m p = 0.9 . Only labeled samples are used to update the prototypes, and the memory updates are detached from gradient propagation.
The acoustic-to-image and image-to-acoustic prototype distillation losses are
L A → I s = 1 | Ω A → I s | ∑ j ∈ Ω A → I s 1 − cos ( f j I s , p y j I s A ) ,
L I s → A = 1 | Ω I s → A | ∑ i ∈ Ω I s → A 1 − cos ( f i A , p y i A I ) ,
where the valid sets contain samples whose teacher-side class prototypes have already been initialized.

3.6.2. Teacher Reliability and Dynamic Directional Weights

Prototype stability for class c is first measured by
ξ c m = 1 + cos ( p c m , old , p c m , new ) 2 .
A smoothed stability variable is updated as
κ c m ← clip m s κ c m + ( 1 − m s ) ξ c m , κ min , 1 ,
where m s = 0.9 and κ min = 0.2 . Classification reliability is computed from the supervised classification loss:
r m cls = exp − L cls m T PKD ,
with T PKD = 1.0 . For each teacher direction, let η m denote the fraction of usable teacher prototypes and κ ¯ m their average stability. The overall modality reliability is
R m = r m cls η m κ ¯ m , m ∈ { A , I s } .
The total prototype-distillation budget λ PKD ( e ) is warmed up to 0.03 during the first ten epochs. It is dynamically divided between the two directions:
λ A → I s ( e ) = λ PKD ( e ) R A R A + R I s + ε ,
λ I s → A ( e ) = λ PKD ( e ) R I s R A + R I s + ε .
Thus, the dynamic prototype-distillation objective is
L DPKD = λ A → I s L A → I s + λ I s → A L I s → A .
This formulation maintains a bounded total distillation strength while allowing the more reliable modality to exert stronger supervision at each stage of training.

3.7. Balanced Complementary Modality Dropout

Modality dropout can prevent a shared network from over-relying on one input source [32]. Directly dropping complete branches, however, may interrupt the supervision required by domain adaptation and prototype updates. The proposed balanced complementary modality dropout (BCMD) is therefore applied only to the residual transformation paths of shortcut-enabled blocks in the two labeled streams. The target-image stream is never dropped. Figure 5 illustrates the complementary gating of the labeled streams and the preserved identity shortcut.
For every labeled batch position, the acoustic gate is sampled as
G A ∼ Bernoulli ( p ) , p = 0.7 .
An auxiliary compensation gate is
Q I s ∼ Bernoulli ( q ) , q = 2 p − 1 p ,
and the labeled image gate is
G I s = ( 1 − G A ) ∨ Q I s .
This construction satisfies
P ( G A = 1 ) = P ( G I s = 1 ) = p ,
while preventing G A = G I s = 0 . For a shortcut block with input h and residual transform F ( h ) , the output is
h ′ = h + F ( G m ⊙ h ) , m ∈ { A , I s } .
The identity path remains unchanged. At inference, all gates are set to one, so BCMD introduces no additional parameters or inference-time computation.

3.8. Joint Objective and Progressive Optimization

The main supervised classification loss is
L cls = L CE A + L CE I s .
The auxiliary classifier loss is
L aux = λ MLP L MLP I s + λ GCN L GCN I s .
The total training objective at epoch e is
L total ( e ) = L cls + L aux + L DPKD ( e ) + L DA ( e ) .
To avoid imposing unreliable constraints before the shared representation becomes discriminative, auxiliary objectives are progressively activated. A generic schedule is
w ( e ; e 0 , E ) = 0 , e < e 0 , min 1 , e − e 0 + 1 E , e ≥ e 0 ,
where e 0 is the activation epoch and E is the warm-up duration. Prototype distillation begins in the first epoch and reaches 0.03 after ten epochs. RW-MMD starts at epoch 5 and reaches a maximum weight of 0.03, whereas CW-CMMD starts at epoch 10, is warmed up for ten epochs, and reaches a maximum weight of 0.005.

4. Experiments and Results

4.1. Experimental Settings

The model is trained for 100 epochs using Adam [33] with an initial learning rate of 10 − 3 , β 1 = 0.9 , and β 2 = 0.999 . No weight decay is used. The mini-batch size is 128 for each stream. The main network dropout rate is 0.5. Image augmentation includes random horizontal flipping and affine transformations with a maximum rotation of ± 8 ∘ and translation of 0.05. Horizontal flipping is not applied to acoustic spectra. The MLP and GCN auxiliary classifiers use 64 hidden units, dropout rates of 0.2 and 0.3, and loss weights of 0.1 and 0.2, respectively. The model used for validation and testing is an exponential moving average of the online parameters with a decay of 0.999, following the stabilizing principle of mean-teacher training [34]. The experiments use an NVIDIA GeForce RTX 5090 (24 GB), CUDA 13.2, and a random seed of 42.
Acoustic accuracy, target-image accuracy, and overall accuracy are reported. Overall accuracy is computed over the combined acoustic and target-image test samples rather than by directly averaging the two modality accuracies. Hyperparameters specific to comparison methods are selected using the validation set. All methods use the same data partitions, preprocessing, input sizes, base backbone, classification heads, batch size, training epochs, optimizer, model-selection rule, EMA evaluation strategy, and evaluation metrics whenever compatible with their original mechanisms.
Table 4 summarizes the principal optimization, augmentation, alignment, distillation, and regularization settings.

4.2. Training Behavior

Figure 6 summarizes the optimization process. The total and task-specific classification losses decrease rapidly in the early epochs and then converge. Acoustic, image, and overall validation accuracies increase consistently and then stabilize. The two image-domain alignment losses activate at their scheduled epochs, while ensemble confidence, classifier agreement, and the valid pseudo-label ratio improve as training proceeds.
The total prototype-distillation weight increases to 0.03 during warm-up. The normalized image-to-acoustic allocation decreases from approximately 0.57 to 0.54, whereas the acoustic-to-image allocation increases from approximately 0.43 to 0.46. In addition, the acoustic-to-image distillation loss remains lower overall than the reverse-direction loss. These results show that the two transfer directions differ in difficulty and that the proposed reliability mechanism maintains a stable, moderately asymmetric allocation rather than imposing a fixed 1:1 transfer ratio.
Figure 6. Training behavior of the proposed framework.
Figure 6. Training behavior of the proposed framework.
Preprints 227477 g006

4.3. Comparison with Representative Methods

Nine comparison methods are evaluated. DAN, DANN, CDAN, MCC, MME, and TriCT represent statistical, adversarial, conditional, discriminative, and semi-supervised image-domain adaptation. CROMA, GMC, and XKD represent weak-pair cross-modal meta-alignment, multimodal contrastive learning, and cross-modal knowledge distillation. For methods requiring acoustic–image inputs, same-class samples are dynamically associated at the batch level, without constructing fixed instance pairs.
Table 5 reports the results. The proposed method achieves 92.02% acoustic accuracy, 95.26% target-image accuracy, and 93.79% overall accuracy. Among the compared methods, CDAN obtains the highest acoustic accuracy of 90.53%, TriCT obtains the highest image accuracy of 94.02%, and TriCT also obtains the highest overall accuracy of 91.32%. The proposed method therefore improves these strongest baseline results by 1.49, 1.24, and 2.47 percentage points, respectively.
Domain adaptation methods improve target-image transfer but generally do not exploit acoustic–image semantic complementarity. TriCT produces strong target-image performance but lower acoustic accuracy, indicating an imbalanced benefit across streams. Compared with TriCT, the proposed method improves target-image accuracy by 1.24 percentage points and acoustic accuracy by 3.99 percentage points, demonstrating a more balanced benefit across the two modalities. Cross-modal methods improve knowledge exchange but do not explicitly resolve the web-to-UGS image shift. Integrating both aspects allows the proposed method to improve the two modalities simultaneously.
Table 5. Classification performance comparison with domain adaptation and cross-modal learning methods. The best results are shown in bold.
Table 5. Classification performance comparison with domain adaptation and cross-modal learning methods. The best results are shown in bold.
Method Acoustic Accuracy (%) Image Accuracy (%) Overall Accuracy (%) Primary Mechanism
DAN [7] 90.40 89.45 89.88 MK-MMD domain alignment
DANN [8] 90.35 91.39 90.92 Adversarial domain alignment
CDAN [9] 90.53 90.16 90.33 Conditional adversarial alignment
MCC [19] 88.98 89.60 89.32 Target class-confusion minimization
MME [20] 89.19 72.91 80.26 Semi-supervised minimax entropy
TriCT [21] 88.03 94.02 91.32 Semi-supervised tri-training
CROMA [10] 90.34 90.12 90.22 Weak-pair meta-alignment
GMC [11] 90.05 91.00 90.57 Geometric multimodal contrast
XKD [12] 90.37 91.08 90.76 Cross-modal distillation and alignment
Proposed method 92.02 95.26 93.79 Reliability-aware joint adaptation
Figure 7 shows the confusion matrices. Pedestrian targets are classified most reliably in both modalities. The remaining errors mainly occur between tracked and wheeled vehicles, whose acoustic spectra can overlap at varying distances and under different operating states and whose low-resolution grayscale appearances can be similar under occlusion or oblique viewpoints.
Figure 8 presents representative correct predictions and typical errors from the acoustic and target-image test sets. The correctly classified samples show that the model can recognize all three categories across varied spectral patterns and visual appearances. The error cases are consistent with the confusion matrices: acoustically similar vehicle signatures are occasionally interchanged, while small, blurred, or structurally ambiguous image targets can be confused with another vehicle category or with pedestrians. These examples illustrate the model’s prediction behavior and the remaining difficulty posed by low-quality field observations.

4.4. Ablation Study

The ablation study progressively adds reliability-weighted marginal alignment, class-wise conditional alignment, prototype distillation, dynamic directional weighting, and BCMD. Table 6 shows that the shared-backbone baseline (S1) obtains 86.79% overall accuracy. RW-MMD (S2) increases the overall accuracy to 88.94%, demonstrating the importance of correcting global image-domain shift. Adding CW-CMMD (S3) yields 90.02%, indicating that class-aware alignment further improves target discrimination.
Fixed symmetric bidirectional prototype distillation (S4) increases overall accuracy to 91.05%, confirming that class prototypes can transfer semantic information without instance pairs. Dynamic reliability-aware weighting (S5) raises overall accuracy to 92.92%, which is 1.87 percentage points higher than fixed symmetric distillation. Finally, BCMD (S6) produces the best overall accuracy of 93.79%.
Table 6. Ablation study of the proposed components.
Table 6. Ablation study of the proposed components.
Setting Added Component Acoustic Accuracy (%) Image Accuracy (%) Overall Accuracy (%) Interpretation
S1 Shared three-stream backbone 88.02 85.78 86.79 Basic shared representation
S2 S1 + RW-MMD 89.14 88.77 88.94 Marginal image-domain alignment
S3 S2 + CW-CMMD 90.06 89.99 90.02 Class-conditional image alignment
S4 S3 + fixed bidirectional PKD 90.18 91.76 91.05 Category-level cross-modal transfer
S5 S4 + dynamic directional weights 91.65 93.97 92.92 Reliability-aware transfer
S6 S5 + BCMD 92.02 95.26 93.79 Complete model

4.5. Feature and Prototype Analysis

Figure 9 visualizes target-image embeddings using t-distributed stochastic neighbor embedding (t-SNE) [27]. Without domain alignment, source and target samples form visibly separated distributions. RW-MMD reduces the global domain gap but retains local class overlap. Adding CW-CMMD produces more compact target clusters and clearer class boundaries. These observations are consistent with the quantitative improvement from S1 to S3.
Table 7 reports acoustic–image prototype similarity. Without prototype distillation, the mean same-class similarity is high, but the mean different-class similarity remains positive, resulting in a margin of only 0.432. Fixed distillation increases the margin to 0.490. Dynamic bidirectional distillation produces a mean same-class similarity of 0.986 and a mean different-class similarity of − 0.493 , increasing the inter-class semantic margin to 1.479. These results indicate that dynamic weighting not only aligns corresponding categories but also improves separation between non-corresponding prototypes.
Figure 10 complements the aggregate statistics in Table 7 by visualizing the class-wise similarity patterns under the three distillation strategies.

4.6. Model Complexity and Deployment

The auxiliary MLP and GCN classifiers, target pseudo-label ensemble, domain-alignment losses, prototype memories, and BCMD are required only during training. At inference, the model contains the shared feature extractor, modality-dependent batch-normalization parameters, and main classifier. Table 8 compares the training and deployment configurations.
The complete training network contains 156.81 K parameters, of which approximately 33.42 K belong to auxiliary classifiers. The deployment network contains 123.40 K parameters and occupies approximately 0.47 MiB in FP32. Training two independent acoustic and image networks with the same basic architecture would require 237.89 K parameters and approximately 0.91 MiB. Shared learning therefore reduces deployment parameters by 48.13%. A single acoustic or image forward pass activates approximately 118.95 K parameters and requires about 0.00924 GMAC. The two class-prototype banks contain only 768 floating-point values, approximately 3 KiB, and are discarded after training.
Table 8. Training and deployment complexity.
Table 8. Training and deployment complexity.
Configuration Parameters (K) FP32 Storage (MiB) Active Parameters per Sample (K) Compute per Sample (GMAC) Notes
Full training network 156.81 – – – Includes auxiliary classifiers
Proposed deployment network 123.40 0.47 118.95 0.00924 Single shared model for both input types
Two separate deployment networks 237.89 0.91 118.95 0.00924 Independent acoustic and image models

5. Discussion

The results support three main observations. First, image-domain adaptation and cross-modal transfer solve different parts of the problem and are complementary. Global and class-conditional alignment reduce the discrepancy between web and UGS images, while prototype distillation transfers acoustic–image category knowledge. Applying only one of these mechanisms leaves either the domain shift or the cross-modal semantic gap insufficiently addressed.
Second, transfer direction should depend on training state. The 1.87-percentage-point improvement of dynamic weighting over fixed symmetric distillation indicates that the acoustic and image streams do not provide equally reliable supervision throughout optimization. Combining classification confidence, prototype availability, and prototype stability offers a compact way to regulate the teacher role without introducing an additional learned weighting network.
Third, the shared backbone provides both statistical and deployment benefits. Sharing convolutional and classifier parameters allows gradients from the three streams to regularize a common semantic representation. Modality-dependent batch normalization preserves low-level statistical differences, and BCMD discourages over-dependence on one labeled stream. The deployment network is approximately half the size of two independent models while permitting acoustic-only or image-only inference.
Several limitations remain. The current study includes three ground-target categories and evaluates a specific UGS collection environment. Generalization to additional categories, sensor types, weather conditions, geographic regions, and more severe target imbalance requires further validation. The target-image branch uses unlabeled data during training, but the acoustic branch is not explicitly adapted across deployment sites. In addition, the class-prototype formulation captures category-level correspondence but does not model fine-grained subcategories or operating-state variation. Future work should investigate open-set targets, incremental prototype updates, site-adaptive acoustic learning, and uncertainty-calibrated pseudo-label selection.

6. Conclusions

This paper presented an asynchronous acoustic–image joint learning method for UGS target classification under non-paired observations and image-domain shift. A shared three-stream backbone learns from labeled acoustic spectra, labeled mixed-source images, and unlabeled target UGS images. Reliability-weighted marginal and class-conditional alignment improve image transfer; dynamic bidirectional prototype distillation establishes category-level acoustic–image knowledge exchange; and BCMD regularizes the shared model without increasing inference cost. The complete method achieves 92.02% acoustic accuracy, 95.26% target-image accuracy, and 93.79% overall accuracy, outperforming representative domain adaptation and cross-modal learning methods. With 123.40 K deployment parameters and 0.47 MiB FP32 storage, the method is suitable for resource-constrained UGS nodes and other asynchronous heterogeneous sensing systems.

Author Contributions

Conceptualization, Y.W. and B.L.; methodology, Y.W. and L.X.; software, Y.W.; validation, Y.W. and J.P.; formal analysis, Y.W.; investigation, Y.W., L.X. and J.P.; resources, B.L.; data curation, Y.W. and L.X.; writing—original draft preparation, Y.W.; writing—review and editing, Y.W., L.X., J.P. and B.L.; visualization, Y.W.; supervision, B.L.; project administration, B.L.; funding acquisition, Y.W. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the Youth Innovation Promotion Association, Chinese Academy of Sciences, grant number E34YCA1001.

Institutional Review Board Statement

Not applicable. This study did not involve experiments with human participants or animals. The web-sourced pedestrian images comprise publicly available data depicting public scenes and involve no interaction or intervention with individuals; therefore, institutional ethical approval was not required.

Data Availability Statement

The data supporting the findings of this study, the trained models, and the source code are available from the corresponding author upon reasonable request. The field-collected UGS data are not publicly available because of security and institutional restrictions, and the web images cannot be redistributed because they are subject to third-party copyright.

Acknowledgments

During the preparation of this manuscript, the authors used OpenAI Codex for language editing and formatting assistance. The authors reviewed and edited the output and take full responsibility for the content of this publication.

Conflicts of Interest

The authors declare no conflicts of interest. The funder had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

Abbreviations

The following abbreviations are used in this manuscript:
UGS Unattended ground sensor
MMD Maximum mean discrepancy
RW-MMD Reliability-weighted marginal maximum mean discrepancy
CW-CMMD Class-wise conditional maximum mean discrepancy
PKD Prototype knowledge distillation
DPKD Dynamic prototype knowledge distillation
BCMD Balanced complementary modality dropout
MLP Multilayer perceptron
GCN Graph convolutional network
EMA Exponential moving average
STFT Short-time Fourier transform
GMAC Giga multiply–accumulate operations

References

  1. Bokareva, T.; Hu, W.; Kanhere, S.; Ristic, B.; Gordon, N.; Bessell, T.; Rutten, M.; Jha, S. Wireless sensor networks for battlefield surveillance. In Proceedings of the Land Warfare Conference; 2006; pp. 1–8.
  2. Sleefe, G.E.; Ladd, M.D.; McDonald, T.S.; et al. Acoustic and seismic modalities for unattended ground sensors. In Unattended Ground Sensor Technologies and Applications; SPIE: Bellingham, WA, USA, 1999; Volume 3713, pp. 2–9. [CrossRef]
  3. Torralba, A.; Efros, A.A. Unbiased look at dataset bias. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2011; pp. 1521–1528.
  4. Recht, B.; Roelofs, R.; Schmidt, L.; Shankar, V. Do ImageNet classifiers generalize to ImageNet? In Proceedings of the 36th International Conference on Machine Learning; 2019; pp. 5389–5400.
  5. Koh, P.W.; Sagawa, S.; Marklund, H.; Xie, S.M.; Zhang, M.; Balsubramani, A.; Hu, W.; Yasunaga, M.; Phillips, R.L.; Gao, I.; et al. WILDS: A benchmark of in-the-wild distribution shifts. In Proceedings of the 38th International Conference on Machine Learning; PMLR: Proceedings of Machine Learning Research, 2021; Volume 139, pp. 5637–5664.
  6. Pan, S.J.; Yang, Q. A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 2010, 22, 1345–1359.
  7. Long, M.; Cao, Y.; Wang, J.; Jordan, M. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on Machine Learning; 2015; pp. 97–105.
  8. Ganin, Y.; Ustinova, E.; Ajakan, H.; Germain, P.; Larochelle, H.; Laviolette, F.; Marchand, M.; Lempitsky, V. Domain-adversarial training of neural networks. J. Mach. Learn. Res. 2016, 17, 1–35.
  9. Long, M.; Cao, Z.; Wang, J.; Jordan, M.I. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems; Curran Associates, Inc.: Red Hook, NY, USA, 2018; Volume 31, pp. 1640–1650.
  10. Liang, P.P.; Wu, P.; Ziyin, L.; Morency, L.P.; Salakhutdinov, R. Cross-modal generalization: Learning in low-resource modalities via meta-alignment. In Proceedings of the 29th ACM International Conference on Multimedia, Virtual Event, China, 20–24 October 2021; pp. 2680–2689. [CrossRef]
  11. Poklukar, P.; Vasco, M.; Yin, H.; Melo, F.S.; Paiva, A. Geometric multimodal contrastive representation learning. arXiv 2022, arXiv:2202.03390.
  12. Sarkar, P.; Etemad, A. XKD: Cross-modal knowledge distillation with domain alignment for video representation learning. Proc. AAAI Conf. Artif. Intell. 2024, 38, 14875–14885. [CrossRef]
  13. Hangloo, S.; Arora, B. Multimodal fusion techniques: Review, data representation, information fusion, and application areas. Neurocomputing 2025, 649, 130827. [CrossRef]
  14. Zhao, F.; Zhang, C.; Geng, B. Deep multimodal data fusion. ACM Comput. Surv. 2024, 56, 1–36. [CrossRef]
  15. Ngiam, J.; Khosla, A.; Kim, M.; Nam, J.; Lee, H.; Ng, A.Y. Multimodal deep learning. In Proceedings of the 28th International Conference on Machine Learning, Bellevue, WA, USA, 28 June–2 July 2011; pp. 689–696.
  16. Owens, A.; Wu, J.; McDermott, J.H.; Freeman, W.T.; Torralba, A. Ambient sound provides supervision for visual learning. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 11–14 October 2016; pp. 801–816.
  17. Arandjelović, R.; Zisserman, A. Look, listen and learn. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 609–617.
  18. Gretton, A.; Borgwardt, K.M.; Rasch, M.J.; Schölkopf, B.; Smola, A. A kernel two-sample test. J. Mach. Learn. Res. 2012, 13, 723–773.
  19. Jin, Y.; Wang, X.; Long, M.; Wang, J. Minimum class confusion for versatile domain adaptation. In Proceedings of the European Conference on Computer Vision, Glasgow, UK, 23–28 August 2020; pp. 464–480.
  20. Saito, K.; Kim, D.; Sclaroff, S.; Darrell, T.; Saenko, K. Semi-supervised domain adaptation via minimax entropy. In Proceedings of the IEEE/CVF International Conference on Computer Vision; 2019; pp. 8050–8058.
  21. Ngo, B.H.; Chae, Y.J.; Kwon, J.E.; Park, J.H.; Cho, S.I. Improved knowledge transfer for semi-supervised domain adaptation via Trico training strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision; 2023; pp. 19214–19223.
  22. Zou, Y.; Yu, Z.; Kumar, B.V.K.V.; Wang, J. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European Conference on Computer Vision, Munich, Germany, 8–14 September 2018; pp. 289–305.
  23. Gupta, S.; Hoffman, J.; Malik, J. Cross modal distillation for supervision transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 2827–2836.
  24. Hinton, G.; Vinyals, O.; Dean, J. Distilling the knowledge in a neural network. arXiv 2015, arXiv:1503.02531. [CrossRef]
  25. Snell, J.; Swersky, K.; Zemel, R.S. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems; Curran Associates, Inc.: Red Hook, NY, USA, 2017; Volume 30.
  26. Wen, Y.; Zhang, K.; Li, Z.; Qiao, Y. A discriminative feature learning approach for deep face recognition. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 11–14 October 2016; pp. 499–515.
  27. Van der Maaten, L.; Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 2008, 9, 2579–2605.
  28. Oppenheim, A.V.; Schafer, R.W. Discrete-Time Signal Processing, 3rd ed.; Pearson: Upper Saddle River, NJ, USA, 2010.
  29. Chang, W.G.; You, T.; Seo, S.; Kwak, S.; Han, B. Domain-specific batch normalization for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 7354–7362.
  30. Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; Chen, L.C. MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2018; pp. 4510–4520. [CrossRef]
  31. Kipf, T.N.; Welling, M. Semi-supervised classification with graph convolutional networks. In Proceedings of the International Conference on Learning Representations; 2017.
  32. Neverova, N.; Wolf, C.; Taylor, G.; Nebout, F. ModDrop: Adaptive multi-modal gesture recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2016, 38, 1692–1706.
  33. Kingma, D.P.; Ba, J. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations; 2015.
  34. Tarvainen, A.; Valpola, H. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems; Curran Associates, Inc.: Red Hook, NY, USA, 2017; Volume 30.
Figure 1. Representative UGS deployment environments and field-collected target observations. The examples illustrate variations in illumination, background, target distance, and apparent target scale.
Figure 1. Representative UGS deployment environments and field-collected target observations. The examples illustrate variations in illumination, background, target distance, and apparent target scale.
Preprints 227477 g001
Figure 2. Examples of the heterogeneous inputs used in this study. The acoustic and image samples share category labels but are not instance-level pairs.
Figure 2. Examples of the heterogeneous inputs used in this study. The acoustic and image samples share category labels but are not instance-level pairs.
Preprints 227477 g002
Figure 4. Reliability-aware dynamic bidirectional prototype distillation. Modality-specific class prototypes provide category-level semantic supervision without requiring synchronized acoustic–image pairs.
Figure 4. Reliability-aware dynamic bidirectional prototype distillation. Modality-specific class prototypes provide category-level semantic supervision without requiring synchronized acoustic–image pairs.
Preprints 227477 g004
Figure 5. Balanced complementary modality dropout. The stochastic gate acts only on the residual transformation path, while the identity shortcut remains active.
Figure 5. Balanced complementary modality dropout. The stochastic gate acts only on the residual transformation path, while the identity shortcut remains active.
Preprints 227477 g005
Figure 7. Confusion matrices of the complete model.
Figure 7. Confusion matrices of the complete model.
Preprints 227477 g007
Figure 8. Representative correct and incorrect predictions on (a) acoustic spectra and (b) target-domain images. Error labels indicate the ground-truth class followed by the predicted class.
Figure 8. Representative correct and incorrect predictions on (a) acoustic spectra and (b) target-domain images. Error labels indicate the ground-truth class followed by the predicted class.
Preprints 227477 g008
Figure 9. t-SNE visualization of image features under different domain-alignment configurations.
Figure 9. t-SNE visualization of image features under different domain-alignment configurations.
Preprints 227477 g009
Figure 10. Acoustic–image class-prototype similarity matrices.
Figure 10. Acoustic–image class-prototype similarity matrices.
Preprints 227477 g010
Table 1. Distribution of the acoustic dataset.
Table 1. Distribution of the acoustic dataset.
Class Training Validation Test Total
Pedestrian 39,261 5,106 4,762 49,129
Tracked vehicle 39,225 4,876 5,004 49,105
Wheeled vehicle 39,709 5,166 4,929 49,804
Total 118,195 15,148 14,695 148,038
Table 2. Distribution of the image dataset. Parentheses in the labeled mixed-source column report web and labeled UGS samples, respectively.
Table 2. Distribution of the image dataset. Parentheses in the labeled mixed-source column report web and labeled UGS samples, respectively.
Class Labeled Mixed-Source Unlabeled Target Target Validation Target Test Total
Pedestrian 40,613 (39,640 + 973) 2,919 6,015 5,839 55,386
Tracked vehicle 41,315 (40,240 + 1,075) 3,225 5,929 6,452 56,921
Wheeled vehicle 41,095 (40,168 + 927) 2,781 6,008 5,563 55,447
Total 123,023 (120,048 + 2,975) 8,925 17,952 17,854 167,754
Table 3. Configuration of the shared lightweight backbone.
Table 3. Configuration of the shared lightweight backbone.
Stage Input Channels Output Channels Stride Shortcut Output Description
Stem 1 32 1 No Initial feature map
B1 32 16 1 No Inverted residual
B2 16 16 1 Yes Inverted residual
B3 16 24 2 No Spatial downsampling
B4 24 24 1 Yes Inverted residual
B5 24 32 2 No Spatial downsampling
B6 32 32 1 Yes Inverted residual
B7 32 64 2 No Spatial downsampling
B8 64 64 1 Yes Inverted residual
B9 64 96 1 No High-level representation
B10 96 96 1 Yes Domain-alignment feature
Head 96 128 – – Global pooled embedding
Table 4. Principal training and method hyperparameters.
Table 4. Principal training and method hyperparameters.
Item Value Item Value
Optimizer Adam Training epochs 100
Learning rate 1 × 10 − 3 Batch size per stream 128
Adam coefficients ( 0.9 , 0.999 ) Weight decay 0
Embedding dimension 128 Main dropout 0.5
Prototype momentum 0.9 Stability momentum 0.9
Prototype stability floor 0.2 Distillation temperature 1.0
Maximum PKD weight 0.03 PKD warm-up 10 epochs
Maximum RW-MMD weight 0.03 RW-MMD activation Epoch 5
Maximum CW-CMMD weight 0.005 CW-CMMD activation Epoch 10
Pseudo-label threshold 0.8 Minimum class samples 2
BCMD retention probability 0.7 EMA decay 0.999
Table 7. Cross-modal prototype similarity under different distillation strategies.
Table 7. Cross-modal prototype similarity under different distillation strategies.
Strategy Same-Class Similarity Different-Class Similarity Similarity Margin Observation
No prototype distillation 0.974 0.542 0.432 High cross-class correlation
Fixed symmetric distillation 0.948 0.458 0.490 Moderate separation improvement
Dynamic bidirectional distillation 0.986 − 0.493 1.479 Strong alignment and separation
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.