Vision Transformer with Multi-Scale Dilated Convolution and Dual Attention Fusion for Robust Zooplankton Classification

1. Introduction

Marine ecosystems face unprecedented challenges from climate change, ocean acidification, and anthropogenic disturbances, making accurate ecological monitoring increasingly critical [1,2]. Zooplankton—microscopic organisms that form the foundation of marine food webs—serve as sensitive bioindicators of environmental perturbations, with their population dynamics reflecting water quality shifts, primary productivity changes, and broader ecosystem health [3,4]. Reliable zooplankton classification is therefore indispensable for harmful algal bloom prediction, fisheries management, carbon cycle research, and climate change impact assessment [5,6].

Traditional zooplankton identification relies on labor-intensive microscopic examination by expert taxonomists—a process plagued by inter-observer variability, subjective biases, and limited throughput that cannot meet the demands of large-scale, real-time monitoring programs [7,8]. The advent of in situ imaging platforms, including ZooScan, FlowCam, and the Underwater Vision Profiler, has generated vast repositories of plankton imagery, catalyzing the transition toward automated, high-throughput classification systems [9,10].

Deep learning has revolutionized automated image classification, with convolutional neural networks (CNNs) achieving remarkable success across diverse domains [11,12,13]. More recently, Vision Transformers (ViTs) have emerged as powerful alternatives, leveraging self-attention mechanisms to model long-range dependencies and achieving competitive performance on fine-grained recognition tasks [14,15,16,17]. However, direct application of these methods to zooplankton classification encounters fundamental challenges: (1) minute morphological distinctions between closely related taxa demand exceptional local feature discrimination; (2) zooplankton exhibit pronounced intra-species variations across developmental stages and preservation conditions; (3) underwater images frequently suffer from motion blur, partial occlusion, and non-uniform illumination; and (4) organism sizes span orders of magnitude, requiring robust multi-scale perception [18,19].

To overcome these challenges, we present ViT-MDFA (Vision Transformer with Multi-scale Dilated Convolution and Dual Attention Fusion), a novel architecture specifically designed for fine-grained zooplankton classification. Our key contributions are:

1.

Novel Architecture: We propose ViT-MDFA, a Vision Transformer enhanced with multi-scale dilated convolution and dual attention mechanisms, specifically tailored for zooplankton fine-grained classification.

2.

Multi-scale Dilated Convolution Module: We introduce the MSDC module, which captures local textures and global structures simultaneously through parallel dilated convolutions with optimally selected dilation rates, efficiently expanding the receptive field.

3.

Dual Attention Mechanism: We design a DA module that combines channel-wise and spatial attention, embedded via a layer-wise alternating insertion strategy to enhance feature representations at both shallow and deep network layers.

4.

Comprehensive Evaluation: We conduct extensive experiments on four zooplankton benchmarks, demonstrating state-of-the-art performance with rigorous ablation studies, sensitivity analyses, and interpretability visualizations.

The remainder of this paper is structured as follows: Section 2 reviews related work in zooplankton classification and Vision Transformers. Section 3 presents the ViT-MDFA architecture in detail. Section 4 reports experimental results and comprehensive analysis. Section 5 concludes the paper with future research directions.

2. Related Work

2.1. Automated Zooplankton Classification

The evolution of automated zooplankton classification reflects the broader trajectory of computer vision—from handcrafted feature engineering to end-to-end deep learning. Early approaches extracted morphological descriptors (area, perimeter, aspect ratio) and texture features (Haralick patterns, Fourier descriptors), then applied shallow classifiers such as support vector machines or random forests [20,21]. While these methods offered interpretability, their representational capacity proved inadequate for capturing complex morphological variations inherent in planktonic organisms.

The deep learning era brought substantial improvements. González et al. [22] pioneered the application of AlexNet and VGG to plankton classification, demonstrating the efficacy of transfer learning from ImageNet pre-training. Subsequent research explored deeper architectures including ResNet [23], DenseNet [24], and EfficientNet [25], each contributing incremental accuracy gains. Recent efforts have incorporated attention mechanisms and multi-scale fusion strategies to enhance discriminative capability [26,27]. Nevertheless, CNN-based approaches remain fundamentally constrained by their local receptive fields, limiting their capacity to model the long-range contextual relationships essential for distinguishing morphologically similar taxa.

2.2. Vision Transformers for Fine-Grained Recognition

The Vision Transformer reformulated image classification as sequence modeling, partitioning images into fixed-size patches and processing them through transformer encoder layers with self-attention [2]. DeiT and ViT-AUG demonstrated that with proper training strategies, ViTs could achieve competitive performance on ImageNet and downstream tasks [28,29]. Recent multimodal advances have further expanded ViT capabilities—Tang et al. [30] demonstrated harmonizing visual text comprehension and generation through unified architectures, while TextSquare scaled up text-centric visual instruction tuning for enhanced representation learning [31].

For fine-grained visual categorization (FGVC), researchers have proposed numerous ViT variants. TransFG introduced part-level attention modules and fine-grained loss functions [32]. RAMS-Trans incorporated recurrent attention with multi-scale token aggregation [33]. AA-Trans employed adaptive attention to emphasize discriminative regions [34]. SM-ViT integrated saliency maps for guided token selection [35]. More recently, INTR [36] and CLE-ViT [37] explored contrastive learning and local enhancement strategies. In marine organism recognition specifically, the Token-Selective Vision Transformer (TSVT) achieved state-of-the-art results by selectively attending to discriminative token regions, validating attention mechanisms for underwater imagery [16]. Despite these advances, existing methods frequently underperform on zooplankton benchmarks due to insufficient local feature modeling and inadequate multi-scale perception capabilities.

2.3. Multi-Scale and Attention Mechanisms

Multi-scale feature fusion has proven essential for recognizing objects at varying scales. Dilated (atrous) convolution expands receptive fields exponentially without resolution loss from pooling [38]. Yu et al. [39] demonstrated multi-scale dilated convolution for semantic segmentation, while parallel dilated convolutions with different rates capture complementary scale information in classification tasks [40,41]. Tang et al. [42] proposed random dilated convolution with multi-branch feature extraction, further establishing the effectiveness of multi-scale perception.

Attention mechanisms enable adaptive feature weighting. SE-Net pioneered channel-wise attention through global pooling and fully connected layers [43]. CBAM combined channel and spatial attention sequentially [44]. Within Vision Transformers, while attention is intrinsic to self-attention, additional learned attention modules can further refine feature selection [45,46,47]. Recent work on deformable attention transformers [47] and dynamic token pruning [17] demonstrates the versatility of attention mechanisms. Comprehensive benchmarks including OCR-Bench v2 [48] and MCTBench [49] have established rigorous evaluation protocols that inspire our experimental design.

Our work distinguishes itself through: (1) direct integration of multi-scale dilated convolution into the Vision Transformer pipeline, compensating for ViT’s inherent limitations in local feature modeling; (2) deployment of a dual attention mechanism with layer-wise alternating insertion optimized for fine-grained zooplankton recognition; and (3) the first comprehensive evaluation across four zooplankton benchmarks with rigorous ablation studies and sensitivity analyses.

3. Methodology

3.1. Architecture Overview

ViT-MDFA builds upon the standard Vision Transformer backbone while introducing two complementary enhancements: the Multi-scale Dilated Convolution (MSDC) module and the Dual Attention (DA) module. Figure 1 illustrates the overall architecture. Our design draws inspiration from efficient vision transformers [17,50] and multimodal representation learning advances [30,31].

Given an input image $\mathbf{X}\in {\mathbb{R}}^{H\times W\times C}$, we partition it into non-overlapping patches of size $P\times P$, yielding $N=(H/P)\times (W/P)$ patch tokens. Each patch is linearly projected to dimension D and combined with positional embeddings ${\mathbf{E}}_{\mathrm{pos}}$:

$${\mathbf{Z}}_0=[{\mathbf{x}}^1\mathbf{E};{\mathbf{x}}^2\mathbf{E};\dots ;{\mathbf{x}}^N\mathbf{E}]+{\mathbf{E}}_{\mathrm{pos}},$$

(1)

where $\mathbf{E}\in {\mathbb{R}}^{P^{2}C\times D}$ denotes the patch embedding matrix.

The encoded tokens ${\mathbf{Z}}_0$ are processed through L transformer encoder layers. Every three layers, we insert a DA module for adaptive feature refinement. Before entering the transformer stack, patch embeddings pass through the MSDC module to enhance local feature extraction.

3.2. Multi-Scale Dilated Convolution Module

The MSDC module captures local texture details and contextual information at multiple scales with minimal computational overhead. As illustrated in Figure 2, the module comprises three parallel dilated convolution branches with dilation rates $r\in \{6,12,18\}$, empirically determined through sensitivity analysis (Section 4.3). This multi-branch design extends the random dilated convolution principle with multi-branch feature extraction [42].

For each branch with dilation rate r, the operation is defined as:

$${\mathbf{F}}_r={\mathrm{Conv}2\mathrm{d}}_{k\times k,r}\left(\mathbf{X}\right)+\mathbf{X},$$

(2)

where ${\mathrm{Conv}2\mathrm{d}}_{k\times k,r}$ denotes a $k\times k$ dilated convolution with rate r, and the residual connection preserves low-level features.

Branch outputs are concatenated and fused via a $1\times 1$ convolution:

$${\mathbf{F}}_{\mathrm{MSDC}}={\mathrm{Conv}2\mathrm{d}}_{1\times 1}\left([{\mathbf{F}}_6;{\mathbf{F}}_{12};{\mathbf{F}}_{18}]\right).$$

(3)

This design simultaneously perceives fine textures (small r) and broader structural patterns (large r), particularly beneficial for zooplankton exhibiting diverse morphological scales.

3.3. Dual Attention Module

The DA module comprises two sequential sub-modules: Channel Attention (CA) and Spatial Attention (SA). Given an intermediate feature map $\mathbf{F}\in {\mathbb{R}}^{H^{\prime }\times W^{\prime }\times C^{\prime }}$, the CA sub-module computes channel-wise weights:

$${\mathbf{M}}_c=\sigma (\mathrm{MLP}\left(\mathrm{AvgPool}\left(\mathbf{F}\right)\right)+\mathrm{MLP}\left(\mathrm{MaxPool}\left(\mathbf{F}\right)\right)),$$

(4)

$${\mathbf{F}}_c={\mathbf{M}}_c\otimes \mathbf{F},$$

(5)

where $\sigma$ denotes the sigmoid function, ⊗ represents element-wise multiplication, and the MLP consists of two fully connected layers with reduction ratio 16. This architecture aligns with attention mechanisms employed in document understanding [51,52].

The SA sub-module subsequently processes ${\mathbf{F}}_c$:

$${\mathbf{M}}_s=\sigma \left({\mathrm{Conv}2\mathrm{d}}_{7\times 7}\left([\mathrm{AvgPool}\left({\mathbf{F}}_c\right);\mathrm{MaxPool}\left({\mathbf{F}}_c\right)]\right)\right),$$

(6)

$${\mathbf{F}}_{\mathrm{DA}}={\mathbf{M}}_s\otimes {\mathbf{F}}_c.$$

(7)

The alternating insertion strategy embeds DA modules after every three transformer encoder layers. This placement enables shallow layers to emphasize texture discrimination (via early DA modules) while deeper layers focus on semantic inter-class separation (via later DA modules). The strategy draws inspiration from token-selective attention mechanisms [16] and dynamic token pruning approaches [17].

3.4. Training Strategy

The model is trained end-to-end using cross-entropy loss:

$${\mathcal{L}}_{\mathrm{CE}}=-{\displaystyle \sum _{i=1}^{N_{\mathrm{cls}}}}{y}_{i}log\left({\widehat{y}}_i\right),$$

(8)

where $N_{\mathrm{cls}}$ denotes the number of classes, $y_i$ represents the ground-truth label, and ${\widehat{y}}_i$ is the predicted probability.

We employ standard data augmentation (random horizontal flip, random resize crop, color jitter) and regularization techniques (Dropout, weight decay) to mitigate overfitting. Optimization uses AdamW with cosine annealing learning rate scheduling, following best practices from visual instruction tuning research [31,49].

4. Experiments and Analysis

4.1. Datasets and Implementation Details

We evaluate ViT-MDFA on four zooplankton benchmarks:

WHOI-Plankton [53]: Contains 67,448 grayscale images spanning 14 plankton taxa, captured by underwater imaging systems.

ZooScanNet [54]: Comprises 85,231 color images from ZooScan instruments, covering 23 zooplankton categories.

Kaggle-Plankton [55]: A benchmark dataset with 55,494 plankton images across 70 classes, originating from the National Data Science Bowl competition.

Dec-22: Our self-constructed dataset collected in December 2022 from the East China Sea, containing 12,847 images across 18 local zooplankton species.

All images are resized to $224\times 224$ and normalized. We employ ViT-B/16 as the backbone, pre-trained on ImageNet-21k. Training utilizes batch size 64, initial learning rate $3\times {10}^{-4}$, and 100 epochs on NVIDIA A100 GPUs.

4.2. Comparison with State-of-the-Art Methods

4.2.1. Backbone Model Comparison

We compare ViT-MDFA against eight Transformer-based fine-grained classification methods: ViT-B/16 [2], TransFG [32], RAMS-Trans [33], AA-Trans [34], SM-ViT [35], IELT [56], CLE-ViT [37], and INTR [36].

Table 1 presents classification performance across all four datasets. ViT-MDFA achieves superior results on all metrics and datasets. Notably, on Dec-22, our model attains 97.46% accuracy and 96.73% F1-score, surpassing baseline ViT-B/16 by 6.14% and 7.79%, respectively. The performance gap widens on challenging datasets (WHOI-Plankton, ZooScanNet), demonstrating ViT-MDFA’s robustness to image quality variations and complex backgrounds. These results compare favorably with recent visual benchmarking studies [48,49,57,58].

4.2.2. Domain-Specific Method Comparison

We further compare against specialized zooplankton classification approaches (Table 2). On Kaggle-Plankton, ViT-MDFA achieves 96.14% accuracy, exceeding the previous best (Maracani et al. [59]) by 0.64%. On ZooScanNet, our method surpasses Zheng et al. [60] by 5.0%, demonstrating competitive advantages over domain-specific solutions. These results align with trends from comprehensive visual benchmarking studies [49,57].

4.3. Ablation Studies

4.3.1. Module Independence and Synergy

We quantify MSDC and DA module contributions through ablation experiments (Table 3). Baseline ViT-B/16 achieves 89.81% accuracy on WHOI-Plankton. Adding MSDC alone improves accuracy by 1.83%, while DA alone contributes 0.81%. Combined usage yields 2.46% improvement, indicating synergistic effects beyond additive contributions.

4.3.2. Attention Insertion Strategy

We evaluate alternative DA insertion patterns: (1) no insertion, (2) insertion after every layer, (3) alternating every 3 layers (our choice), and (4) insertion only in deep layers (9–12). The alternating strategy achieves optimal accuracy-efficiency trade-off, with 0.38% higher accuracy than uniform insertion and 1.2× faster inference. This finding aligns with dynamic token selection strategies [16,17].

4.3.3. Dilation Rate Sensitivity

We test dilation rate combinations $\left\{\right[3,6,9],[6,12,18],[9,18,27],[12,24,36\left]\right\}$. The [6,12,18] configuration yields optimal performance across all datasets, balancing local detail preservation and global context capture. Larger rates ([12,24,36]) introduce boundary artifacts, while smaller rates ([3,6,9]) provide insufficient receptive field expansion.

4.3.4. Scale Grouping Analysis

Organisms are categorized by size (small: <100 pixels, medium: 100–300 pixels, large: >300 pixels). Small targets benefit most from low dilation rates (superior local feature retention), while large targets gain from high dilation rates (enhanced global context). The [6,12,18] combination delivers balanced performance across all size groups.

4.4. Visualization and Interpretability

We employ Grad-CAM [63] to visualize attention heatmaps (Figure 3). Compared to baseline ViT, ViT-MDFA produces more focused and stable attention on diagnostically relevant morphological features (e.g., antennae, body segments, caudal spines). This enhanced localization validates that the DA mechanism successfully guides the model toward species-discriminative regions.

Attention weight distribution analysis across transformer layers reveals that shallow layers (1–4) exhibit diffuse attention patterns, while deeper layers (9–12) concentrate on key structures. The alternating DA insertion amplifies this layer-wise specializatio Attention weight distribution analysis across transformer layers reveals that shallow layers (1–4) exhibit diffuse attention patterns, while deeper layers (9–12) concentrate on key structures. The alternating DA insertion amplifies this layer-wise specialization. These visualization techniques follow established practices in multimodal model interpretation [30,52].

5. Conclusions

This paper presents ViT-MDFA, a novel Vision Transformer architecture enhanced with multi-scale dilated convolution and dual attention mechanisms for fine-grained zooplankton classification. The MSDC module addresses ViT’s inherent limitations in local feature modeling by capturing multi-scale texture and structural information through parallel dilated convolutions. The DA module, deployed via an alternating insertion strategy, refines feature representations at both shallow and deep network layers.

Comprehensive experiments across four challenging benchmarks demonstrate state-of-the-art performance, with accuracy improvements of 3–6% over strong baselines. Ablation studies validate the individual and synergistic contributions of proposed modules. Visualization analysis confirms enhanced attention focus on biologically meaningful morphological structures.

Future research directions include: (1) extending ViT-MDFA to video-based zooplankton tracking for temporal dynamics analysis; (2) integrating unsupervised domain adaptation for cross-instrument generalization; and (3) deploying on edge devices for real-time in situ classification. Building on recent advances in document understanding [51,52] and multimodal benchmarks [48,49], we plan to develop comprehensive evaluation frameworks for marine vision systems.