Enhancing Early Skin Cancer Detection: A Deep Learning Approach with Multi-Scale Feature Refinement and Fusion

1. Introduction

In recent years, environmental pollution and increased ultraviolet radiation have contributed to a significant rise in the incidence of skin cancers such as melanoma, drawing considerable attention from the global medical community [1,2]. Early diagnosis is crucial for improving treatment outcomes and patient survival rates [2,3]. Traditional diagnostic methods, which primarily rely on clinical observation and tissue biopsy, are subjective, invasive, and unsuitable for large-scale screening [4,5]. Medical image segmentation technology offers a non-invasive, high-precision alternative for skin cancer analysis, providing clinicians with richer and more accurate diagnostic information [6,7] and holding promise for breakthroughs in early detection and treatment.

Automatic segmentation of skin lesions plays a vital role in computer-aided diagnosis systems for melanoma [6,7]. Data from 2022 reveal significant geographical disparities in skin cancer incidence, with Australia reporting 37 cases per 100,000 people, China reporting only 0.37 cases, and a global average of 3.2 cases [1,2]. These variations highlight the need for adaptable diagnostic tools that can perform effectively across diverse populations and clinical settings. Public datasets such as ISIC have become critical resources for developing and validating such technologies, enabling researchers to train robust models that generalize across different demographic and geographic contexts [14,15].

Dermoscopy, a non-invasive magnification imaging technique, has emerged as a key tool in melanoma diagnosis [4,5]. Studies indicate that early diagnosis can achieve 10-year survival rates of 84% to 98%, whereas late-stage diagnosis results in only 10% to 15% survival [3]. However, manual interpretation of dermoscopic images is time-consuming and labor-intensive, underscoring the necessity of automatic segmentation technology to enhance diagnostic accuracy and efficiency [6,7]. Automatic skin lesion segmentation aims to perform pixel-level annotation of lesion areas in dermoscopic images. However, this task faces substantial challenges, including variations in skin color and lesion size, texture diversity, and artifacts such as shadows and body hair [6,9]. Early segmentation techniques, such as thresholding and color space analysis, relied on predefined feature extraction and often failed to handle the variability in lesion appearance and interference from external artifacts [8,9].

Against the backdrop of the continuously rising global incidence of skin cancer, achieving precise early segmentation of malignant skin tumors such as melanoma has become critical for enhancing quantitative analysis accuracy, assisting early clinical diagnosis, and optimizing treatment strategies. However, existing segmentation methods face the fundamental challenge of balancing computational efficiency with feature capture capability, limiting their effectiveness in practical clinical applications.Traditional methods based on threshold segmentation, edge detection, or color-texture feature extraction can achieve preliminary lesion identification [8]. However, they are highly dependent on image quality and struggle to handle interfering factors such as lesion morphological diversity, blurred boundaries, and artifacts (e.g., shadows, hair follicles) [6,9]. Although deep learning models such as U-Net have significantly improved segmentation performance, they still exhibit limitations in multi-scale feature fusion, long-range contextual modeling, and detail recovery [10,11]. These limitations are particularly evident in cases involving discontinuous boundaries or small lesions, where under-segmentation and mis-segmentation frequently occur [11].

To address these shortcomings, this paper proposes a novel deep learning segmentation framework that integrates a Pyramid Vision Transformer v2 (PVT v2) backbone with two collaborative modules—Spatial-Aware Feature Enhancement (SAFE) and Multi-scale Dual Cross-attention Fusion (MDCF) to enhance feature representation and improve decoding reconstruction [12,13]. As illustrated in Figure 1, we benchmarked our proposed method against current state-of-the-art approaches on the ISIC 2016 and ISIC 2018 datasets. The results demonstrate notable enhancements in both the IOU and DICE metrics. The core advantages of this method are reflected in three key aspects:

First, the PVT v2 encoder leverages its pyramidal hierarchical structure to effectively extract lesion features at multiple scales [13]. This multiscale representation enables the model to adapt to lesions of varying sizes and shapes, from small isolated spots to large irregular regions, thereby improving robustness across diverse clinical scenarios.

Second, the SAFE module refines encoder features at each resolution level through a dual-branch architecture that jointly leverages shallow and deep representations. The base-feature branch preserves fine-grained spatial details from shallow layers, while the offset-compensation branch employs dilated convolutions to predict adaptive offsets that align shallow features with deep semantic information. This mechanism effectively bridges the feature discrepancy across network depths, improving the discriminability and completeness of multi-level feature representation.

Third, the MDCF module establishes bidirectional cross-attention between decoder and SAFE-enhanced encoder features to enable dynamic semantic information transfer via skip connections. It employs a four-branch multi-scale deformable convolution unit to capture lesion boundaries and small fragments at heterogeneous receptive fields. This design significantly improves boundary segmentation accuracy and fine structure recovery while suppressing background responses. In summary, the main contributions of this paper are as follows:

• Novel Architecture with PVT v2 Backbone: We propose a new deep learning framework for automatic skin lesion segmentation that incorporates the Pyramid Vision Transformer v2 (PVT v2) as the core encoder. This design effectively captures multi-scale contextual features from dermoscopic images, enhancing the model’s ability to recognize lesions of varying sizes and complexities [13].
• Dual-Module Cooperative Mechanism: The framework introduces a synergistic dual-module mechanism within the decoder pathway. The first module is dedicated to the sophisticated integration of low-level spatial details and high-level semantic features, ensuring a more discriminative and complete feature representation. The second module employs an enhanced skip-connection strategy for dynamic feature fusion between the encoder and decoder, significantly improving the precision of boundary delineation and the recovery of fine-grained structures [10,12].
• Comprehensive Evaluation on Benchmark Datasets: We conduct extensive experiments on publicly available datasets, such as ISIC 2016 and SISC 2018. The results demonstrate that our proposed method achieves state-of-the-art performance, particularly in handling boundary ambiguity and small lesions, outperforming existing leading approaches in terms of standard segmentation metrics like Dice Coefficient and IOU. This validates the practical effectiveness and robustness of our model in a clinical simulation setting.

2. Related Work

2.1. CNNs for Skin Lesion Segmentation

Convolutional Neural Networks (CNNs) have served as the cornerstone and dominant paradigm in automated skin lesion segmentation, revolutionizing the field through their ability to automatically learn hierarchical features from raw pixel data. The evolution of CNN-based approaches can be largely traced through the adoption and refinement of encoder-decoder architectures [9,10,29,30].

Among these, the U-Net architecture [10] has arguably been the most influential model in medical image segmentation, including skin lesion analysis. Its symmetric encoder-decoder structure, combined with skip connections, effectively addresses the challenge of integrating high-resolution spatial information from the encoder pathway with the high-level semantic information recovered during decoding. Skip connections are particularly critical for preserving fine-grained details along lesion boundaries, which are often irregular and poorly defined. Following U-Net, variants such as SegNet [16]—which utilizes pooling indices for upsampling in the decoder—and other encoder-decoder frameworks have been widely applied, cementing this architectural paradigm as the standard backbone for the task.

Building upon the encoder-decoder foundation, researchers have introduced various modifications to tackle challenges specific to dermoscopic images. While architectures like U-Net set the standard under full supervision, later work has sought to reduce dependence on costly pixel-level annotations. Weakly supervised methods, for instance, have attracted growing interest. A notable contribution by Anggasta Aji Azhari et al. [17] introduced a two-stage CNN that integrates Grad-CAM for weak localization to refine feature learning. This marks a conceptual shift: rather than using explainability tools only for post-hoc analysis, they are embedded directly into training to improve model confidence and interpretability—bridging high discriminative performance with clinical practicality. Beyond supervision strategies, a key challenge has been enhancing the ability of CNNs to capture long-range contextual information while preserving local detail. While recent efforts have incorporated self-attention mechanisms like Transformers, an emerging direction involves state space models. In this vein, BEFNet [20] employs a Mamba-based encoder to capture global context, while retaining a dedicated CNN branch to preserve local spatial precision—ensuring that critical boundary details are not lost. This reflects a design philosophy of complementary integration, where CNNs play a specialized role alongside global modeling approaches. Furthermore, the computational cost of deep models remains a barrier to clinical deployment. To address efficiency, Chun et al. [21] proposed UCM-Net, which combines MLPs with CNNs in a lightweight architecture requiring under 50KB parameters and 0.05 GLOPS, yet maintains competitive segmentation accuracy. This makes it a promising benchmark for resource-conscious applications in dermatology.

Another significant challenge in skin lesion segmentation is the severe class imbalance between small lesion regions and extensive background areas. To mitigate this issue, loss functions such as Dice Loss [18] and Focal Loss [19] have been widely adopted as alternatives to standard cross-entropy. These losses explicitly encourage the model to focus on underrepresented lesion pixels, thereby improving segmentation accuracy for the target structures.

2.2. Transformers for Skin Lesion Segmentation

With the rapid advancement of Vision Transformers (ViTs) [33], transformer-based architectures have emerged as powerful alternatives to traditional CNNs in medical image segmentation. Unlike convolutional networks that inherently focus on local spatial relationships, Transformers model long-range dependencies through self-attention, enabling more comprehensive global context understanding—an ability particularly beneficial for skin lesion segmentation, where lesions often present irregular textures and ambiguous boundaries. This advantage has been validated by the Medical Transformer [34], which introduces gated axial-attention to efficiently capture long-range dependencies and has demonstrated strong performance across multiple medical image segmentation tasks.

Hybrid architectures such as TransUNet [12] combine CNN encoders with transformer blocks to incorporate both local feature extraction and global dependency modeling, resulting in improved performance on medical imaging benchmarks. Similarly, hierarchical transformer models like Swin-UNet [22], built upon the shifted-window self-attention mechanism originally proposed in Swin Transformer [35], achieve efficient global modeling while maintaining linear computational complexity with respect to image size.

Pyramid-style transformers further push the performance for dense prediction tasks. The Pyramid Vision Transformer (PVT) and its improved variant PVT v2 [13] utilize spatial-reduction attention and multi-scale feature hierarchies, addressing computational bottlenecks while preserving global contextual cues—making them well-suited for high-resolution dermoscopic images. Meanwhile, segmentation frameworks such as SegFormer [23] adopt lightweight pyramid Transformers with MLP decoders, demonstrating robust generalization across diverse medical datasets.

Recent studies also incorporate cross-attention modules in the decoder to enhance dynamic feature fusion, improving boundary localization and facilitating better integration of multi-level semantic features. Overall, transformer-based approaches significantly outperform conventional CNNs in capturing long-range structure and resolving fuzzy lesion boundaries, and have thus become a key trend in skin lesion segmentation.

3. Materials and Methods

3.1. Architectural Overview

As illustrated in Figure 2, The decoder reconstructs a high-resolution segmentation map from these hierarchical features through a multi-stage refinement process. Starting from the deepest encoder output (Stage 4), feature maps are progressively upsampled and refined at each decoder level. At each resolution, the decoder features are enriched through two complementary mechanisms.

Each lateral connection from the encoder is processed by a SAFE module, which employs a dual-branch architecture to bridge the semantic gap between encoder and decoder features. The base-feature branch preserves fine-grained spatial details from shallow encoder layers, while the offset-compensation branch uses dilated convolutions and global context to predict adaptive spatial offsets, aligning shallow features with deep semantic information before fusion. After receiving SAFE-enhanced encoder features, each decoder stage applies the MDCF module to establish bidirectional cross-attention between encoder and decoder representations. This is followed by a four-branch multi-scale deformable convolution unit that captures lesion boundaries and small fragments at heterogeneous receptive fields, enabling adaptive feature aggregation while suppressing background responses.

A series of convolutional refinement layers and a lightweight prediction head convert the final high-resolution features into a pixel-wise probability map. This design synergistically combines the Transformer’s global context modeling with SAFE-guided multi-scale fusion and MDCF-based adaptive aggregation to preserve fine structures while maintaining robust semantic consistency.

3.2. Encoder Model

The proposed network adopts a hierarchical Transformer-based encoder to extract multi-scale representations from the input dermoscopic image. As illustrated in Figure 2, the encoder is organized into four stages. Each stage consists of a patch embedding layer followed by a stack of Transformer blocks, and produces a feature map with reduced spatial resolution but increased channel dimension. The resulting feature maps are later used both to drive the decoder and as inputs to the SAFE modules.

At stage i, the input feature map $X_{i-1}\in {\mathbb{R}}^{H_{i-1}\times W_{i-1}\times C_{i-1}}$ is first processed by a linear projection implemented as a convolutional patch embedding layer. This layer uses a kernel size and stride larger than 1 to merge neighboring pixels into non-overlapping (or slightly overlapping) patches and to down-sample the spatial resolution. A layer normalization is then applied, and the 2-D feature map is reshaped into a sequence of $N_i=H_i{W}_i$ tokens of dimension $C_i$. To preserve spatial layout information, we add a learnable 2-D positional embedding to the token sequence before feeding it to the Transformer blocks.

Each Transformer block follows a pre-norm architecture and is equipped with spatial-reduction attention (SRA) to capture long-range dependencies with manageable computational cost. Specifically, the input tokens are first normalized and then fed into an SRA module. In SRA, the query tokens are kept at full resolution, while the key and value tokens are obtained from a spatially down-sampled version of the feature map, enabling global receptive fields with reduced quadratic complexity. The output of the multi-head attention is added to the input via a residual connection. A second normalization layer followed by a position-wise feed-forward network (two linear layers with a non-linearity in between) is then applied, again with a residual connection. Finally, the token sequence is reshaped back to a 2-D feature map to serve as input to the next encoder stage. This process can be expressed by the following formula:

$$\begin{array}{cc}\hfill & T_0^{\left(i\right)}=\mathrm{Reshape}\left(\mathrm{Norm}\left(\mathrm{Linear}\left(X^{\left(i\right)}\right)\right)\right)\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill & T_{l+{\displaystyle \frac{1}{2}}}^{\left(i\right)}=T_0^{\left(i\right)}+{MHA}_{\mathrm{SRA}}\left(\mathrm{Norm}\left(T_0^{\left(i\right)}\right)\right)\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill & T_{l+1}^{\left(i\right)}=T_{l+{\displaystyle \frac{1}{2}}}^{\left(i\right)}+FFN\left(\mathrm{Norm}\left(T_{l+{\displaystyle \frac{1}{2}}}^{\left(i\right)}\right)\right)\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill & Y^{\left(i\right)}={\mathrm{Reshape}}^{-1}\left(T_{l+1}^{\left(i\right)}\right)\hfill \end{array}$$

(4)

where $X^{\left(i\right)}$ denotes the input feature map of the i-th stage, $T_0^{\left(i\right)}$ is the token sequence obtained by linearly projecting, normalizing and reshaping $X^{\left(i\right)}$, $T_{l+{\displaystyle \frac{1}{2}}}^{\left(i\right)}$ and $T_{l+1}^{\left(i\right)}$ are the intermediate features after spatial-reduction multi-head self-attention and the subsequent feed-forward network at the l-th layer, respectively, $Y^{\left(i\right)}$ is the output feature map of this stage, $Norm(·)$ denotes layer normalization, $MH{A}_{\mathrm{SRA}}(·)$ denotes multi-head self-attention with spatial reduction, $FFN(·)$ denotes a position-wise feed-forward network, $Reshape(·)$ and $Reshap{e}^{-1}(·)$ convert between 2D feature maps and 1D token sequences.

The encoder outputs from all stages are forwarded to the decoder through SAFE blocks. For a given stage, the corresponding feature map is fed into a SAFE block that exploits both global and local cues. A $k\times k$ dilated convolution is first applied to generate an offset-enriched representation capturing a larger receptive field. In parallel, the global features of the same stage are processed by a weight-shared $1\times 1$ convolution. The two branches are fused and passed through another $1\times 1$ convolution followed by a sigmoid activation to produce a spatial attention map. This attention map is then multiplied element-wise with the original stage feature map, yielding an enhanced representation that emphasizes lesion-related regions while suppressing irrelevant background responses. These SAFE-refined features at multiple scales provide rich contextual information for the subsequent decoder and segmentation head.

3.3. SAFE Model

The SAFE module is designed to refine the encoder features at each resolution level by jointly leveraging shallow-layer and deep-layer representations, while explicitly suppressing background responses. As shown in Figure 2, one SAFE block is attached to the output of each encoder stage, and the resulting enhanced features are forwarded to the decoder.

Given an input feature map from stage (i), denoted as ${\mathbf{F}}_i\in {\mathbb{R}}^{H_i\times W_i\times C_i}$, SAFE first prepares the features and then computes a stage-specific attention map. To reduce computational cost and stabilize training, ${\mathbf{F}}_i$ is optionally passed through a $1\times 1$ convolution and a normalization layer to align the channel dimension. The processed feature map is then fed into two parallel branches: a base-feature branch and an offset-compensation branch. The base-feature branch preserves the shallow representation at stage (i), while the offset-compensation branch predicts offsets to compensate for the discrepancy between the shallow features and the corresponding deep features.

In the base-feature branch, a $1\times 1$ convolution is applied to ${\mathbf{F}}_i$ to obtain a refined shallow representation ${\mathbf{F}}_i^{\mathrm{base}}$. This projection mainly adjusts the channel dimension and preserves the stage-(i) content, so that low-level cues and fine details are maintained in a compact form. A normalization layer and a non-linear activation are used to stabilize training and further refine ${\mathbf{F}}_i^{\mathrm{base}}$, providing a clean base feature for subsequent fusion with deeper layers.

In the offset-compensation branch, SAFE estimates a bias field that aligns the shallow features with their deep counterparts by jointly capturing local context and global statistics. First, a $k\times k$ dilated convolution is applied on ${\mathbf{F}}_i$ to obtain a locally enriched representation ${\mathbf{F}}_i^{\mathrm{loc}}$. The dilation rate is chosen according to the spatial scale of the current stage, so that shallow stages focus on fine details while deeper stages capture larger structures. This operation increases the effective receptive field without reducing resolution, thereby aggregating neighboring cues that are important for lesion boundaries and texture. A $1\times 1$ convolution and activation function then project ${\mathbf{g}}_i$ into an offset vector that is broadcast back to the spatial size $H_i\times W_i$ to produce an offset map ${\mathbf{F}}_i^{\mathrm{off}}$. This offset map is used to modulate the base features, providing a learned bias that compensates for the discrepancy between shallow and deep features.

The outputs of the two branches are then fused to generate a stage-aware representation. Specifically, the offset-compensation branch output ${\mathbf{F}}_i^{\mathrm{off}}$ is first passed through a non-linear activation function and used to modulate the base features via element-wise multiplication, yielding an intermediate feature ${\mathbf{F}}_i^{\mathrm{mod}}=\sigma \left({\mathbf{F}}_i^{\mathrm{off}}\right)\odot {\mathbf{F}}_i^{\mathrm{base}}$. This modulated feature is then added back to the original offset feature in a residual manner, i.e., ${\mathbf{F}}_i^{\mathrm{res}}={\mathbf{F}}_i^{\mathrm{off}}+{\mathbf{F}}_i^{\mathrm{mod}}$, and fed into the ShareadSA module to produce the final stage-aware output ${\widehat{\mathbf{F}}}_i=\mathrm{ShareadSA}\left({\mathbf{F}}_i^{\mathrm{res}}\right)$. This fusion strategy enables the learned offsets to gate the shallow base features while preserving offset information, leading to stage-specific refinement for lesion segmentation.

$${\mathbf{F}}_i^{\mathrm{off}}={\mathrm{Conv}}_{1\times 1}\left({\mathrm{DilatedConv}}_{k\times k}\left({\mathbf{F}}_i\right)\right)$$

(5)

$${\mathbf{F}}_i^{\mathrm{base}}={\mathrm{Conv}}_{1\times 1}\left({\mathbf{F}}_i\right)$$

(6)

$${\widehat{\mathbf{F}}}_i=\mathrm{ShareadSA}\left((\sigma \left({\mathbf{F}}_i^{\mathrm{off}}\right)\odot {\mathbf{F}}_i^{\mathrm{base}}+{\mathbf{F}}_i^{\mathrm{base}})\right)$$

(7)

where ${\mathbf{F}}_i$ denotes the input feature map at stage i, $\mathbf{F}{i}^{\mathrm{off}}$ and $\mathbf{F}{i}^{\mathrm{base}}$ denote the features of the offset branch and the base branch, respectively; $\mathrm{DilatedConv}k\times k(·)$ and $\mathrm{Conv}1\times 1(·)$ denote a $k\times k$ dilated convolution and a $1\times 1$ convolution, respectively; $\sigma (·)$ denotes the sigmoid activation function, ⊙ denotes element-wise multiplication, and $\mathrm{ShareadSA}(·)$ denotes the shared stage-aware attention module.

3.4. Decoder Model

The decoder adopts a cascaded architecture that alternates MDCF blocks with Upsampling blocks, progressively transforming coarse high-level representations into fine-grained, high-resolution predictions. Three MDCF modules are arranged in sequence, each followed by an Upsampling operation (except at the finest scale). At every stage, the MDCF block receives the current decoder feature and its scale-consistent SAFE-enhanced encoder feature, so that spatial resolution is increased while scale-specific encoder information is injected throughout the decoding process.

At the i-th decoding stage, we denote the input decoder feature as ${\mathbf{X}}_i$ and the corresponding SAFE-refined encoder feature as ${\tilde{\mathbf{F}}}_i$. The MDCF block first establishes a bidirectional interaction between these two feature streams via a cross-attention mechanism, as illustrated in Figure 2. Specifically, we compute two directional cross-attention responses,

$${\mathbf{Z}}_{d\leftarrow e}=\mathrm{CA}({\mathbf{X}}_i,{\tilde{\mathbf{F}}}_i),{\mathbf{Z}}_{e\leftarrow d}=\mathrm{CA}({\tilde{\mathbf{F}}}_i,{\mathbf{X}}_i),$$

(8)

and then fuse them by a $1\times 1$ convolution:

$$\mathbf{X}={\mathrm{Conv}}_{1\times 1}\left(\mathrm{Concat}({\mathbf{Z}}_{d\leftarrow e},{\mathbf{Z}}_{e\leftarrow d})\right).$$

(9)

Here, $\mathrm{CA}({\mathrm{feature}}_1,{\mathrm{feature}}_2)$ denotes a cross-attention operator that treats `feature_1` as the query and `feature_2` as the source of keys and values. In a standard implementation,

$$\mathrm{CA}(\mathbf{A},\mathbf{B})=\mathrm{softmax}\left({\displaystyle \frac{\left(W_q\mathbf{A}\right){\left(W_k\mathbf{B}\right)}^{\top }}{\sqrt{d}}}\right)W_v\mathbf{B},$$

(10)

where $W_q$, $W_k$, and $W_v$ are learnable projections, and d is the feature dimension. Thus, $\mathrm{CA}({\mathbf{X}}_i,{\tilde{\mathbf{F}}}_i)$ lets each location in the decoder feature selectively aggregate complementary information from the encoder feature, while $\mathrm{CA}({\tilde{\mathbf{F}}}_i,{\mathbf{X}}_i)$ performs the reverse aggregation. This bidirectional design encourages the decoder to focus on encoder regions that are most informative for lesion refinement and to suppress background responses persisting after encoding. The subsequent $1\times 1$ convolution aligns the channel dimension and mixes the information coming from both directions.

As illustrated in Figure 2, the fused feature $\mathbf{X}$ is then refined by a four-branch multi-scale deformable unit. First, $\mathbf{X}$ is evenly split along the channel dimension:

$${\mathbf{x}}_1,{\mathbf{x}}_2,{\mathbf{x}}_3,{\mathbf{x}}_4=\mathrm{Split}\left(\mathbf{X}\right),$$

(11)

where, if $\mathbf{X}$ has C channels, each sub-feature ${\mathbf{x}}_j$ contains $C/4$ channels. These four groups are processed in parallel by heterogeneous convolutions:

$${\mathbf{y}}_1={\mathrm{DWConv}}_{1\times 1}\left({\mathbf{x}}_1\right),$$

(12)

$${\mathbf{y}}_2={\mathrm{DeConv}}_{3\times 3}^{\left(1\right)}\left({\mathbf{x}}_2\right),$$

(13)

$${\mathbf{y}}_3={\mathrm{DeConv}}_{3\times 3}^{\left(2\right)}\left({\mathbf{x}}_3\right),$$

(14)

$${\mathbf{y}}_4={\mathrm{DeConv}}_{3\times 3}^{\left(3\right)}\left({\mathbf{x}}_4\right).$$

(15)

The first branch applies a depthwise $1\times 1$ convolution to perform lightweight channel-wise filtering, while the remaining three branches adopt deformable convolutions with different dilation settings (or offset patterns). As a result, the three deformable branches possess different effective receptive fields and jointly form a multi-scale detail-enhancement module that is able to capture both thin boundaries and small lesion fragments. The outputs of all branches are concatenated and fused with the original decoder feature via a residual connection:

$${\mathbf{F}}_i^{\mathrm{mdcf}}={\mathbf{X}}_i+\mathrm{Concat}({\mathbf{y}}_1,{\mathbf{y}}_2,{\mathbf{y}}_3,{\mathbf{y}}_4).$$

(16)

In this formulation, ${\mathbf{F}}_i^{\mathrm{mdcf}}$ denotes the output of the MDCF block; ${\mathbf{X}}_i$ is the input decoder feature; ${\tilde{\mathbf{F}}}_i$ is the SAFE-refined encoder feature; ${\mathrm{DWConv}}_{1\times 1}$ is a depthwise $1\times 1$ convolution; ${\mathrm{DeConv}}_{3\times 3}^{\left(i\right)}$ is the i-th $3\times 3$ deformable convolution branch (not a transposed convolution); and $\mathrm{Split}(·)$ and $\mathrm{Concat}(·)$ denote channel-wise splitting and concatenation, respectively.

The Upsampling block that follows each MDCF stage increases the spatial resolution of the decoder feature and prepares it for the next, finer decoding level. Concretely, the MDCF output ${\mathbf{F}}_i^{\mathrm{mdcf}}$ is first passed through a $1\times 1$ convolution to adjust the number of channels and reduce redundancy. The resulting feature map is then upsampled by a factor of 2, either using a learnable deconvolution or an interpolation-plus-convolution scheme, yielding the input feature for the subsequent decoding stage.

4. Results

4.1. Datasets

To comprehensively evaluate the performance of the proposed model, we conducted a series of experiments using two publicly accessible skin lesion segmentation datasets from ISIC (International Skin Imaging Collaboration), which originated from the "Skin Lesion Analysis Toward Melanoma Detection" challenge. These two datasets are ISIC-2016 [14] and ISIC-2018 [15], respectively. The images were meticulously annotated by professional experts with absolutely accurate labels. Specifically, 900 images were designated as training data, while 350 images were allocated for validation in the ISIC-2016 dataset [14]. The ISIC-2018 dataset [15] comprises 2,594 images, which were randomly divided into training, validation, and test sets at a ratio of 8:1:1.

4.2. Evaluation Metrics

To evaluate the effectiveness of our method in skin lesion segmentation, the primary evaluation metrics adopted in this paper include the Dice coefficient (also known as Dice Similarity Coefficient, DSC), Intersection over Union (IoU), Sensitivity, Specificity, and Accuracy. The Dice Coefficient is a statistical metric employed to measure the similarity between two sample sets, which has been extensively applied in medical image segmentation:

$$DC={\displaystyle \frac{2\times TP}{2\times TP+FP+FN}}$$

(17)

The IoU metric quantifies the extent of overlap between the predicted segmentation region and the ground truth annotation:

$$IoU={\displaystyle \frac{TP}{TP+FP+FN}}$$

(18)

The Accuracy (ACC) metric quantifies the proportion of correctly classified instances, encompassing both positive and negative predictions. The Sensitivity (SEN) assesses the proportion of actual positive cases that are correctly identified.

$$ACC={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(19)

$$SEN={\displaystyle \frac{TP}{TP+FN}}$$

(20)

Where TP (True Positive) represents the number of correctly predicted positive cases, TN (True Negative) represents the number of correctly predicted negative cases, FP (False Positive) indicates the number of incorrectly predicted positive cases, and FN (False Negative) represents the number of incorrectly predicted negative cases. All metrics are dimensionless, providing a reliable evaluation of the model’s segmentation performance.

4.3. Implementation Details

All experiments in this paper are conducted using the PyTorch 1.12.1 framework (with CUDA 11.3 support). The experiments are carried out on a server running Ubuntu 18.04.5 LTS, equipped with two Intel Xeon Gold 6230 CPUs , ten NVIDIA GeForce RTX 3090 GPUs, and a 2 TB solid-state drive.

For all experiments involving Our model, input images are resized to $352\times 352$ pixels before being fed into the network. We employ the AdamW optimizer for model training, with the initial learning rate and weight decay both set to 1e-4. The network is trained for 200 epochs with a batch size of 8. A step-wise learning rate decay strategy is adopted, where the learning rate is multiplied by a factor of 0.1 every 200 epochs to ensure stable convergence.

To enhance the generalization capability of the model, data augmentation techniques such as random rotation and random flipping are utilized. The loss function is defined as a weighted combination of Binary Cross-Entropy (BCE) loss and Intersection over Union (IoU) loss (formulated here using Dice for structural supervision). The total objective function is defined as follows:

$$\mathcal{L}=\alpha ·{\mathcal{L}}_{CE}+\beta ·{\mathcal{L}}_{Dice}$$

(21)

where $\alpha$ and $\beta$ represent the weighting coefficients. To determine the values of the weighting coefficients $\alpha$ and $\beta$, we performed a grid search by varying both parameters from 0 to 1 in increments of 0.1. The configuration yielding the best performance was $\alpha$ = 0.5 and $\beta$ = 0.5, which was subsequently adopted for our experiments. To ensure a fair comparison with other baseline models, we followed their officially recommended training protocols and hyperparameters. The experiments primarily focus on the ISIC-2016 and ISIC-2018 datasets.

4.4. Ablation Study

To understand the importance of each component in our model, we tested different versions of our model on the ISIC 2016 dataset [14]. We measured performance using four metrics: Accuracy (ACC), Intersection over Union (IOU), Dice coefficient (DC), and Sensitivity (SEN). The results of the structural ablation experiments are presented in Table 1. The experimental results demonstrate that the Baseline model performs reasonably well in lesion detection, achieving an IOU of 86.9% and a DC of 82.6% on the ISIC 2016 dataset. However, its representation capability for lesion regions is insufficient. When we introduce PVT v2 as the backbone network, the model performance shows noticeable enhancement across all metrics. Specifically, the accuracy increases from 95.9% to 96.3%, the IOU improves from 86.9% to 87.0%, the Dice coefficient rises from 82.6% to 83.1%, and the sensitivity advances from 91.2% to 91.5%. These improvements indicate that the PVT v2 backbone network is more capable of extracting multi-scale features, thereby enhancing the model’s representation ability for lesion regions.

Further incorporating the SAFE module on top of the PVT v2 backbone yields additional performance gains. The model achieves an accuracy of 96.6%, an IOU of 87.2%, a Dice coefficient of 93.6%, and a sensitivity of 91.8%. The SAFE module is specifically designed to improve encoder features at each resolution level by jointly leveraging both shallow and deep representations. By explicitly suppressing background responses while enhancing foreground lesion features, the SAFE module addresses one of the key challenges in skin lesion segmentation where background regions often exhibit similar visual characteristics to lesion areas. The integration of multi-level features enables the model to capture both fine-grained details from shallow layers and high-level semantic information from deep layers, resulting in more discriminative feature representations. This synergistic combination of shallow and deep features, coupled with explicit background suppression, leads to more precise localization of lesion boundaries and improved differentiation between lesion and non-lesion regions. As illustrated in Figure 3, the model equipped with SAFE demonstrates superior capability in capturing lesion regions within the images.

The complete model, which combines the PVT v2 backbone, SAFE module, and MDCF module, achieves the best performance across all evaluation metrics. It attains an accuracy of 97.0%, an IOU of 97.3%, a Dice coefficient of 94.1%, and a sensitivity of 92.1%. The MDCF module, through its multi-scale dense connection fusion mechanism, effectively integrates feature information from different hierarchical levels, further enhancing the model’s capability to segment lesions with complex morphologies.

The ablation study conclusively demonstrates that each proposed component contributes meaningfully to the overall performance improvement. The progressive introduction of the PVT v2 backbone, SAFE module, and MDCF module results in incremental performance gains, validating the rationality of our design strategy. Moreover, the synergistic effects among these modules collectively drive the advancement of skin lesion segmentation performance, highlighting the effectiveness of our comprehensive architectural design.

4.5. Comparison with the State-of-the-Art Methods

To demonstrate the effectiveness of our proposed method, we conducted comprehensive comparative experiments against several state-of-the-art segmentation approaches on the ISIC-2016 [14] and ISIC-2018 [15] benchmark datasets. The compared methods include U-Net [10], Swin-Unet [22], TransUNet [12], H2Former [26], SUnet [27], and MSDUnet [28], which represent both classical CNN-based architectures and recent transformer-based models. All models were trained under identical experimental settings to ensure fair comparison, including the same data augmentation strategies, optimization parameters, and evaluation protocols.

4.5.1. Quantitative Evaluation

Table 2 presents the quantitative comparison results on the ISIC-2016 dataset across four widely-used evaluation metrics: Accuracy (ACC), Intersection over Union (IOU), Dice Coefficient (DC), and Sensitivity (SEN). Our proposed method achieves the best performance across all metrics, with ACC of 0.970, IOU of 0.873, DC of 0.941, and SEN of 0.921. Compared to the second-best performing method MSDUnet, our approach demonstrates consistent improvements of 1.1% in ACC, 0.4% in IOU, 1.5% in DC, and 0.9% in SEN. When compared to the classical U-Net baseline, our method achieves substantial improvements of 6.8% in ACC, 7.0% in IOU, 5.9% in DC, and 3.5% in SEN, which clearly validates the effectiveness of our architectural innovations.

The results on the ISIC-2018 dataset, summarized in Table 3, further confirm the superiority of our proposed method. Our approach consistently achieves the highest performance with ACC of 0.970, IOU of 0.836, DC of 0.902, and SEN of 0.928. The improvements over MSDUnet are 0.8% in ACC, 0.3% in IOU, 0.5% in DC, and 1.2% in SEN, while the gains over U-Net reach 4.9% in ACC, 3.0% in IOU, 4.4% in DC, and 2.9% in SEN. These consistent improvements across two different datasets demonstrate the strong generalization capability and robustness of our method.

It is worth noting that transformer-based methods such as Swin-Unet, TransUNet, and H2Former generally outperform the classical U-Net, indicating the advantages of self-attention mechanisms in capturing long-range dependencies for medical image segmentation. However, our method surpasses these transformer-based approaches by effectively integrating multi-scale feature extraction with enhanced attention mechanisms, enabling more comprehensive representation learning and precise boundary delineation.

4.5.2. Qualitative Analysis

Beyond quantitative metrics, we provide qualitative visualization results to offer deeper insights into the segmentation performance of different methods. Figure 4 illustrates representative segmentation results on the ISIC-2016 dataset, comparing our method with TransUNet, SUnet, and MSDUnet alongside the ground truth annotations. The visual comparison reveals that our method produces segmentation masks with significantly clearer and more accurate boundaries, particularly for lesions with irregular shapes as shown in the second and third rows. In challenging cases where artifacts or noise are present, such as the example in the third row, our approach demonstrates superior robustness by generating clean segmentation masks that closely match the ground truth, while other methods tend to produce fragmented or incomplete predictions. Furthermore, for lesions with complex morphologies illustrated in the fourth row, our method better preserves the overall shape and structural integrity of the lesion regions, avoiding the over-segmentation or under-segmentation issues observed in competing approaches.

Figure 5 presents the qualitative comparison on the ISIC-2018 dataset, which further validates the effectiveness of our proposed method across diverse lesion characteristics and imaging conditions. The second row demonstrates a particularly challenging case involving a small lesion, where our method successfully achieves precise segmentation while TransUNet exhibits significant under-segmentation, failing to capture the complete lesion extent. For lesions with highly irregular and fuzzy boundaries shown in the first and second rows, our approach produces more coherent and accurate segmentation masks that better align with the ground truth compared to competing methods, which often struggle with boundary ambiguity. Across all four representative cases spanning various lesion types, sizes, and imaging conditions, our method maintains consistent high-quality performance, demonstrating strong generalization capability and robustness to the inherent variability in dermoscopic images.

The superior performance of our proposed method can be attributed to the synergistic integration of the PVT v2 backbone with the SAFE and MDCF modules within our architecture. The pyramid structure of PVT v2 enables effective multi-scale feature extraction, capturing both fine-grained textural details and global contextual information, which is crucial for handling lesions of varying sizes and appearances. The SAFE module further refines these multi-scale features by bridging the feature discrepancy across network depths through its dual-branch architecture, where the base-feature branch preserves shallow-layer details while the offset-compensation branch adaptively aligns them with deep semantic information using dilated convolutions and global statistics. The MDCF module then establishes bidirectional cross-attention between decoder and SAFE-enhanced encoder features, facilitating precise localization of lesion boundaries by adaptively emphasizing discriminative features while suppressing irrelevant background information. Its four-branch multi-scale deformable convolution unit captures lesion boundaries and small fragments at heterogeneous receptive fields, which is particularly effective in challenging scenarios with low contrast, ambiguous edges, or irregular morphologies. This comprehensive architectural design enhances the model’s ability to handle diverse lesion characteristics, including variations in size, shape, color, and texture patterns commonly observed in clinical dermoscopic images, ultimately achieving a superior balance between segmentation accuracy and computational efficiency.

In summary, both quantitative metrics and qualitative visualizations conclusively demonstrate that our proposed method establishes a new state-of-the-art for skin lesion segmentation on the ISIC-2016 and ISIC-2018 benchmark datasets, outperforming existing advanced approaches in terms of segmentation accuracy, boundary precision, and robustness to challenging imaging conditions.

5. Discussion

The experimental results presented in this study demonstrate that our proposed framework achieves state-of-the-art performance on both ISIC 2016 and ISIC 2018 datasets, validating the effectiveness of integrating PVT v2 backbone with SAFE and MDCF modules for skin lesion segmentation. The hierarchical pyramid structure of PVT v2 proves particularly effective for capturing lesion features at different scales, maintaining global contextual awareness while managing computational complexity through its spatial-reduction attention mechanism. This capability is especially valuable for dermoscopic images where lesions exhibit significant size variation, ranging from small melanomas to large nevi.

The consistent superior performance on both ISIC 2016 and ISIC 2018 datasets indicates strong generalization capability. Despite differences in image acquisition protocols, lesion types, and annotation quality between these datasets, our method maintains robust performance, suggesting that the learned representations capture fundamental characteristics of skin lesions rather than dataset-specific artifacts. While recent transformer-based approaches demonstrate advantages over classical CNN architectures, our method surpasses them by effectively combining the global modeling capability of transformers with targeted feature enhancement and fusion strategies. The performance gap suggests that architectural innovations beyond simply adopting self-attention mechanisms are necessary for optimal medical image segmentation. The qualitative results reveal that our method excels in challenging scenarios commonly encountered in clinical practice, including lesions with fuzzy boundaries, small lesions that are easily overlooked, and cases with imaging artifacts such as hair or air bubbles. The ability to produce clean, accurate segmentation masks in these difficult cases has direct implications for computer-aided diagnosis systems, potentially reducing false negatives in early melanoma detection and improving diagnostic confidence for dermatologists.

However, several limitations warrant discussion. While our method achieves superior segmentation accuracy, the integration of PVT v2 backbone, SAFE modules at multiple scales, and MDCF blocks with cross-attention mechanisms inevitably increases computational cost compared to lightweight models. Although we did not encounter memory constraints during training, deployment on resource-limited devices may require model compression techniques such as knowledge distillation, pruning, or quantization. Our evaluation is conducted on ISIC challenge datasets which, while widely used as benchmarks, primarily contain images acquired under controlled dermoscopic conditions that may not fully represent the variability encountered in real-world clinical practice. The datasets also exhibit class imbalance, with melanoma cases being relatively rare compared to benign lesions, and inter-annotator variability in ground truth labels may introduce noise affecting both training and evaluation.

The current framework provides deterministic predictions without quantifying uncertainty, which is crucial for clinical decision support. Knowing when the model is uncertain about its predictions can help clinicians identify cases requiring additional scrutiny or expert consultation. Additionally, our model relies solely on image data and does not leverage complementary information such as patient metadata or clinical history that dermatologists consider during diagnosis. While we provide feature visualization to demonstrate that SAFE enhances focus on lesion regions, the overall decision-making process remains largely opaque, and incorporating explainable AI techniques would increase transparency and build trust among medical practitioners. Furthermore, our method is specifically designed for melanoma and common skin lesion segmentation, and its performance on other dermatological conditions has not been assessed. The model also performs single-image segmentation without considering temporal information from follow-up images, which is important for monitoring lesion changes over time.

6. Conclusions

In this paper, we have presented a novel deep learning framework for automatic skin lesion segmentation that addresses key challenges in balancing segmentation accuracy and computational efficiency. By integrating the Pyramid Vision Transformer v2 backbone with Spatial-Aware Feature Enhancement and Multi-scale Dual Cross-attention Fusion modules, our method achieves state-of-the-art performance on the ISIC 2016 and ISIC 2018 benchmark datasets, with IOU scores of 87.33% and 83.67%, respectively.

The PVT v2 encoder leverages its hierarchical pyramid structure and spatial-reduction attention mechanism to extract multi-scale features efficiently, enabling the model to capture both fine-grained textural details and global contextual information essential for handling diverse morphological characteristics of skin lesions. The SAFE module refines encoder features at each resolution level through a dual-branch architecture that preserves shallow-layer spatial details while employing dilated convolutions to predict adaptive offsets that align shallow features with deep semantic representations, effectively bridging the feature discrepancy across network depths. The MDCF module enhances the decoder pathway through bidirectional cross-attention and multi-scale deformable convolutions, establishing dynamic feature fusion that emphasizes relevant lesion characteristics while filtering out redundant background information and capturing lesion boundaries at heterogeneous receptive fields.

Extensive experiments demonstrate that our method achieves superior performance compared to existing state-of-the-art approaches, with both quantitative metrics and qualitative visualizations confirming that our framework excels in challenging scenarios including lesions with irregular shapes, fuzzy boundaries, small sizes, and imaging artifacts. The ablation study validates the incremental contribution of each proposed component, with the complete model achieving the best performance across all evaluation metrics. The clinical implications of our work are significant, as accurate and efficient skin lesion segmentation is a critical step in computer-aided diagnosis systems for melanoma, where early detection can dramatically improve patient survival rates.

Future research directions include developing lightweight variants through model compression techniques to enable deployment on resource-constrained devices, incorporating uncertainty quantification to provide confidence estimates alongside predictions, integrating multi-modal information to enhance diagnostic accuracy, extending the framework to handle temporal analysis for lesion monitoring, exploring weakly-supervised learning to reduce annotation requirements, and validating performance on a broader spectrum of dermatological conditions. Our proposed framework represents a significant advancement in automatic skin lesion segmentation, offering a promising tool for improving early detection of skin cancer and demonstrating the potential of deep learning in addressing critical challenges in medical image analysis.