3SGAN: Semisupervised and Multitask GAN for Stain Normalization and Nuclei Segmentation of Histopathological Images

1. Introduction

Histopathology based on hematoxylin and eosin (H&E) staining remains the cornerstone and gold standard for most clinical diagnoses. However, achieving cellular-level analysis of pathology images presents significant challenges. Variations in staining techniques lead to inconsistencies in staining styles, complicating automated analysis. Additionally, extracting quantitative measurements of cell morphology and density from histopathology images is hindered by the vast amount of data requiring processing. Moreover, pixel-level annotation at the cellular level is time-consuming and labor-intensive. These challenges underscore the need for efficient and robust methods for cellular-level intelligent analysis.

Cellular-level intelligent diagnosis depends on precise assessment of cell proportions and spatial composition. However, variations in histochemical staining, particularly H&E, introduce significant obstacles for automated computational pathology. H&E staining, widely used in histopathology, highlights tissue and cellular structures by imparting distinct color contrasts. Yet, differences in staining protocols often result in color variations, known as staining differences, which, while negligible for visual diagnosis by pathologists, impair automated tasks such as cell segmentation and classification [1]. Consequently, the importance of staining style normalization has gained increasing recognition in recent years [2,3,4].

The scarcity of pixel-level annotations further complicates automated nuclei segmentation in pathology images, hindering progress in cellular-level analysis. In digital pathology, fully supervised deep learning models such as Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) dominate nuclei segmentation [5] and classification tasks [6,7]. Among CNN-based approaches, U-Net and its variants are widely adopted for nuclei segmentation [8,9], while ViT-based architectures and their derivatives have recently gained prominence due to their strong performance on histopathology benchmarks [10]. However, these models require extensive pixel-level annotations to achieve robust segmentation and reliable classification. ViTs, with their larger network capacity and more complex architecture, typically demand even more annotated data than CNNs. Given the labor-intensive nature of acquiring pixel-level labels, fully supervised methods face significant constraints in practice, making semi-supervised learning frameworks a more practical solution for nuclei segmentation.

To address staining variations, which manifest as differences in staining intensity and color, we propose a multitask network structure based on a cycle-consistency Generative Adversarial Network (GAN) [11]. This network enhances the generators of the cycle-consistency GAN by adopting a dual-decoder structure with a hard parameter-sharing strategy. By leveraging shared features extracted from a common encoder, this approach facilitates accurate staining normalization and nuclei segmentation with improved quality.

To overcome the limitations in nuclei segmentation accuracy due to insufficient pixel-level annotations, we introduce a semi-supervised teacher-student framework. The teacher model generates high-quality pseudo-labeled datasets from limited training data, including pseudo-labeled nucleus segmentation and staining normalization outputs. The student model, trained on augmented datasets, further refines segmentation and normalization performance. To capitalize on the stronger modeling capabilities of larger models, the framework employs a low-capacity teacher model and a high-capacity student model.

By integrating the semi-supervised framework with the multitask network architecture, we develop 3SGAN. The teacher model, AttCycle, incorporates dual-decoder generators with attention gates [9] within the multitask network, chosen for its efficacy in learning from small datasets. The student model, TransCycle, integrates a hybrid CNN-Transformer encoder into the dual-decoder generator, leveraging its capacity for large datasets. The selection of teacher and student models is guided by their network parameter capacities, reflecting their suitability for datasets of varying sizes.

The main contributions of this study are:

• A novel multitask network architecture that simultaneously performs staining normalization and nuclei segmentation, with the two tasks designed to mutually enhance performance, improving overall segmentation accuracy.
• A semi-supervised teacher-student paradigm that mitigates performance degradation in nuclei segmentation due to scarce pixel-level annotations, leveraging unlabeled data to boost accuracy.
• The integration of the multitask architecture and semi-supervised paradigm into 3SGAN, achieving superior staining normalization and nuclei segmentation performance with minimal annotated data on a large dataset of 1,408 WSIs from two medical institutions, encompassing 101 staining styles and only 5% nucleus-level annotations.

The paper is organized as follows. Section 2 reviews related work. Section 3 details the proposed method and dataset. Section 4 presents experimental results and comparisons with state-of-the-art methods. Section 5 discusses the proposed method's effectiveness in addressing the two challenges. Finally, Section 6 concludes with a summary of findings and directions for future research.

2. Related Work

2.1. Stain Normalization of Histopathological Images

Stain normalization is a critical preprocessing step in histopathological image analysis, addressing variability in staining styles caused by differences in staining protocols, scanner configurations, and tissue reactivity. Existing methods for stain normalization can be broadly categorized into color-matching techniques and style transfer generative models.

Color-matching methods align the color histograms of reference and target images. For instance, Reinhard et al. [12] proposed a global histogram matching approach by converting RGB images to LAB color space and aligning their histograms. However, histogram-based methods often introduce visual artifacts. To mitigate this, stain separation techniques have been developed [13,14]. Khan et al. [14] employed a color-based classifier to estimate the stain matrix, followed by nonlinear color normalization. Such transformations, however, risk altering the image's biological structure. To address this, Vahadane et al. [15] introduced a stain separation method using Sparse Non-negative Matrix Factorization (SNMF) and structure-preserving color normalization (SPCN), which better preserves tissue morphology. Several other SNMF-based approaches have been proposed for stain separation and color unmixing in histopathology [16], demonstrating quantitative benefits for color consistency and downstream analysis. In our work, SNMF-derived stain features are used to characterize stain-style variability and support data-driven style clustering, whereas stain normalization and nuclei segmentation are handled jointly by the proposed 3SGAN framework.

Style transfer generative models, particularly those based on Generative Adversarial Networks (GANs) [17], treat stain normalization as a style transfer problem. Various GAN structures have been explored in medical imaging, including conditional GANs for paired translation [18], cycle-consistent GANs for unpaired translation [19], multi-domain StarGAN that can normalize to multiple target stains with one model [20], and StyleGAN-based methods offering superior high-resolution style control [21]. In histopathology, BenTaieb and Hamarneh [22] combined a stain generation network with an auxiliary task-specific discriminator. However, excessive focus on global appearance may overlook biologically critical regions. Recent works therefore leverage specialized variants such as StainGAN [23] and Colour Adaptive GAN [24], which build upon CycleGAN and incorporate ideas from StarGAN and StyleGAN families to achieve both high visual fidelity and better preservation of tissue structure.

In recent years, diffusion models have emerged as a powerful alternative for stain normalization. Jeong et al. [25] proposed a method using score-based diffusion models, which involves separating H&E stains via SNMF and applying patch-wise normalization with an overlapped moving window strategy to avoid grid artifacts in Whole-Slide Images (WSIs). This approach ensures high-fidelity normalization while preserving tissue structures. Similarly, Yang et al. [26] introduced SAStainDiff, a self-supervised method based on denoising diffusion probabilistic models. SAStainDiff transforms the normalization task into a pixel-wise color-to-color mapping problem, leveraging stain augmentation and a rescheduled sampling strategy to improve inference speed without compromising performance. These diffusion-based methods demonstrate superior generalization and the ability to handle diverse staining styles, making them promising for robust stain normalization in histopathology.

Despite these advancements, existing unpaired image-to-image translation methods, including diffusion-based approaches, have not fully exploited the synergistic relationship between stain normalization and downstream tasks like nuclei segmentation. Moreover, they often fail to adequately address the challenge of limited annotations, which is critical for accurate segmentation in digital pathology.

2.2. Nuclei Segmentation

Nuclei segmentation in histopathology images is a challenging task, complicated by staining variability, overlapping nuclei, and limited pixel-level annotations. Fully supervised deep learning models, such as CNNs, have been widely employed but often struggle to effectively integrate multi-resolution information, resulting in suboptimal pixel-level segmentation outcomes.

Long et al. [27] proposed Fully Convolutional Networks (FCNs), which surpass traditional CNNs in segmentation tasks by leveraging multi-scale information. However, FCNs exhibit inconsistencies in information fusion, leading to reduced sensitivity to fine details. Encoder-decoder architectures, such as U-Net [8], mitigate this issue by incorporating skip connections to better capture contextual details. Nevertheless, the localized nature of convolutional layers restricts U-Net's capacity to model long-range dependencies [28]. To address this limitation, TransUNet [28] integrates transformers into an encoder-decoder framework, enabling global context modeling. Despite its potential, TransUNet's high computational complexity and dependence on large annotated datasets render it less feasible for scenarios characterized by staining variability and limited labeled data.

To overcome these challenges, weakly supervised and semi-supervised approaches have gained traction. X-Net [29], a weakly supervised model combining master and slave U-Nets, employs advanced strategies to enhance segmentation accuracy with fewer annotations. Similarly, Meta-MTL [30] utilizes optimization-based meta-learning and multi-task learning to reduce reliance on extensive labeled data. However, these methods may not consistently deliver robust performance across diverse staining styles.

More recently, foundation models such as the Segment Anything Model (SAM) [31] have been adapted for nuclei segmentation in histopathology. Domain-specific large-scale models, such as MedSAM [32], have demonstrated superior zero-shot segmentation performance in medical imaging contexts. Building on this, pathology-specific segmentation models have emerged. Griebel et al. [33] introduced PathoSAM, a vision foundation model trained on diverse histopathology datasets, achieving state-of-the-art performance in both automated and interactive nucleus instance segmentation.

Despite the impressive zero-shot segmentation capabilities of large-scale models, their general-purpose design limits the scope for prompt engineering optimization, hindering their ability to fully exploit the potential of private datasets. In particular, significant variations in staining styles can lead to unstable segmentation performance. Furthermore, the substantial GPU requirements and prolonged inference times of these models present considerable barriers to their practical implementation in clinical settings.

2.3. Multitask Strategy

Multitask learning has shown promise in improving image style transfer by integrating complementary tasks. Zhang et al. [34] combined image-to-image translation with segmentation, using independent segmentors to enhance synthetic image quality. However, the quality and quantity of synthetic data can destabilize training [34]. Ren et al. [35] proposed a segmentation-renormalization approach to mitigate inter-scanner heterogeneity while preserving anatomical structures across imaging modalities. Similarly, Tomczak et al. [36] employed auxiliary classification and segmentation networks to improve the quality of reconstructed stained images. Jin et al. [37] introduced an asymmetric semi-supervised generative adversarial network (ASSGAN) for breast ultrasound segmentation, leveraging dual generators to mutually supervise segmentation tasks and enhance boundary delineation, though challenges remain in balancing labeled and unlabeled data.

Although these approaches highlight the value of auxiliary tasks in improving image-to-image translation, their effectiveness in segmentation tasks with limited annotations remains constrained. Moreover, incorporating auxiliary tasks elevates training complexity, underscoring the need for efficient network architectures and optimized training strategies.

2.4. Semisupervised Learning

Semisupervised learning offers a robust solution for medical image processing when labeled data is scarce, often by generating pseudo-labeled data to augment training. Yalniz et al. [38] developed a teacher-student pipeline that improved ImageNet performance using up to 1 billion unlabeled images. Shaw et al. [39] demonstrated that semi-supervised teacher-student models can rival fully supervised methods. Earlier, Lee [40] explored a teacher-student paradigm with a high-capacity teacher and a low-capacity student to optimize inference efficiency.

Recent advances include tailored teacher-student frameworks for medical imaging. Zhang et al. [41] introduced a Weakly-Supervised Teacher-Student network (WSTS) for liver tumor segmentation without contrast agents, effectively handling weak labels. Xiao et al. [42] proposed a Dual-Teacher structure combining CNN and transformer teachers to guide a student segmentation model, significantly improving performance with unlabeled data. Lu et al. [43] proposed a framework combining high-uncertainty-aware pseudo-labeling with dual-consistency regularization, using cycle-loss regularization to improve uncertainty estimation and enhance consistency in teacher-student networks. These studies highlight the efficacy of teacher-student frameworks in addressing limited or imperfect annotations in medical imaging.

Despite progress in stain normalization and nuclei segmentation, few studies have tackled both challenges simultaneously, particularly under limited annotations. This paper investigates this gap using a multi-task learning approach, integrating stain normalization and nuclei segmentation to achieve robust performance with minimal labeled data.

3. Materials and Methods

3.1. Methodological Overview

To tackle the challenges in histopathology at the cellular level, this study introduces 3SGAN, a novel semisupervised, multitask generative adversarial network. The 3SGAN framework adopts a teacher-student paradigm, with both the teacher model (AttCycle) and student model (TransCycle) built upon a proposed multitask network. This network enhances CycleGAN's generators by incorporating a dual-decoder structure. The teacher and student models in 3SGAN integrate specialized modules to distinguish their functionalities. Figure 1 illustrates the 3SGAN architecture: (a) outlines the data flow, (b) depicts the network structures of AttCycle and TransCycle, and (c) and (d) detail the core modules of the teacher and student models, respectively.

While 3SGAN aligns with the conventional teacher-student paradigm, it introduces key innovations. Both AttCycle and TransCycle leverage the multitask network's dual-decoder structure, enabling simultaneous execution of multiple tasks to improve individual task performance. The teacher model's dual-decoder generator incorporates attention gates to produce high-quality pseudolabeled data for the student model. In contrast, the student model's generator integrates a CNN-Transformer architecture to elevate its performance ceiling. The Transformer's strength in global feature modeling shines when abundant training data is available, allowing the student model to capitalize on the teacher's high-quality pseudolabeled data. This approach mitigates the scarcity of annotated data, enhancing outcomes in nuclei segmentation and staining normalization. By combining CNN's local detail preservation with the Transformer's global feature modeling, the student model optimizes both local and global feature relationships. Figure 1(a) details the semisupervised framework’s data processing steps:

• The teacher model (AttCycle) is pretrained using a limited set of labeled samples.
• The pretrained teacher model generates nuclei segmentation and staining normalization results from large volumes of unlabeled data.
• A pseudolabeled dataset selection algorithm (described in Section 3.4) filters the teacher's outputs to create a high-quality pseudolabeled dataset, encompassing pseudolabeled nuclei and staining normalization data.
• The student model (TransCycle) is trained on both the pseudolabeled dataset and the original labeled dataset used for the teacher model.

The teacher model employs attention gates in its generators, while the student model adopts a CNN-Transformer-based encoder architecture, further refining performance.

Figure 1(b) delineates the architectural framework of the teacher model (AttCycle) and the student model (TransCycle) within the 3SGAN system. Consistent with the CycleGAN paradigm, each model comprises two generators and two discriminators. However, unlike the conventional CycleGAN architecture, the generators in 3SGAN adopt a dual-decoder design. Specifically, in the teacher model (AttCycle), the generator ${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}\_\mathrm{t}}$ processes an input $\mathrm{X}\_\mathrm{t}$ and simultaneously produces two outputs: a stain-normalized image ${\tilde{\mathrm{X}}}_{\mathrm{r}\mathrm{s}\_\mathrm{t}}$, achieved by translating image patches from domain X to domain Y, and a nuclei segmentation map ${\tilde{\mathrm{X}}}_{\mathrm{s}\mathrm{e}\mathrm{g}\_\mathrm{t}}$ This dual-output process is formally represented as:

$${\tilde{\mathrm{X}}}_{\mathrm{r}\mathrm{s}\_\mathrm{t}},{\tilde{\mathrm{X}}}_{\mathrm{s}\mathrm{e}\mathrm{g}\_\mathrm{t}}={{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}\_\mathrm{t}}(\mathrm{X}\_\mathrm{t})$$

Conversely, the generator ${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{Y}2\mathrm{X}\_\mathrm{t}}$ performs the inverse style translation. The discriminator ${\mathrm{D}}_{\mathrm{Y}\_\mathrm{t}}$ evaluates the translated image ${\tilde{\mathrm{X}}}_{\mathrm{r}\mathrm{s}\_\mathrm{t}}$ to ensure its fidelity to the target domain Y. Given the unpaired nature of the cross-domain data, a cycle-consistency loss is employed to enforce structural preservation, ensuring that the reconstructed image, defined as ${\tilde{\mathrm{X}}}_{\mathrm{r}\mathrm{s}\_\mathrm{t}}={{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{Y}2\mathrm{X}\_\mathrm{t}}$(${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}\_\mathrm{t}}(\mathrm{X}\_\mathrm{t}))$ , closely approximates the original input $\mathrm{X}\_\mathrm{t}$. This approach effectively reduces staining variations in histopathological images, thereby improving the reliability of downstream image analysis. A parallel adversarial mechanism operates within the student model (TransCycle), utilizing the generators ${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}\_\mathrm{s}}$ and ${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{Y}2\mathrm{X}\_\mathrm{s}}$, alongside the discriminator ${\mathrm{D}}_{\mathrm{Y}\_\mathrm{s}}$ .

Additionally, the structural configurations of the attention gate and transformer layer are depicted in Figure 1(c) and Figure 1(d), respectively. The attention gate, originally proposed in Attention U-Net [9], is integrated into the teacher model (AttCycle). This mechanism enhances the network’s focus on nucleus-specific features, yielding more accurate segmentation and preserving stylistic integrity during normalization. Meanwhile, the student model (TransCycle) incorporates transformer layers to bolster its global feature extraction capabilities, making it particularly adept for tasks requiring extensive training data.

3.2. Stain Normalization and Segmentation Multitask Network

In histopathological image analysis, staining normalization and cell nucleus segmentation are mutually influential tasks that enhance each other's performance. Variations in staining across histopathological images can significantly impair the accuracy of computer-aided analysis, particularly in segmentation tasks. For instance, when processing identical tissue slices with the same deep learning segmentation model, differing staining styles can lead to markedly inconsistent segmentation outcomes. Standardizing the staining style mitigates this issue by reducing noise introduced by staining variability, thereby improving the consistency and precision of segmentation results. Conversely, the segmentation task's emphasis on the biological structure of the nucleus contributes to refining the quality of staining normalization.

To capitalize on this interdependence, we propose a novel multitask network, as illustrated in Figure 2. The network comprises:

• Shared encoders (illustrated in blue)
• Stain normalization branches (depicted in green)
• Segmentation branches (represented in gray)
• PatchGAN-based discriminators (shown in yellow)

This multitask framework enhances both the AttCycle teacher model and the TransCycle student model, as depicted in Figure 1(b).

Our approach leverages a hard parameter sharing strategy to improve the generators within the CycleGAN framework. Specifically, the multitask network employs a shared encoder, enabling the two parallel task-specific decoders to share weights and provide mutual support. This design not only increases the network's efficiency but also facilitates the extraction of features pertinent to both tasks, ultimately leading to superior overall performance.

The nuclei segmentation and stain normalization branches are integrated and included as decoders in${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}}$and ${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{Y}2\mathrm{X}}$. Specifically, ${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}}$denotes the multitask network that translates image patches from domain X to domain Y, while ${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{Y}2\mathrm{X}}$performs the inverse transformation. The segmentation branch is trained with nuclear labels of the input images, allowing ${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}}$ to produce outputs for both tasks simultaneously: the nuclei segmentation output ${\tilde{\mathrm{X}}}_{\mathrm{s}\mathrm{e}\mathrm{g}}$and the stain normalization output ${\tilde{\mathrm{X}}}_{\mathrm{r}\mathrm{s}}$. Hence, we have ${{\tilde{\mathrm{X}}}_{\mathrm{s}\mathrm{e}\mathrm{g}},\tilde{\mathrm{X}}}_{\mathrm{r}\mathrm{s}}={{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}}$(X). The segmentation output ${\tilde{\mathrm{X}}}_{\mathrm{s}\mathrm{e}\mathrm{g}}$/${\tilde{\mathrm{Y}}}_{\mathrm{s}\mathrm{e}\mathrm{g}}$ in ${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}}$/${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{Y}2\mathrm{X}}$ is obtained by training the segmentation branch with nuclei annotations ${\mathrm{X}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}$/${\mathrm{Y}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}$ paired with input X/Y. For each image x in domain X and each y in domain Y, the objectives can be expressed as follows:

• Segmentation Terms: We utilized a binary cross entropy loss for the segmentation task in our approach. The use of this segmentation loss allowed for the effective weight updates of the segmentation task decoder within the multitask network. Through this approach, we were able to improve the stain normalization results by focusing specifically on the cell nucleus region with the aid of the segmentation loss. This resulted in more accurate segmentation of cell nuclei.

$${\mathrm{L}}_{\mathrm{S}\mathrm{e}\mathrm{g}}\left({\tilde{\mathrm{X}}}_{\mathrm{s}\mathrm{e}\mathrm{g}},{\mathrm{X}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}\right)=-{\mathrm{E}}_{{\tilde{\mathrm{x}}}_{\mathrm{s}\mathrm{e}\mathrm{g}}~{\tilde{\mathrm{X}}}_{\mathrm{s}\mathrm{e}\mathrm{g}},{\mathrm{x}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}~{\mathrm{X}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}}\left[{\mathrm{x}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}\mathrm{l}\mathrm{o}\mathrm{g}\left({\tilde{\mathrm{x}}}_{\mathrm{s}\mathrm{e}\mathrm{g}}\right)+\left(1-{\mathrm{x}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}\mathrm{l}\mathrm{o}\mathrm{g}\left(1-{\tilde{\mathrm{x}}}_{\mathrm{s}\mathrm{e}\mathrm{g}}\right)\right)\right]$$

(1)

Adversarial Terms: We use the least square loss for adversarial training. ${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}}$and ${\mathrm{D}}_{\mathrm{Y}}$ are optimized with analogous optimization applied for ${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{Y}2\mathrm{X}}$ and ${\mathrm{D}}_{\mathrm{X}}$. ${\tilde{\mathrm{x}}}_{\mathrm{r}\mathrm{s}}$is the staining normalization output of ${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}}$. Through the use of this loss function, the dual-decoder generator of our multitask network is capable of generating images that closely resemble the target domain images in terms of color style. Meanwhile, the discriminator is more effective in discerning whether the generated images match the color style of the target domain images.

$$\underset{{\mathrm{D}}_{\mathrm{Y}}}{\min }{\mathrm{L}}_{\mathrm{G}\mathrm{A}\mathrm{N}}\left({\mathrm{D}}_{\mathrm{Y}}\right)={\displaystyle \frac{1}{2}}{\mathrm{E}}_{\mathrm{y}~\mathrm{Y}}\left[{‖1-{\mathrm{D}}_{\mathrm{Y}}\left(\mathrm{y}\right)‖}_2^2\right]+{\displaystyle \frac{1}{2}}{\mathrm{E}}_{{\tilde{\mathrm{x}}}_{\mathrm{r}\mathrm{s}}~{\tilde{\mathrm{X}}}_{\mathrm{r}\mathrm{s}}}\left[{‖{\mathrm{D}}_{\mathrm{Y}}\left({\tilde{\mathrm{x}}}_{\mathrm{r}\mathrm{s}}\right)‖}_2^2\right]$$

(2)

$$\underset{{{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}}}{\min }{\mathrm{L}}_{\mathrm{G}\mathrm{A}\mathrm{N}}\left({{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}}\right)={\displaystyle \frac{1}{2}}{\mathrm{E}}_{{\tilde{\mathrm{x}}}_{\mathrm{r}\mathrm{s}}~{\tilde{\mathrm{X}}}_{\mathrm{r}\mathrm{s}}}\left[{‖{\mathrm{D}}_{\mathrm{Y}}\left({\tilde{\mathrm{x}}}_{\mathrm{r}\mathrm{s}}\right)-1‖}_2^2\right]$$

(3)

• Cycle-Consistency Terms: The cycle-consistency loss is used to enforce the consistency of histopathological structures between X and ${\widehat{\mathrm{X}}}_{\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{d}}$:

$${\mathrm{L}}_{\mathrm{C}\mathrm{y}\mathrm{c}}\left({{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}},{{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{Y}2\mathrm{X}}\right)={\mathrm{E}}_{\mathrm{x}~\mathrm{X}}\left[{‖{{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{Y}2\mathrm{X}}\left({{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}}\right(\mathrm{x})-\mathrm{x}‖}_1\right]+{\mathrm{E}}_{\mathrm{y}~\mathrm{Y}}\left[{‖{{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}}\left({{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{Y}2\mathrm{X}}\right(\mathrm{y})-\mathrm{y}‖}_1\right]$$

(4)

• Total objective: This total loss function establishes a connection between the segmentation task and the stain normalization task by facilitating the hard parameter sharing of the common encoder during network training. This approach yields enhanced results for both tasks, leading to improved cell nucleus segmentation and stain normalization. The complete objective function of this total loss to minimize is summarized as following.

$$\begin{array}{c}L\left({{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}}{{,\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{Y}2\mathrm{X}},{\mathrm{D}}_{\mathrm{Y}},{\mathrm{D}}_{\mathrm{X}},{\mathrm{X}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}\right)=\left[{\mathsf{\lambda }}_{\mathrm{X}2\mathrm{Y}}{\mathrm{L}}_{\mathrm{G}\mathrm{A}\mathrm{N}}\left({{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}}\right){+{\mathrm{L}}_{\mathrm{G}\mathrm{A}\mathrm{N}}\left({\mathrm{D}}_{\mathrm{Y}}\right)+\mathrm{L}}_{\mathrm{G}\mathrm{A}\mathrm{N}}\left({{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{Y}2\mathrm{X}}\right){+\mathrm{L}}_{\mathrm{G}\mathrm{A}\mathrm{N}}\left({\mathrm{D}}_{\mathrm{X}}\right)\right]\\ +{\mathsf{\lambda }}_{\mathrm{C}\mathrm{y}\mathrm{c}}{\mathrm{L}}_{\mathrm{C}\mathrm{y}\mathrm{c}}\left({{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}},{{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{Y}2\mathrm{X}}\right)+{\mathsf{\lambda }}_{\mathrm{S}\mathrm{e}\mathrm{g}}{\mathrm{L}}_{\mathrm{S}\mathrm{e}\mathrm{g}}\left({\tilde{\mathrm{X}}}_{\mathrm{s}\mathrm{e}\mathrm{g}},{\mathrm{X}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}\right)\\ \end{array}$$

(5)

where ${\mathsf{\lambda }}_{\mathrm{X}2\mathrm{Y}}$, ${\mathsf{\lambda }}_{\mathrm{C}\mathrm{y}\mathrm{c}}$ and ${\mathsf{\lambda }}_{\mathrm{S}\mathrm{e}\mathrm{g}}$ denote the weights of translation loss from the X domain to the Y domain, cycle-consistency loss and segmentation loss, respectively.

3.3. Semisupervised Teacher-Student Pipeline

In traditional teacher-student paradigms aimed at reducing model complexity and computational expenses [38,40], the teacher model is typically a large-parameter model valued for its superior feature extraction capabilities, while the student model is designed with fewer parameters to maintain comparable performance with reduced computational and memory demands. However, this conventional approach may falter when the labeled dataset is limited, such as when it comprises only a few hundred images. To overcome this challenge, we propose a novel teacher-student framework that leverages a low-capacity CNN-based teacher model paired with a high-capacity student model featuring a CNN-Transformer hybrid architecture.

Our design choices are driven by the unique strengths of CNNs and Transformers, tailored to the constraints of a small labeled dataset:

• Advantages of CNNs in the Teacher Model: CNNs require fewer annotations for training due to their relatively small parameter count compared to Transformers. This is particularly beneficial in medical image analysis, where annotating pathological cell nuclei is labor-intensive and prone to errors. Complex teacher models often struggle to produce stable, high-quality pseudolabels under such conditions. In contrast, our CNN-based teacher model, AttCycle, excels at generating reliable pseudolabeled data even with minimal annotated samples.
• Benefits of a High-Capacity Student Model: Large models generally outperform smaller ones when trained on abundant data. Thus, we employ a student model, TransCycle, which integrates CNN and Transformer architectures. This hybrid design harnesses CNNs' prowess in preserving fine-grained details and Transformers' strength in modeling global contextual information, aiming to elevate overall performance.

Our approach unfolds in a structured, two-stage process:

• Training the Teacher Model (AttCycle):

• We begin by training AttCycle, a low-capacity CNN-based teacher model, on a small local dataset. AttCycle enhances its focus on cell nucleus regions by incorporating attention gates from Attention U-Net into a multitask network, improving both stain normalization and cell nucleus segmentation.
• Owing to its CNN architecture, AttCycle reliably generates high-quality pseudolabels with limited annotated data, outperforming Transformer-based alternatives in this constrained setting.
• To ensure pseudolabel quality, we apply a sigmoid probability method: only pseudolabels where each confidential pixel's sigmoid probability exceeds a predefined threshold, and the proportion of such pixels surpasses a preset confidence ratio, are retained.

2.

Training the Student Model (TransCycle):

• The augmented dataset, enriched with high-quality pseudolabels from AttCycle, is then used to train TransCycle, our high-capacity student model. TransCycle employs a hybrid CNN-Transformer encoder within the dual-decoder generators of a multitask network.
• With a greater capacity than AttCycle, TransCycle excels at recovering localized spatial details and enhancing finer features when trained on large datasets, leveraging the complementary strengths of CNNs and Transformers.

Our innovative teacher-student paradigm synergizes the efficiency of a low-capacity CNN-based teacher model with the robustness of a high-capacity CNN-Transformer student model. By addressing the limitations of small labeled datasets, this approach significantly enhances the performance of medical image analysis, offering a practical solution for scenarios where annotated data is scarce.

3.4. Architecture and Design Details

The proposed teacher and student models are shown in Figure 1(b) and can be decomposed into their corresponding multitask generators and discriminators.

3.4.1. Teacher Model Generators

The teacher model, AttCycle utilizes generators ${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}\_\mathrm{t}}$and ${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{Y}2\mathrm{X}\_\mathrm{t}}$ with attention gates (AGs) incorporated into a 3-level upsample-downsample architecture, featuring one encoder and two decoders. AGs are used to emphasize salient features that are passed through the skip connections between the encoder and decoders at the corresponding level. This enables the teacher model to generate high-quality staining normalization and segmentation results, which is crucial for the student model's training. Specifically, ${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}\_\mathrm{t}}$ translates images from X domain to Y domain in the teacher model, while ${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{Y}2\mathrm{X}\_\mathrm{t}}$ performs the inverse translation. The structure details of ${{\mathrm{G}}^{\mathrm{\text{'}}}}_{\mathrm{X}2\mathrm{Y}\_\mathrm{t}}$ are shown in Figure 1(b) of the teacher model.

AGs select features by using contextual information extracted at coarser scales. Coarse feature maps capture contextual information and emphasize the class and location of target objects. The output of the AG is a multiplication of input feature maps ${\mathrm{x}}_{\mathrm{i},\mathrm{c}}^{\mathrm{l}}$and attention coefficients${\mathsf{\alpha }}_{\mathrm{i}}^{\mathrm{l}}$, where ${\mathrm{x}}^{\mathrm{l}}$ denotes higher dimensional image representations, and i and c denote spatial and channel dimensions, respectively:

$${\widehat{\mathrm{x}}}_{\mathrm{i},\mathrm{c}}^{\mathrm{l}}={\mathrm{x}}_{\mathrm{i},\mathrm{c}}^{\mathrm{l}}·{\mathsf{\alpha }}_{\mathrm{i}}^{\mathrm{l}}$$

(6)

A gating vector ${\mathrm{g}}_{\mathrm{i}}\in {\mathrm{R}}^{{\mathrm{F}}_{\mathrm{g}}}$ is used for each pixel i to determine the focus region, and ${\mathrm{F}}_{\mathrm{g}}$ corresponds to the number of feature maps in layer g. The gating vector contains contextual information to prune lower-level feature responses. The gating coefficient is obtained by additive attention. Additive attention is formulated as follows:

$${\mathrm{q}}_{\mathrm{a}\mathrm{t}\mathrm{t}}^{\mathrm{l}}={\mathsf{\psi }}^{\mathrm{T}}\left({\mathsf{\sigma }}_1\left({\mathrm{W}}_{\mathrm{x}}^{\mathrm{T}}{\mathrm{x}}_{\mathrm{i}}^{\mathrm{l}}+{\mathrm{W}}_{\mathrm{g}}^{\mathrm{T}}{\mathrm{g}}_{\mathrm{i}}+{\mathrm{b}}_{\mathrm{g}}\right)\right)+{\mathrm{b}}_{\mathsf{\psi }}$$

(7)

$${\mathsf{\alpha }}_{\mathrm{i}}^{\mathrm{l}}={\mathsf{\sigma }}_2\left({\mathrm{q}}_{\mathrm{a}\mathrm{t}\mathrm{t}}^{\mathrm{l}}\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{l}},{\mathrm{g}}_{\mathrm{i}};{\mathsf{\theta }}_{\mathrm{a}\mathrm{t}\mathrm{t}}\right)\right)$$

(8)

where ${\mathsf{\sigma }}_1$represents the ReLU activation function and${\mathsf{\sigma }}_2\left({\mathrm{x}}_{\mathrm{i},\mathrm{c}}\right)=1/1+\mathrm{e}\mathrm{x}\mathrm{p}(-{\mathrm{x}}_{\mathrm{i},\mathrm{c}})$corresponds to the sigmoid activation function. AG is characterized by a set of parameters ${\mathsf{\theta }}_{\mathrm{a}\mathrm{t}\mathrm{t}}$ containing linear transformations ${\mathrm{W}}_{\mathrm{x}}\in {\mathrm{R}}^{{\mathrm{F}}_{\mathrm{l}}\times {\mathrm{F}}_{\mathrm{i}\mathrm{n}\mathrm{t}}},$ ${\mathrm{W}}_{\mathrm{g}}\in {\mathrm{R}}^{{\mathrm{F}}_{\mathrm{g}}\times {\mathrm{F}}_{\mathrm{i}\mathrm{n}\mathrm{t}}},\mathsf{\psi }\in {\mathrm{R}}^{{\mathrm{F}}_{\mathrm{i}\mathrm{n}\mathrm{t}}\times 1}$ and bias terms ${\mathrm{b}}_{\mathsf{\psi }}\in \mathrm{R}$, ${\mathrm{b}}_{\mathrm{g}}\in$ ${\mathrm{R}}^{{\mathrm{F}}_{\mathrm{i}\mathrm{n}\mathrm{t}}}$. The linear transformations are computed using channelwise $1\times 1$convolutions for the input tensors.

3.4.2. Student Model Generators

The student model, TransCycle, uses a CNN-Transformer hybrid architecture as the backbone of its generators’ encoder in the proposed multitask structure. This hybrid architecture combines the advantages of detailed high-resolution spatial information from CNN features and the global context encoded by transformers. Additionally, it has more parameters, which may result in better performance when trained on more data [38] . Inspired by U-Net, the CNN-Transformer hybrid model is employed as an encoder where CNN first extracts and generates a feature map for the input. Transformers are then used to obtain self-attention features, which are upsampled to be combined with different high-resolution CNN features skipped from the encoder. Two cascaded decoders consisting of multiple upsampling steps, are used to decode the hidden feature for stain normalization and nuclei segmentation tasks. These cascaded decoders incorporate the hybrid encoder merge feature at different levels via skip connections.

Specifically, this paper implements the hybrid CNN-Transformer encoder by combining ResNet-50 with 12 Transformer layers, illustrated in Figure 1(b) student model. Each Transformer layer comprises multihead self-attention (MSA) and multilayer perceptron (MLP) blocks. Following patch embedding [28], ResNet-50 extracts feature map $\mathrm{M}$ with input $\mathrm{X}\in {\mathrm{R}}^{\mathrm{H}\times \mathrm{W}\times \mathrm{C}}$, where $\mathrm{H}\times \mathrm{W}$ denotes the spatial resolution and C denotes the number of channels. M is tokenized into a sequence of flattened $1\times 1$patches ${\mathrm{M}}_{\mathrm{p}}\in {\mathrm{R}}^{\mathrm{N}\times (1\times 1\times \mathrm{C})}$, where $\mathrm{N}=\mathrm{H}\times \mathrm{W}/1\times 1$ is the number of patches. Then, we employ positional information embeddings ${\mathrm{E}}_{\mathrm{p}\mathrm{o}\mathrm{s}}$ to retain the positional information of patches in the previous feature map and map the vectorized patches ${\mathrm{M}}_{\mathrm{p}}$ into a latent D-dimensional embedding space ${\mathrm{E}}_{\mathrm{D}}$ via a trainable linear projection as follows:

$${\mathrm{Z}}_0={\mathrm{M}}_{\mathrm{p}}·{\mathrm{E}}_{\mathrm{D}}+{\mathrm{E}}_{\mathrm{p}\mathrm{o}\mathrm{s}}$$

(9)

where ${\mathrm{E}}_{\mathrm{D}}\in {\mathrm{R}}^{(1\times 1\times \mathrm{C})\times \mathrm{D}}$ denotes the linear projection and ${\mathrm{E}}_{\mathrm{p}\mathrm{o}\mathrm{s}}\in {\mathrm{R}}^{\mathrm{N}\times \mathrm{D}}$ denotes the position embedding. In the $\mathcal{l}\mathrm{t}\mathrm{h}$ transformer layer, the output ${\mathrm{Z}}_{\mathcal{l}}$ is formulated as follows:

$${\mathrm{Z}}_{\mathcal{l}}^{\mathrm{\text{'}}}=\mathrm{M}\mathrm{S}\mathrm{A}\left(\mathrm{L}\mathrm{N}\left({\mathrm{Z}}_{\mathcal{l}-1}\right)\right)+{\mathrm{Z}}_{\mathcal{l}-1}$$

(10)

$${\mathrm{Z}}_{\mathcal{l}}=\mathrm{M}\mathrm{L}\mathrm{P}\left(\mathrm{L}\mathrm{N}\left({\mathrm{Z}}_{\mathcal{l}}^{\mathrm{\text{'}}}\right)\right)+{\mathrm{Z}}_{\mathcal{l}}^{\mathrm{\text{'}}}$$

(11)

where LN ($·$) denotes the layer normalization operator. The structure of one transformer layer is shown in Figure 1(d).

3.4.3. Pseudolabeled Dataset Selection

In this paper, we propose a semisupervised teacher-student framework for improving nuclei segmentation accuracy in histopathology images. In this framework, the teacher model generates segmentation results for a limited number of annotated images, which are then used as the training set for the student model. However, to minimize noise interference in the teacher output, we introduce a selection method to screen the pseudolabeled data for credibility.

Specifically, we create a Sigmoid probability matrix for each segmentation result generated by the teacher model. To define the data used in our framework, we consider annotated images $\mathrm{X}=\{{\mathrm{x}}_1,{\mathrm{x}}_2,\dots ,{\mathrm{x}}_{\mathrm{n}}\}$ and their corresponding labels ${\mathrm{X}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}=\{{{\mathrm{x}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}}_1,{{\mathrm{x}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}}_2,\dots ,{{\mathrm{x}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}}_{\mathrm{n}}\}$, teacher model segmentation results ${\tilde{\mathrm{X}}}_{\mathrm{s}\mathrm{e}\mathrm{g}}=\{{\mathrm{x}}_{\mathrm{s}\mathrm{e}{\mathrm{g}}_1},{\mathrm{x}}_{\mathrm{s}\mathrm{e}{\mathrm{g}}_2},\dots ,{\mathrm{x}}_{\mathrm{s}\mathrm{e}{\mathrm{g}}_{\mathrm{n}}}\}$, and unannotated images $\mathrm{U}=\{{\mathrm{u}}_1,{\mathrm{u}}_2,\dots ,{\mathrm{u}}_{\mathrm{n}}$}. In the labeled dataset X, the images are categorized into two staining style domains, with an equal distribution of samples in each domain.

To determine the credibility of the segmentation results generated by the teacher model, we apply the Sigmoid activation function to obtain a classification probability map ${\mathsf{\alpha }}_{\mathrm{i}}$ for each segmentation result ${\mathrm{p}}_{\mathrm{i}}\mathsf{ϵ}\mathrm{P}$ generated for an unannotated image. ${\mathrm{n}}_{\mathrm{i}}\mathsf{ϵ}\mathrm{N}$ is a staining normalization result generated by the teacher model with an unannotated image from U. We classify each pixel in ${\mathsf{\alpha }}_{\mathrm{i}}$ as a confident pixel if the Sigmoid prediction value of that pixel is greater than the confidence threshold $\mathsf{\theta }=0.7$, and we increase the count of confident pixels by 1 if the confidence value of the pixel at position (i, j) is greater than $\mathsf{\theta }$. Finally, we compute the ratio ${\mathsf{\mu }}_{\mathrm{i}}$ of the number of confident pixels to the total number of pixels in the segmentation result for each unannotated image. If ${\mathsf{\mu }}_{\mathrm{i}}$ is greater than the specified confidence pixel ratio threshold of $\mathsf{\mu }$ =0.6, then the segmentation result of that image and its corresponding staining normalization result can be used as pseudolabeled dataset for the student model's training. The nucleus pseudolabels $\mathrm{P}({\mathsf{\mu }}_{\mathrm{i}}>\mathsf{\mu })$ and pixel-wise nucleus label ${\mathrm{X}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}$ are used for the training of student model’ segmentation branch. And the pseudostaining normalization results $\mathrm{N}({\mathsf{\mu }}_{\mathrm{i}}>\mathsf{\mu })$ and their corresponding unlabeled data $\mathrm{U}({\mathsf{\mu }}_{\mathrm{i}}>\mathsf{\mu })$ work with labeled data X as the training dataset of the student’s staining normalization branch.

Overall, our semi-supervised framework with the selection method for credible pseudolabeled data provide a promising approach for improving nuclei segmentation accuracy in histopathology images. The algorithm of pseudolabeled data selection is outlined as follows.

Algorithm 1. Teacher-student pseudolabel selection with confidence thresholding.

Algorithm 1 Pseudolabeled data selection Input: Labeled data $\mathrm{X}=\{{\mathrm{x}}_1,{\mathrm{x}}_2,\dots ,{\mathrm{x}}_{\mathrm{n}}\}$, pixel-wise label ${\mathrm{X}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}$ = $\{{{\mathrm{x}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}}_1,{{\mathrm{x}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}}_2,\dots ,{{\mathrm{x}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}}_{\mathrm{n}}\}$, teacher model segmentation output ${\tilde{\mathrm{X}}}_{\mathrm{s}\mathrm{e}\mathrm{g}}$=$\{{\mathrm{x}}_{\mathrm{s}\mathrm{e}{\mathrm{g}}_1},{\mathrm{x}}_{\mathrm{s}\mathrm{e}{\mathrm{g}}_2},\dots ,{\mathrm{x}}_{\mathrm{s}\mathrm{e}{\mathrm{g}}_{\mathrm{n}}}\}$, unlabeled data$\mathrm{U}=\{{\mathrm{u}}_1,{\mathrm{u}}_2,\dots ,{\mathrm{u}}_{\mathrm{n}}$}, confidence threshold θ, confidence pixel ratio μ. 1: ${\mathrm{L}}_{\mathrm{S}\mathrm{e}\mathrm{g}}\left({\tilde{\mathrm{X}}}_{\mathrm{s}\mathrm{e}\mathrm{g}},{\mathrm{X}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}\right)=-{\mathrm{E}}_{{\tilde{\mathrm{x}}}_{\mathrm{s}\mathrm{e}\mathrm{g}}~{\tilde{\mathrm{X}}}_{\mathrm{s}\mathrm{e}\mathrm{g}},{\mathrm{x}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}~{\mathrm{X}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}}\left[{\mathrm{x}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}\mathrm{l}\mathrm{o}\mathrm{g}\left({\tilde{\mathrm{x}}}_{\mathrm{s}\mathrm{e}\mathrm{g}}\right)+\left(1-{\mathrm{x}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}\mathrm{l}\mathrm{o}\mathrm{g}\left(1-{\tilde{\mathrm{x}}}_{\mathrm{s}\mathrm{e}\mathrm{g}}\right)\right)\right]$ 2: For i = 1 to n do 3: ${\mathrm{p}}_{\mathrm{i}}$, ${\mathrm{n}}_{\mathrm{i}}$ = teacher(${\mathrm{u}}_{\mathrm{i}}$) 4: ${\mathsf{\alpha }}_{\mathrm{i}}$ = sigmoid(${\mathrm{p}}_{\mathrm{i}}$) 5: If ${\mathsf{\alpha }}_{{\mathrm{i}}_{\mathrm{h},\mathrm{w}}}$ ≥ θ then 6: number += 1 7: EndIf 8: ${\mathsf{\mu }}_{\mathrm{i}}$= ${\displaystyle \frac{number}{H\times W}}$ 9: EndFor 10: Training dataset of student model: S = X ∪ U(${\mathsf{\mu }}_{\mathrm{i}}$ ≥ $\mathsf{\mu }$) ∪ N(${\mathsf{\mu }}_{\mathrm{i}}$ ≥ $\mathsf{\mu }$), ${\mathrm{S}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}$= ${\mathrm{X}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}$∪ P(${\mathsf{\mu }}_{\mathrm{i}}$ ≥ $\mathsf{\mu }$

4. Results

4.1. Experimental Dataset Composition

Our dataset comprises 1,408 H&E-stained glioma WSIs from 1,408 distinct patients sourced from two medical centers: Huashan Hospital of Fudan University (HS; n=848, 60.2%) and the Chinese University of Hong Kong/Prince of Wales Hospital (CUHK; n=560, 39.8%). Glioma cases were retrospectively retrieved from the digital pathology archives of these institutions between January 2014 and December 2017. Cases were included if they had a confirmed glioma diagnosis according to WHO classification, were based on formalin-fixed paraffin-embedded (FFPE) tissue processed with standard H&E staining protocols, contained sufficient tumor content, and yielded technically adequate WSIs without major folds, tears, or severely out-of-focus regions. Slides with severe artifacts or uncertain diagnoses were excluded.

All included slides were FFPE sections captured at 40× magnification and digitized at 0.25 μm/pixel using institutional whole-slide scanners (Hamamatsu NanoZoomer) under default acquisition specifications. By WHO classification, the dataset included 360 grade II WSIs (25.6%), 312 grade III WSIs (22.2%), and 736 grade IV WSIs (52.3%), corresponding to 672 lower-grade (II–III) and 736 high-grade (IV) WSIs.

Tumor-rich regions of interest (ROIs) were selected by experienced neuropathologists. Regions dominated by normal parenchyma, severe tissue loss, pen marks, large hemorrhage, or scanning artifacts were excluded. Selected ROIs were tiled into a total of 52,680 non-overlapping 512×512-pixel patches at 40× magnification (128–768 patches per ROI). Three board-certified neuropathologists annotated glioma nuclei in 1,715 patches across varied staining styles using a multi-stage consensus protocol: the first nuclei performed initial annotations, the second reviewed them, and the third resolved any discrepancies. Structures were labeled as nucleus only if they presented a clear contour and sufficient chromatin contrast; vessels, necrosis, edema, hemorrhage, and other non-nuclear structures were excluded.

Training data labels were created exclusively for glioma tumor cell nuclei, which served as the only annotated structures used to supervise model training. Nuclei pixels were assigned a value of 0, with the background set to 1.

To quantify staining-style variability across the full cohort of 1,408 WSIs, we extracted stain matrices via sparse non-negative matrix factorization (SNMF) and computed optical density statistics for each WSI. A Gaussian mixture model (GMM) with the number of components selected by Bayesian Information Criterion (BIC) identified 101 stain-style clusters representing diverse chromatic profiles (Table 1).

The full cohort of 1,408 WSIs was randomly split patient-wise in an approximate 3:1:1 ratio while maintaining balanced cross-domain directions (X→Y and Y→X) for stain normalization and domain adaptation evaluation (Table 2). During model development, we utilized two stain-style domains: domain X contained patient-level staining variations, whereas domain Y represented a consistent reference style from a single patient. For the segmentation task, nuclei labels were paired with input patches from domains X and Y. The teacher model was first trained on the limited manually annotated 1,715 patches. The teacher then automatically generated pseudolabels for numerous additional unlabeled patches (500×2, 2,000×2, and 5,000×2), which were used to train the student model in a semi-supervised teacher-student framework. Both models were validated using the same independent testing set.

4.2. Evaluation Metrics and Comparison Methods

Our objective is to address staining style inconsistencies in pathological images by translating images from domain X to domain Y, thereby normalizing diverse staining appearances to a standardized style. To evaluate the quality of stain normalization, we compare the output images, ${\tilde{\mathrm{X}}}_{\mathrm{r}\mathrm{s}}$, with reference images from the Y domain, ${\mathrm{X}}_{\mathrm{r}\mathrm{e}\mathrm{f}}$, stained using the method proposed by Vahadane et al. [15]. We adopt three metrics for this assessment: Structural Similarity Index (SSIM), Root Mean Squared Error (RMSE), and Peak Signal-to-Noise Ratio (PSNR) [44]. For pixel-level evaluation of nuclei segmentation predictions, ${\tilde{\mathrm{X}}}_{\mathrm{s}\mathrm{e}\mathrm{g}}$, we employ Mean Intersection over Union (mIOU), F1-score, and Aggregated Jaccard Index (AJI), as proposed by Kumar et al. [45].

In our study, we benchmarked the performance of our proposed 3SGAN model against several established models in medical image analysis. These include traditional segmentation models such as U-Net [8], Attention U-Net [9], TransUNet [28], the Segment Anything Model (SAM) [31], and its medically optimized variant, MedSAM [32], as well as the image-to-image translation model CycleGAN [19]. U-Net, a foundational model for medical image segmentation, utilizes an encoder-decoder architecture with skip connections to enhance contextual feature learning. Attention-UNet extends U-Net by incorporating attention gates to prioritize salient features at coarser scales, improving target localization. TransUNet further augments U-Net with transformers to capture robust global contextual information. SAM, a versatile model pretrained on large-scale natural image datasets, excels in prompt-driven segmentation tasks but struggles with medical images due to domain differences. MedSAM [32], fine-tuned on a large-scale 2D medical image dataset, significantly improves segmentation accuracy for medical applications. CycleGAN, widely adopted for unpaired image-to-image translation, has been effectively applied to stain normalization tasks [46]. Our results demonstrate that 3SGAN outperforms these models in both stain normalization and segmentation tasks, achieving superior performance across the evaluated metrics.

4.3. Implementation Details

The teacher model (AttCycle) and student model (TransCycle) were developed using the PyTorch 1.11.0 framework and trained for 200 epochs with a batch size of 16. Input patches and labels were sized at 256×256 pixels, and the patches were augmented through scaling, random cropping, and flipping. The initial learning rates for both the student and teacher models were 0.0002, which decayed linearly to 0 starting from the 100th epoch. The discriminators for both models had a learning rate of 0.0002. The weights for the translation loss from the X domain to the Y domain (${\mathsf{\lambda }}_{\mathrm{X}2\mathrm{Y}}$), cycle-consistency loss (${\mathsf{\lambda }}_{\mathrm{C}\mathrm{y}\mathrm{c}}$), and segmentation loss (${\mathsf{\lambda }}_{\mathrm{S}\mathrm{e}\mathrm{g}}$) were set to 2, 15, and 20, respectively. Adam optimizers were used with ${\mathsf{\beta }}_1$=0.5 and ${\mathsf{\beta }}_2$= 0.999. In the proposed multitask network, the output channel number of the stain normalization branch is set to 3, and the output channel number of nuclei segmentation is set to 1.

To facilitate faster convergence of the student model during training, the weights of its transformer layers were initialized with those of a transformer model that was pre-trained on ImageNet [47], which is a large-scale dataset of natural images. This initialization allows the student model to leverage the knowledge learned from the pre-trained model and initialize its parameters close to the optimal values, which can speed up the learning process and improve performance.

4.4. Experimental Design and Results

In our work, we adopt a training scheme based on the teacher-student paradigm. We implement teacher and student models using the CycleGAN framework since the unavailability of pixel-level paired data. To demonstrate the effectiveness of our proposed approach in addressing the challenges of staining style differences and lack of pixelwise manual annotations, we conducted a series of experiments on the Joint Glioma dataset. Specifically, we compared traditional segmentation networks, normalization networks, and our proposed 3SGAN, as detailed in Table 3. Additionally, we studied the performance of multitask architecture and semisupervised teacher-student models on the Joint Glioma dataset within our proposed pipeline, revealing that our approach leads to further improvements in both tasks.

4.4.1. Superiority of the Proposed Multitask Network

To demonstrate the effectiveness of our proposed multitask network in the 3SGAN, we conducted an ablation study by removing the nuclei segmentation branch, resulting in the 2SGAN, a semisupervised stain normalization network. The teacher and student models in the 2SGAN were trained identically to those in the 3SGAN, except for the absence of the segmentation branch.

The results, presented in Table 3, reveal that the 2SGAN's performance was inferior to that of the 3SGAN, highlighting the advantage of the multitask architecture in achieving superior stain normalization quality. Furthermore, visual comparisons of stain normalization outcomes between 3SGAN and 2SGAN are illustrated in Figure 3, where each column displays the results of applying different normalization techniques to the same input image. Green arrows indicate variations in specific regions across the normalized images. Our findings demonstrate that 3SGAN more effectively preserves the original stylistic characteristics of certain regions in the input image compared to 2SGAN and CycleGAN.

4.4.2. Further Improvement Through the Teacher-Student Paradigm

To identify the most suitable teacher and student models for our proposed semisupervised framework, we conducted experiments evaluating various models within our designed multitask architecture. In this framework, the student model is expected to possess greater learning capacity than the teacher model when trained on large datasets. According to Cho and Hariharan [48], a higher number of model parameters correlates with enhanced learning capacity under conditions of abundant training data. To meet the requirements of our semisupervised paradigm, we compared the parameter counts of different models, all of which incorporate our multitask architecture and utilize the cycle-consistency loss from CycleGAN. These models—mCycle, ScCycle, and AttCycle—are distinguished by their unique dual-decoder generator designs.

The mCycle model enhances the single-encoder-single-decoder generator of CycleGAN by adopting a dual-decoder structure. Employing a hard parameter-sharing strategy in multitask learning, mCycle effectively leverages the complementary relationship between stain normalization and nuclei segmentation tasks. It is trained using a combination of supervised loss for nuclei segmentation and unsupervised loss for stain normalization, achieving outstanding performance even with limited annotated data for both tasks.

Building on mCycle, ScCycle also adopts a hard parameter-sharing multitask learning framework to capitalize on the synergy between stain normalization and nuclei segmentation. Additionally, ScCycle incorporates skip connections to better utilize multi-level feature information, thereby improving the quality of multitask outcomes. These technical advancements enable ScCycle to achieve superior accuracy in both stain normalization and nuclei segmentation, even with small-scale annotated datasets.

Figure 4. The structures of generators in ScCycle and mCycle. Either of them has a shared encoder and dual decoders. The generator of ScCycle integrates skip connections into mCycle’ generator.

Figure 4. The structures of generators in ScCycle and mCycle. Either of them has a shared encoder and dual decoders. The generator of ScCycle integrates skip connections into mCycle’ generator.

The selection of suitable teacher and student models is pivotal to the success of our semisupervised learning framework. In all designed multitask models, the discriminators share an identical structure. Consequently, our comparison of parameter counts focused solely on the generator parameters of each model. For the 3SGAN, we selected TransCycle as the student model due to its generator's high parameter count of approximately 116 million, as shown in Table 4. This substantial parameter capacity enables the student model to achieve optimal performance in stain normalization and nuclei segmentation tasks when provided with sufficient pseudolabeled data.

Based on the findings presented in Table 4, we selected AttCycle as the teacher model for 3SGAN, as it generates high-quality pseudostaining normalization results and nuclei segmentation pseudolabels, effectively supporting our semisupervised learning framework. To further enhance the framework’s performance, we trained U-Net, attention-UNet, and TransUNet using the same nuclei-labeled data and trained CycleGAN with images from both source and target staining style domains.

Overall, the careful selection of teacher and student models in 3SGAN, combined with the generation of high-quality pseudolabeled data, is critical to the success of our proposed framework for medical image analysis. This approach has the potential to overcome the limitations of traditional supervised learning methods, enabling accurate analysis of medical images with limited labeled data.

Table 5 demonstrates that our proposed semisupervised multitask network, 3SGAN, surpasses its baseline models, AttCycle and TransCycle, in both nuclei segmentation and stain normalization, owing to its effective teacher-student paradigm. This semisupervised framework significantly enhances performance in both tasks. All multitask networks we designed—AttCycle, TransCycle, ScCycle, and mCycle—outperformed traditional state-of-the-art (SOTA) networks in nuclei segmentation and stain normalization.

The results, illustrated in Figure 5 and Figure 6, show that 3SGAN achieves the highest segmentation accuracy while producing stain normalization outputs that effectively preserve the original details of diagnostically critical regions, such as nuclei. Quantitatively, 3SGAN’s superior F1, mIoU, and AJI scores directly indicate minimal false-positive nuclei detection, as these metrics penalize over-segmentation. Our multi-task design inherently constrains the generator to maintain realistic nuclear structures, thereby greatly mitigating hallucinations compared to methods like standard CycleGAN.

4.4.3. Effect of Different Quantities of Pseudolabeled Data

To evaluate the influence of pseudolabeled data quantity on the student model's performance within the proposed 3SGAN framework, we conducted experiments using varying amounts of unlabeled patches to generate pseudolabeled data, specifically 500×2, 2,000×2, and 5,000×2 patches. The results, shown in Table 6, indicate that increasing the volume of pseudolabeled data enhances the 3SGAN framework's performance in both stain normalization and nuclei segmentation tasks. However, a saturation point exists beyond which further increases in dataset size yield limited performance gains.

4.5. External Validation

To rigorously assess the generalization ability and robustness of 3SGAN beyond its training distribution, we conducted comprehensive external validation in two complementary settings: (i) cross-dataset evaluation on two established public multi-organ nuclei segmentation benchmarks, and (ii) testing on previously unseen, tumor-rich non-ROI regions extracted from our in-house WSIs. We jointly report nuclei segmentation performance (F1 score, mean IoU, and Aggregated Jaccard Index – AJI) and stain normalization quality (RMSE, PSNR, and SSIM), thereby providing a unified evaluation of structural fidelity and stain invariance.

4.5.1. External Validation on MoNuSeg and PanNuke

We first evaluated 3SGAN on the MoNuSeg [49] and PanNuke [50] datasets, which are widely recognized for their substantial inter-organ heterogeneity, diverse nuclear morphologies, and pronounced staining variations. These characteristics make them stringent testbeds for evaluating cross-cohort transferability.

For nuclei segmentation, we compared 3SGAN against classical fully supervised architectures (U-Net, Attention-UNet, TransUNet) as well as recent large-scale foundation models (SAM and MedSAM). For stain normalization, comparisons were made with representative CycleGAN-based and multitask baselines (CycleGAN, AttCycle, TransCycle, ScCycle, and mCycle). Quantitative results are consolidated in Table 7. Across both datasets, 3SGAN consistently achieves the highest segmentation scores (F1, mIoU, AJI) while simultaneously yielding the best stain normalization performance (lowest RMSE, highest PSNR and SSIM), substantially outperforming all competing methods.

4.5.2. Generalization to Non-ROI Regions on In-House WSIs

To further examine real-world clinical robustness, we assessed 3SGAN on manually curated tumor-rich regions that were intentionally excluded from both the annotated training set and the pseudo-labeling pipeline (non-ROI regions). These areas typically exhibit complex tissue architecture and were never exposed to the model during training.

Results are presented in Table 8. Even under this challenging out-of-distribution scenario, 3SGAN maintains superior segmentation accuracy and the strongest stain normalization quality, demonstrating that the proposed semi-supervised multitask framework effectively mitigates staining heterogeneity while preserving fine-grained nuclear structures.

Collectively, the consistent superiority observed across public benchmarks and clinically realistic non-ROI regions underscores the efficacy of 3SGAN’s unified semi-supervised multitask design in delivering state-of-the-art performance with remarkable domain invariance and structural preservation.

5. Discussion

In this section, we examine four pivotal design aspects of the proposed 3SGAN framework: (1) the selection of teacher and student models, (2) the relationship between pseudo-labeled dataset size and performance saturation, (3) ablation studies comparing self-training baselines with the semi-supervised teacher-student paradigm, and (4) the role of explicit stain normalization in the era of large foundation models. We also discuss key limitations observed in our experiments and provide insights into potential directions for future work.

5.1. Teacher and Student Model Selection

The choice of teacher and student models is critical to the success of our semisupervised framework. Beyond model capacity, preserving the structural integrity of target regions, such as nuclei, is a key consideration in model selection. For the stain normalization task, it is imperative that the model maintains nuclear features post-style translation. To evaluate this, we assessed three multitask models—AttCycle, ScCycle, and mCycle—by generating heatmaps of the output feature bypass to visualize their focus on target regions. As shown in Figure 7, AttCycle, equipped with attention gates, exhibited reduced focus on nuclear regions during stain normalization, thereby better preserving the original structure and color information. Additionally, AttCycle demonstrated superior performance in the nuclei segmentation task. Consequently, we selected AttCycle as the teacher model to generate high-quality pseudostaining normalization patches and corresponding nuclei pseudolabels. This strategic selection ensures that the student model benefits from high-quality pseudolabeled data, enhancing its performance in both nuclei segmentation and stain normalization.

5.2. Pseudolabeled Dataset Size Saturation Point.

The capacity of the implemented models influences their saturation points for pseudolabeled dataset size. As illustrated in Figure 8, the performance of 3SGAN improves with increasing pseudolabeled dataset size up to 2,000×2 patches. Beyond this threshold, further increasing the dataset to 5,000×2 patches yields no additional performance gains, indicating that 3SGAN reaches its saturation point at 2,000×2 patches. This suggests that the model’s learning capacity is fully utilized at this dataset size, and additional pseudodata provides negligible benefits.

5.3. Self-Training Ablation Experiments on Teacher and Student Models

When the teacher and student models share the same architecture and parameters (both based on AttCycle), the semisupervised strategy reduces to self-training. In this scenario, the teacher model generates pseudolabeled data and subsequently trains itself with this data. We conducted a statistical analysis to compare the saturation point performance gains of the semisupervised teacher-student paradigm and the self-training approach. The results, presented in Table 9, show that self-training requires 5,000×2 pseudolabeled patches to reach its saturation point, compared to 2,000×2 for the teacher-student paradigm. Despite utilizing more pseudodata, self-training yields only marginal performance improvements compared to the teacher-student paradigm. This is likely due to the amplification of errors and noise in self-training, as the same model iteratively refines its own predictions. In contrast, the teacher-student paradigm leverages distinct models, mitigating error propagation and achieving superior performance.

5.4. Relevance of Stain Normalization in the Era of Foundation Models

Although foundation models such as SAM and MedSAM are pretrained on vast and heterogeneous corpora, they remain vulnerable to performance degradation when exposed to rare staining protocols, atypical pre-analytic conditions, or site-specific chromatic variations that are underrepresented in their training data. As demonstrated in Table 5, 3SGAN consistently surpasses both SAM and MedSAM across F1, mIoU, and AJI on our nuclei segmentation benchmark, highlighting the continued efficacy of an explicit stain-normalization-plus-segmentation paradigm in domains characterized by pronounced stain variability.

From a methodological perspective, stain normalization retains considerable value when integrated with foundation models. By standardizing input color distributions, it effectively mitigates local domain shifts, reduces reliance on extensive fine-tuning in label-constrained scenarios, and enables the deployment of lightweight, task-specific pipelines (e.g., 3SGAN) that are readily adaptable to resource-limited or regulatory-constrained clinical environments. Moreover, it functions as a transparent and interpretable preprocessing module that harmonizes inputs prior to foundation-model inference, thereby enhancing generalization without costly retraining while preserving mechanistic insight into stain-induced effects on downstream predictions.

Thus, far from being supplanted, stain-normalization-based frameworks exemplified by 3SGAN remain highly relevant and complementary within the contemporary landscape dominated by foundation models.

6. Conclusions

In this study, we introduced 3SGAN, a novel semi-supervised multitask framework designed to simultaneously address two major challenges in digital pathology: stain style normalization and nuclei segmentation under severely limited pixel-level annotations. Trained and rigorously evaluated on a large-scale glioma H&E dataset from two institutions—characterized by extensive staining variability and minimal nucleus-level labels—3SGAN effectively alleviates staining inconsistencies while substantially improving segmentation performance and preserving diagnostically critical structural details.

By leveraging a semi-supervised teacher-student paradigm, our approach overcomes the constraints imposed by scarce annotations. Comprehensive ablation studies and comparisons with state-of-the-art methods confirm the superiority of 3SGAN over both traditional supervised segmentation models and contemporary foundation models. External validation on the MoNuSeg and PanNuke datasets further demonstrates excellent generalization beyond glioma histopathology, underscoring the framework’s ability to robustly handle staining heterogeneity and complex nuclear morphologies with limited annotated data.

Despite these promising results, rigorous benchmarking on additional challenging histologies (e.g., lymphomas and epithelial malignancies exhibiting extremely dense, small, or highly pleomorphic nuclei) remains an important direction for future work. We also plan to quantitatively assess the preservation of diagnostically relevant structural features during stain normalization and to conduct expert pathologist evaluations of normalization outcomes for clinical acceptability. Finally, refining the pseudo-labeling process to better suppress noise in teacher-generated annotations will be critical for further enhancing the robustness and reliability of the student model.

Overall, 3SGAN represents a significant step toward practical, annotation-efficient computational pathology pipelines capable of delivering consistent, high-quality nuclei segmentation and stain-invariant analysis across diverse clinical settings.