Exploring Architectural Enhancements for HoVer-Net: Deep Learning-based Method of Automated Nuclear Segmentation and Classification

1. Introduction

Nuclear instance segmentation is a fundamental task in computational pathology, underpinning a range of downstream applications such as cell classification, tissue phenotyping, and tumor microenvironment analysis  [1]. Accurate delineation of individual nuclei is critical for automated diagnosis and quantitative analysis in histopathological images, which are often challenged by dense cellular arrangements, overlapping nuclei, and diverse morphological variations  [2].

Conventional image processing techniques have shown limited robustness in these complex settings due to their sensitivity to occlusion and shape variability. In contrast, deep learning approaches—particularly convolutional neural network CNN-based models like U-Net  [3] and nnU-Net  [4]—have greatly advanced performance by learning semantic features directly from data. Among these, HoVer-Net  [5] has emerged as a state-of-the-art multi-task architecture capable of simultaneously predicting nuclear pixel maps (NP), horizontal–vertical distance maps (HoVer), and nuclear type maps (TP). This design has established a strong benchmark for nuclear instance segmentation and classification, particularly on large-scale datasets such as PanNuke  [6].

However, despite its effectiveness, HoVer-Net is built upon a residual CNN encoder, which may constrain its ability to capture long-range dependencies and global context. Recent developments in Transformer-based models—such as the Vision Transformer (ViT)  [7], Swin Transformer  [8], and hybrid models like CellViT  [9]—have demonstrated powerful capabilities in modeling spatial relationships, which are especially important for the heterogeneous patterns seen in histopathology images  [10]. Nevertheless, the incorporation of Transformer backbones into multi-task segmentation frameworks like HoVer-Net remains relatively underexplored.

In this study, I present a systematic investigation into enhancing the HoVer-Net architecture through the integration of advanced modules and modern design principles. My key contributions are as follows:

• I investigate the integration of Squeeze-and-Excitation (SE) blocks  [11] into the original CNN-based encoder of HoVerNet, aiming to enhance channel-wise feature recalibration. This modification leads to measurable improvements in segmentation accuracy.
• I explore the replacement of HoVerNet’s residual CNN encoder with Transformer-based modules, including Vision Transformer (ViT) and Swin Transformer. A unified architecture is proposed to preserve compatibility with HoVerNet’s three-branch output structure.
• I design an enhanced decoder architecture that leverages dense connections and dropout regularization, facilitating improved information flow and mitigating overfitting risks during training.
• I conduct comparative experiments against the current state-of-the-art model, CellViT, and perform ablation studies to assess the individual contributions of each architectural component.
• All experiments are conducted on the PanNuke dataset, following consistent training protocols—such as fixed epoch count, learning rate, pretrained models, optimizer, and train-validation splits—to ensure fair evaluation. While not all proposed models surpass the original HoVerNet in every metric, the findings emphasize the value of attention mechanisms and architectural enhancements in advancing segmentation and classification performance.

2. Materials and Methods

2.1. Dataset

All experiments in this study were conducted on the publicly available PanNuke dataset, a large-scale benchmark specifically curated for nuclei instance segmentation and classification tasks. PanNuke contains pixel-wise annotated histopathology image patches spanning 19 distinct tissue types. Each nucleus within the dataset is labeled according to one of five predefined categories: neoplastic, inflammatory, connective, dead, and epithelial  [6].

The dataset was chosen for its comprehensive and high-quality annotations, as well as its established usage in prior research. These characteristics make PanNuke particularly suitable for evaluating multi-class nuclear segmentation methods and enable consistent comparisons with other state-of-the-art approaches.

2.2. Preprocessing and Augmentation

To ensure fair and consistent evaluation across all models, I adopted the preprocessing and augmentation procedures from the original HoVerNet pipeline. Input histopathology images were first normalized and resized, followed by padding to align with architectural constraints and to maintain spatial consistency across samples. Random shuffling of training data was applied to mitigate sampling bias and enhance the models’ generalization performance [12].

The PanNuke dataset was partitioned into training, validation, and testing subsets using a ratio of 2:6:2, ensuring a balanced distribution of tissue types and cell classes across splits.

2.3. Model Variants Based on HoVerNet Architecture

To investigate the effects of architectural modifications on nuclei instance segmentation and classification, we developed six variants of the original HoVerNet framework. Each variant retains HoVerNet’s distinctive three-head output structure—np_map, hv_map, and tp_map—which enables joint nuclear segmentation and classification [5]. The modifications focus on enhancing the encoder and decoder designs, and are detailed as follows (the pretrained model can be found in Appendix A and training code in Appendix B):

• HoVerNet + SE (HoverNetEnhanced): This variant augments the encoder with Squeeze-and-Excitation (SE) blocks, which are integrated within the residual units. SE blocks perform adaptive channel-wise recalibration, emphasizing informative features while suppressing less useful ones, thereby enhancing representational capacity [11].
• HoVerNet + Multi-head Attention (Multihead-HoverNet): In this model, multi-head self-attention (MHSA) modules [13] are embedded into the encoder to capture long-range dependencies and global contextual cues. Inspired by prior analysis of attention heads [14], this design seeks to improve performance on complex spatial structures.
• HoVerNet + SE + MHSA + Enhanced Decoder (MSDHV-Net): This is the most comprehensive CNN-based modification. It combines both SE and MHSA modules in the encoder and introduces a redesigned decoder featuring deeper DenseBlock structures, additional skip connections, and convolutional refinement layers. This design aims to facilitate robust feature propagation and enhanced spatial resolution restoration [15].
• HoVerNet + ViT Encoder (HoverViTNet): Here, the conventional CNN encoder is replaced with a Vision Transformer (ViT) [7], enabling the model to extract patch-wise global representations using self-attention. These transformer-derived features are decoded via a CNN-based decoder, allowing comparative evaluation of attention-driven global context modeling [16].
• HoVerNet + Custom SwinViT Encoder (HoVerIT): This architecture integrates a custom Swin Transformer encoder into the HoVerNet pipeline. The hierarchical design and shifted window self-attention in SwinViT capture both local and global dependencies more effectively than vanilla ViT [17]. The idea draws inspiration from the Swin-UNETR architecture [18], while maintaining compatibility with HoVerNet’s three-branch outputs.
• HoVerNet + SwinUNETR from MONAI (HoverSwinNet): In this variant, the encoder is directly replaced with the SwinUNETR backbone from MONAI [18]. Transformer-derived multi-scale features are routed through HoVerNet’s original three-branch decoders, serving as a strong baseline to assess the integration feasibility and performance of prebuilt transformer encoders.

2.4. Training Configuration

All models were trained using the Adam optimizer [19] with a fixed learning rate of 1e^-4. Pre-trained ResNet-50 weights [20] were used to initialize the encoder when applicable. Although alternative optimizers such as AdamW [21] were evaluated, they did not produce noticeable improvements in performance. Attempts to modify the learning rate led to unstable training dynamics, including instances of gradient explosion, thereby justifying the choice of a conservative fixed schedule.

Each model was trained for 80 epochs, based on empirical observations that validation performance typically saturated between epochs 50 and 60 (See Figure A1). A batch size of 16 was used to optimize memory utilization on an NVIDIA L20 GPU with 44 GB VRAM.

The loss functions followed the original HoVerNet formulation: binary cross-entropy (BCE) combined with Dice loss for nuclear pixel (np_map) and nuclear type (tp_map) predictions, and a joint loss of mean squared error (MSE) and mean squared gradient error (MSGE) for the horizontal-vertical distance (hv_map) regression task.

2.5. Evaluation Metrics

To comprehensively evaluate model performance, we employed several quantitative metrics: DICE coefficient, Panoptic Quality (PQ), Precision, F1-score, and Recall. These metrics were computed separately for each predicted cell type, enabling a fine-grained analysis of segmentation and classification performance. This multi-metric evaluation framework facilitated rigorous comparisons across different architectural variants and offered insight into the effectiveness and limitations of each proposed enhancement [22].

3. Results

3.1. Baseline Performance

To establish a reliable benchmark, I trained the original HoVer-Net architecture on the PanNuke dataset using the standardized training pipeline described earlier. The baseline model achieved an NP-Dice score of 0.8686, indicating strong segmentation accuracy for nuclear regions. For type prediction (TP), Dice scores were 0.9703, 0.7978, 0.6728, and 0.6939 across the four nuclear subtypes, reflecting varying levels of classification performance.

In terms of instance-level evaluation, Panoptic Quality (PQ) scores were 0.4005 for neoplastic, 0.2830 for inflammatory, and 0.4526 for other cell types. Additionally, standard classification metrics such as F1-score, Precision, and Recall were computed to provide a comprehensive view of the model’s strengths and weaknesses. These results serve as a foundational reference for comparing the performance of all proposed architectural variants in subsequent experiments.

3.2. CNN-Based Architectural Enhancements

Three key modifications were explored within the traditional CNN-based design framework: the incorporation of Squeeze-and-Excitation (SE) blocks [11], the integration of multi-head self-attention modules [13], and the use of an enhanced DenseBlock-based decoder. The most effective configuration, referred to as MSDHV-Net, combined all three enhancements.

Compared to the original HoVerNet, MSDHV-Net achieved consistent performance gains across multiple metrics. Specifically, the TP-Dice score for inflammatory nuclei increased from 0.6728 to 0.7162, while the TP-Dice score for the “other” cell category improved from 0.6939 to 0.7241. The NP-Dice and overall classification Dice scores also exhibited an average improvement of approximately 2% (See Table 1 and Table 2 for more details).

However, despite these enhancements, performance on neoplastic cells remained challenging. Both the PQ and F1 scores for this class experienced a slight decline of around 2% relative to the baseline, suggesting that additional strategies may be required to address the morphological variability and contextual ambiguity characteristic of neoplastic nuclei.

3.3. Transformer-Based Architectural Variants

The effectiveness of replacing the original CNN encoder with Transformer-based architectures was also evaluated. Three model variants were explored: HoverSwinNet, which directly employed the SwinViT encoder from the MONAI SwinUNETR implementation [18]; HoVerIT, which integrated a customized Swin Transformer encoder into the original HoVerNet three-branch framework; and HoverViTNet, which substituted the encoder with a vanilla Vision Transformer (ViT) [7].

All Transformer-based models underperformed compared to the CNN-based baseline. Among them, HoverViTNet demonstrated slightly better results than HoverSwinNet, but overall, the Transformer variants exhibited a consistent degradation of 3–5% across Dice, Panoptic Quality (PQ), and classification-related metrics (See Table 1 and Table 2 for more details). These results suggest that while Transformer encoders offer strong theoretical advantages in capturing long-range dependencies, their practical integration into multi-task nuclei segmentation frameworks remains a non-trivial challenge that warrants further investigation.

3.4. Summary of Comparative Performance

Among all proposed architectures, MSDHV-Net demonstrated the most consistent and meaningful improvements (See Figure 1 and Figure 2 for comparision), particularly for inflammatory and other nuclear types. The transformer-based models, while conceptually promising, require further tuning or hybridization to outperform well-optimized CNN backbones on the PanNuke dataset.

3.5. Formatting of Mathematical Components

Dice Score:

$$\mathrm{Dice}={\displaystyle \frac{2\times |X\cap Y|}{|X|+|Y|}}$$

Binary Cross-Entropy (BCE):

$$\mathrm{BCE}=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\left[y_{i}log(p_i)+(1-y_i)log(1-p_i)\right]$$

Mean Squared Error (MSE):

$$\mathrm{MSE}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{(y_i-p_i)}^2$$

Panoptic Quality (PQ):

$$\mathrm{PQ}={\displaystyle \frac{{\sum }_{(p,g)\in TP}IoU(p,g)}{|TP|+\frac{1}{2}|FP|+\frac{1}{2}|FN|}}$$

F1 Score:

$$\mathrm{F}1={\displaystyle \frac{2·\mathrm{Precision}·\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

Precision and Recall:

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}},\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

4. Discussion

This study aimed to investigate architectural enhancements to the HoVer-Net framework for simultaneous nuclei segmentation and classification [5]. My findings demonstrate that thoughtful modifications within the CNN paradigm—specifically the integration of Squeeze-and-Excitation (SE) blocks, multi-head attention, and an enhanced DenseBlock decoder—can lead to measurable gains across multiple evaluation metrics. The best-performing model, MSDHV-Net (See Figure 3 for architecture), showed improved TP-Dice scores for Neoplastic, inflammatory and other nuclei types, indicating enhanced discriminative power in challenging classification scenarios.

These improvements are consistent with prior research emphasizing the importance of adaptive channel recalibration (as in SE blocks) and attention mechanisms in deep feature extraction for biomedical imaging tasks [11]. The enhanced decoder appears to better capture spatial dependencies and refine instance boundaries, especially in complex tissue environments.

In contrast, my exploration of Vision Transformer (ViT and SwinViT) encoders—although theoretically appealing due to their global receptive field—did not yield superior performance. HoverSwinNet and HoVerIT (See Figure 4 for architecture) underperformed compared to both the baseline and CNN-enhanced models. One likely reason is that transformer-based models may require significantly larger training data or more domain-specific pretraining to outperform CNNs in medical imaging, for example the SAM-ViT pretrained model used in CellVit [9]. Another limitation lies in integrating transformer encoders into multi-task frameworks like HoVer-Net, which may require carefully aligned intermediate feature representations.

Figure 4. Overview of the HoVerIT architecture.

Figure 4. Overview of the HoVerIT architecture.

Figure 5. Segmentation and classification results of three different models on multiple test patches.

Figure 5. Segmentation and classification results of three different models on multiple test patches.

Interestingly, all models struggled to improve classification metrics for neoplastic nuclei. This suggests either intrinsic ambiguity in their visual features or insufficient discriminatory signal in current feature representations. Future work could explore class-specific loss weighting or incorporate cell microenvironment context to enhance neoplastic classification.

Looking forward, promising directions include hybrid CNN–transformer architectures, domain-adaptive pretraining strategies, and exploring self-supervised representation learning to leverage unlabeled histopathology data. Additionally, integrating spatially aware attention mechanisms and refining decoder design may further enhance both instance segmentation and fine-grained classification capabilities.

Overall, this work highlights the value of selectively integrating modern deep learning components into established architectures, balancing innovation with task-specific constraints in computational pathology.

5. Conclusions

In this work, I systematically investigated architectural enhancements to the HoVer-Net framework [5] for nuclei instance segmentation and classification on the PanNuke dataset. Through the integration of SE blocks [11], multi-head attention, and a more expressive decoder, I developed MSDHV-Net, which consistently outperformed the original HoVer-Net in both segmentation and classification tasks—particularly for inflammatory and other nuclei types.

In contrast, transformer-based variants such as HoverSwinNet and HoVerIT did not demonstrate improved performance, suggesting that CNN-based backbones remain more robust and effective under the current dataset and training conditions. These results underscore the value of carefully engineered improvements to established CNN architectures, while also highlighting that successful transformer integration—such as those attempted in CellViT with SAM-ViT pretraining [9]—requires further domain adaptation and architectural refinement.

Overall, these findings provide practical insights for designing more accurate and efficient nuclei analysis models and lay a foundation for future work exploring hybrid CNN-transformer architectures and self-supervised learning approaches in computational pathology [21].