Enhanced Cellular Detection in Cervical Cytopathology: A Systematic Study of YOLOv11 Training Paradigms

Automated cellular detection using deep learning is a key strategy for optimising cervical cancer screening by reducing the healthcare workload and inter-observer variability. However, analyzing Whole Slide Image (WSI) patches presents challenges like annotation scarcity, morphological complexity, and class imbalance. This study conducts a systematic evaluation of YOLOv11 (n, s, and m variants) to assess the impact of target variable granularity and training paradigms on performance. Four strategies were analyzed: independent and multi-class models, each evaluated at both specific cell label and diagnostic macro-group levels. To ensure clinical robustness, patient-level data partitioning was implemented to prevent information leakage. Performance was measured using precision, recall, and mAP (0.5 and 0.5:0.95). The results reveal critical trade-offs between fine-grained discrimination and model generalization when varying architectural complexity and labeling strategies. Findings indicate that diagnostic aggregation improves stability, while single-class training optimizes specialized detection. These results provide methodological guidelines for designing AI-assisted screening systems and establish a foundation for integrating YOLOv11 detectors into Multiple Instance Learning (MIL) frameworks at the WSI scale.