Hierarchical Sparse Neural Networks for Structure-Aware Ransomware Detection Under Distribution Shift

Behavioral ransomware detection often achieves high accuracy in standard evaluations; however, these results frequently fail to generalize under distribution shifts or when encountering previously unseen families. This study evaluates detection performance on the MLRan dataset (4,880 samples across 64 families) using four rigorous evaluation protocols: stratified, temporal, family-disjoint, and open-set. To ensure a strict separation of learned features, the family-disjoint and open-set splits were executed at the family level. We propose the Hierarchical Sparse Neural Network (HSNN), a taxonomy-aligned model with group-level and branch-level gating for structured interpretability. Unlike flat architecture, HSNN introduces a hierarchical gating mechanism aligned with a predefined behavioral taxonomy, enabling structured interpretability and modality-level analysis. The baseline FlatMLP had a slightly higher average macro-F1 score (0.9860 vs. HSNN's 0.9839), but the HSNN was better calibrated and more parameter efficient. The HSNN reduced calibration error by 34.1% (absolute reduction of 0.0056 in ECE) and model complexity by 42% in terms of parameter count. HSNN showed slightly lower variability than FlatMLP and broadly stable gate patterns across seeds. The proposed HSNN achieved one of the highest performances under the paper’s open-set family protocol (0.9930 vs. 0.9913) using a maximum-softmax novelty baseline. Our feature analysis shows that string-based artifacts remain strong predictors, but the HSNN’s hierarchical structure encourages a more balanced weighting across behavioral modalities, reducing reliance on any single feature type. These results indicate that structured, sparse architecture presents a competitive and well-calibrated alternative to conventional dense models under the evaluated settings.