Cyberattack Dataset Engineering and Benchmarking for Realistic Intrusion Detection: A Survey

Intrusion detection research depends on datasets. However, dataset design is often treated as secondary to model selection. In this survey, we examine cyberattack dataset engineering and benchmarking for realistic intrusion detection system (IDS) evaluation. Specifically, we review dataset releases, benchmark critiques, and domain-specific detection studies that cover general-purpose network benchmarks, Android spyware, multi-step and APT-inspired attacks including SUNBURST, smart-grid injection, SNMP-MIB and DDoS monitoring, IoT botnets, phishing, streaming traffic, and GAN-based synthetic attack data. We argue that dataset realism must be judged through collection context, attack procedure, benign behaviour, labeling, feature extraction, temporal splitting, privacy, reproducibility, and benchmark saturation, rather than through dataset size or reported accuracy. To this end, we position the survey against prior IDS and dataset reviews, introduce a six-dimensional dataset-centred taxonomy (attack environment, data representation, collection method, label strategy, evaluation protocol, and threat model), compare seven dataset families, and consolidate an evaluation protocol that covers split policy, leakage checks, baselines, metrics, and dataset documentation. We conclude that next-generation IDS datasets should be released with dataset cards, scenario cards, leakage reports, baseline scripts, chronological and holdout splits, adversarial and zero-day partitions, and explicit limitations. This shift would make IDS results more comparable, make inflated performance claims easier to detect, and support realistic evaluation for IoT, 5G, smart grids, Android malware, phishing, and APT-inspired attacks.