Do HTTP Features Carry the Signal? Shortcut Learning and Label Errors inWeb Attack Detection on CIC-IDS2017 and CSIC 2010

1. Introduction

Machine learning for web attack detection is a crowded field, and one of its most common proving grounds is the CIC-IDS2017 dataset [1]. Very high scores on a fixed benchmark can arise from issues unrelated to the stated contribution, among them duplicated records, label errors, operating points chosen after the fact and features that act as shortcuts [2] because they correlate with how a capture was produced rather than with the attack. Audits of CIC-IDS2017 have documented duplicated flows and labelling mistakes [3,4], and methodological work has catalogued the pitfalls that inflate apparent performance [5] and called for pragmatic evaluation [6].

This study examines a concrete design from the SentinelSphere platform [7,8,9], a deep neural network over 21 standard flow statistics and 10 HTTP aware features, among them a request complexity score, a path entropy estimate and binary flags for SQL injection and cross site scripting patterns, reported to reach a 94% F1 score with a large reduction in false positives on CIC-IDS2017. It asks whether the HTTP features contribute on the flow benchmark, whether the reported score is reproducible from the described model, and how well the features work on data that genuinely contains HTTP requests. A companion study applied the same kind of scrutiny to the speedup reported for the Rust rewrite of the threat scorer of the platform [10].

2. Method

The study uses the web attack capture of CIC-IDS2017 in its flow export form, with 170,366 flow records of which 2,180 are web attacks, the corrected release of the same traffic by Engelen et al. [3], which separates successful attacks from unsuccessful attempts, and the CSIC 2010 HTTP dataset [11], which contains labelled normal and anomalous requests with their bodies. The network is reimplemented as described, two hidden layers of 64 units, dropout of 0.2, L2 regularisation of 0.001, the Adam optimiser at a learning rate of 0.001 and a decision threshold of 0.4, and the ten HTTP features from the published extractor. As reference points, a random forest, histogram gradient boosting and logistic regression are used. Every configuration runs on five stratified splits with fixed seeds, and per class detection uses five fold out of fold prediction with Wilson intervals [12]. The study tests the effect of three shortcut candidates, the forward and backward initial TCP window sizes and the flow byte rate, ranks features by their individual area under the curve, and on CSIC 2010 compares the ten features with 24 handcrafted request features and a character ngram model, with a drop one ablation of the ten features.

3. Results

On the flow export the ten HTTP features are zero for every record, so adding them leaves the detector unchanged, with an F1 of 0.123 with the features present as zeros and 0.112 without them. Table 1 reports the reassessment. The network as described labels almost everything benign at the conventional threshold, with a recall of 6%, and reaches an F1 of only 0.36 at its stated threshold of 0.4, while inverse frequency weighting raises recall to 97% but collapses precision to 0.23. In out of fold scoring it detects none of the 21 SQL injection flows and fewer than half of the brute force and cross site scripting flows (Figure 1). A high score is reachable only with a different model, a random forest on all features reaching 0.988 and gradient boosting 0.996.

Much of the apparent skill is a dataset artefact. The backward initial TCP window size alone has an area under the curve of about 0.94, and a depth three decision tree on that single feature reaches 0.985, with an F1 of 0.83. Removing the three shortcut candidates lowers the random forest F1 from 0.988 to 0.948, the signature of shortcut learning [2]. The corrected labels change the target as well, since only 104 of 2,056 web attack flows are successful attacks. When attempts are counted as attacks a random forest reaches an F1 of 0.992 and the network 0.985, but when the detector must find only the 104 successful attacks the F1 falls to 0.940 and 0.909 with the attempts counted as benign, and to 0.927 and 0.873 with the attempts removed, so a single score on the original labels conflates an easy task with a harder one. A random forest on the 21 flow features trained on the original labels flags all 73 successful brute force and all 18 successful cross site scripting flows of the corrected release but only 1 of 1,947 brute force and cross site scripting attempts, so it keys on the flow signature of a completed attack.

On CSIC 2010 the ten features reach an F1 of 0.586 on the request line alone and 0.908 only once the body is included and decoded, while 24 handcrafted features reach 0.948 and a character ngram model 0.967 with the lowest false positive rate (Figure 2). Removing the path entropy feature costs about 0.15 in F1, whereas removing any of the attack pattern flags changes F1 by less than 0.003, and up to five of the ten features, depending on the input, are constant on this dataset because it carries no response code, no response size and a single user agent.

4. Discussion and Conclusions

The HTTP features cannot contribute on the flow export because the data they need is absent, the described network does not reach the reported score at a standard operating point, a high score rests partly on shortcut features, most benchmark attacks are attempts, and on genuine requests a generic entropy feature and a plain ngram model do the work. None of this says that web attack detection underperforms. It says that evaluation practice matters [5,6]. For SentinelSphere the constructive consequences are concrete, the HTTP features should be evaluated on request logs, detection scores on CIC-IDS2017 should be reported against the corrected labels with the successful attack task shown separately, and a check that no single flow metadata feature carries the result should accompany any headline number.