Cross-Model SHAP Agreement in Network Intrusion Detection: Consensus, Divergence, and Dataset Artifacts

Abstract

Context. Tree-based ensemble methods consistently rank among the best-performing classifiers for network intrusion detection systems (IDS), and explainability tools such as SHAP are increasingly used to interpret their predictions. However, most studies apply SHAP to a single model in isolation, leaving open the question of whether different models actually agree on the features that drive their decisions. Objective. We address this gap by quantifying cross-model SHAP agreement on UNSW-NB15 and re-framing it as a diagnostic tool for detecting dataset artifacts that single-model interpretation cannot surface. Method. This paper applies SHAP TreeExplainer to XGBoost, LightGBM, and RandomForest trained on the official UNSW-NB15 partition and measures pairwise inter-model consensus using Spearman rank correlation, Kendall τ, and top-k feature overlap, with 95% bootstrap confidence intervals (B = 1,000) on every agreement metric. Results. XGBoost and LightGBM exhibit strong agreement (ρ = 0.861, 100% top-5 overlap), while both boosting models agree only moderately with RandomForest (ρ ≈ 0.65). All three models rank sttl (source-to-destination TTL) as the most important feature by a wide margin, a finding that reveals a shared dependence on a network-layer artifact of the UNSW-NB15 lab topology rather than a generalizable attack signature. An ablation experiment confirms that removing sttl does not degrade accuracy (Δ ≤ 0.04% for all three models), because the artifact is distributed across a cluster of correlated TTL features. Adding a multi-layer perceptron separates two effects that accuracy and single-model interpretation conflate: rank agreement halves outside the tree family (ρ = 0.480–0.536), yet three of the MLP's five top-ranked features remain TTL-cluster members. Consensus is therefore partly a property of the model family; the artifact is a property of the dataset. Permutation importance, an attribution method independent of SHAP, identifies the same cluster (permuting it costs 22–28 accuracy points against 1.4 for random features), and of the 56,157 flows with sttl = 31 not one is an attack. Implication. We argue that cross-model SHAP agreement should become a standard diagnostic in IDS evaluation: consensus on a small feature set is not evidence of robustness if the consensus features lack semantic validity; pairwise agreement metrics together with semantic inspection of consensus features should be reported alongside accuracy in every SHAP-based IDS study.