A Hierarchical Spatial-Temporal CNN-BiLSTM Hybrid Model for Brute-Force Attack Detection in High-Speed Networks

Abstract

As computer networks become faster, cyberattacks – particularly SSH and FTP brute-force attacks – have become more sophisticated, exposing limitations in traditional detection systems, including high false positive rates. This study proposes a hierarchical hybrid deep learning model integrating Convolutional Neural Networks (CNN) for spatial feature extraction and Bi-directional Long Short-Term Memory (Bi-LSTM) for temporal analysis. Principal Component Analysis (PCA) reduced 82 features to 18 key attributes, improving computational efficiency. The model was implemented using a GPU-enabled TensorFlow framework and evaluated on CIC-IDS 2017 and CSE-CIC-IDS 2018 datasets. Results show that the hybrid CNN–Bi-LSTM model outperforms standalone approaches, achieving 99.27% accuracy, 99.89% precision, 98.19% F1-score, and 97.84% recall, with a low false positive rate of 0.018%. Reliability analysis using Monte Carlo Dropout yielded 92.3% predictive certainty, while a Dietterich 5x2cv paired t-test confirmed statistically significant improvement over the HAST-IDS baseline. These findings demonstrate a scalable and high-accuracy approach for detecting brute-force attacks in modern network environments.