Multi-level deep generative model with poisson variational autoencoders and reinforcement learning for enhanced intrusion detection system

Abstract

Network intrusion detection systems face two critical challenges: the need to identify specific attack types beyond binary classification and the ability to adapt to evolving threats while maintaining low false positive rates. In response, we propose a hierarchical multi-level Poisson Variational Autoencoder (PVAE) system augmented with reinforcement learning-based ensemble weighting for intrusion detection. At its core, a three-level PVAE chain with normalizing flows captures network traffic patterns at packet, flow and session levels. In our research, we applied class-conditional radial recalibration fitted on validation data to align the latent space separately for each attack class. Rather than training multiple separate models, we create ensemble diversity by instantiating three architectural variants from a single trained checkpoints. A Proximal Policy Optimization (PPO) agent then learns dynamic persample weights based on prediction confidence and other patterns. Finally, an XGBoost meta-learner refines the PPO-weighted ensemble outputs using uncertainty diagnostics to produce the final classification. Evaluated on the NF-UQ-NIDSv2 dataset with 14 traffic classes, our system achieves 89.17% multiclass accuracy with 88.01% weighted F1-score, while maintaining strong binary discrimination with 98.11% AUROC and 99.11% PR-AUC for normal versus attack detection.