A robust ensemble learning framework for binary and multiclass malware classification over diverse datasets

Abstract

Cybercrime has surged due to the widespread use of the Internet, posing a significant threat to global digital security, with obfuscated malware presenting a particular challenge through its sophisticated code-hiding techniques. This study addresses the classification of obfuscated malware using the CIC-MalMem-2022 dataset, comprising 29,298 memory dump samples across benign instances and multiple malware families (e.g., Spyware, Ransomware, Trojan Horse), alongside three additional datasets for testing our model. Our primary objective is to create a model that enhances the accuracy of multi-class classification, focusing on malware families, while also evaluating binary and malware category classifications. To mitigate class imbalance, we employ the Random UnderSampler technique, paired with feature selection using feature importance to identify discriminative memory-based features. The highest accuracy achieved was 99.99% for binary classification. Besides,for multiclass classification, we obtained approximately 96% and 94% (balanced dataset) for 4-class classification using RandomForest-LightGBM, and 99% and 99.99% (balanced dataset) for 16-class classification using AdaBoost-LightGBM on the primary dataset. We evaluated a range of machine learning (ML) models, including AdaBoost, Decision Tree, Random Forest, and hybrid ensembles with confidence-based refinement, among which AdaBoost-LightGBM and RandomForest-LightGBM are our proposed models.