Hossain, Muhammad IqbalArafath, YeasinHossain, AponNawreen, Nazihah IslamTalukder, Tousiqul IslamAziz, Raisa Tahiatul2025-09-152025-09-1520252025-06ID 20201052ID 20301355ID 24141096ID 21301679ID 20301426http://hdl.handle.net/10361/26728Cataloged from PDF version of thesis.Includes bibliographical references (pages 62-65).This thesis is submitted in partial fulfillment of the requirements for the degree of Bachelor of Science in Computer Science, 2025.Cybercrime has surged due to the widespread use of the Internet, posing a significant threat to global digital security, with obfuscated malware presenting a particular challenge through its sophisticated code-hiding techniques. This study addresses the classification of obfuscated malware using the CIC-MalMem-2022 dataset, comprising 29,298 memory dump samples across benign instances and multiple malware families (e.g., Spyware, Ransomware, Trojan Horse), alongside three additional datasets for testing our model. Our primary objective is to create a model that enhances the accuracy of multi-class classification, focusing on malware families, while also evaluating binary and malware category classifications. To mitigate class imbalance, we employ the Random UnderSampler technique, paired with feature selection using feature importance to identify discriminative memory-based features. The highest accuracy achieved was 99.99% for binary classification. Besides,for multiclass classification, we obtained approximately 96% and 94% (balanced dataset) for 4-class classification using RandomForest-LightGBM, and 99% and 99.99% (balanced dataset) for 16-class classification using AdaBoost-LightGBM on the primary dataset. We evaluated a range of machine learning (ML) models, including AdaBoost, Decision Tree, Random Forest, and hybrid ensembles with confidence-based refinement, among which AdaBoost-LightGBM and RandomForest-LightGBM are our proposed models.68 pagesenBRAC University theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission.Cyber crimeObfuscated malwareSpywarePolymorphic malwareMachine learningHybrid modelsEnsemble learning frameworkEnsemble learning (Machine learning).Computer networks--Security measures.Malware (Computer software).Cyberspace--Security measures.Computer security.Thesis