On the impact of dataset size and class imbalance in evaluating machine-learning-based windows malware detection techniques
원문 초록
The purpose of this project was to collect and analyse data about the comparability and real-life applicability of published results focusing on Microsoft Windows malware, more specifically the impact of dataset size and testing dataset imbalance on measured detector performance. Some researchers use smaller datasets, and if dataset size has a significant impact on performance, that makes comparison of the published results difficult. Researchers also tend to use balanced datasets and accuracy as a metric for testing. The former is not a true representation of reality, where benign samples significantly outnumber malware, and the latter is approach is known to be problematic for imbalanced problems. The project identified two key objectives, to understand if dataset size correlates to measured detector performance to an extent that prevents meaningful comparison of published results, and to understand if good performance reported in published research can be expected to perform well in a real-world deployment scenario. The research's results suggested that dataset size does correlate with measured detector performance to an extent that prevents meaningful comparison of published results, and without understanding the nature of the training set size-accuracy curve for published results conclusions between approaches on which approach is "better" shouldn't be made solely based on accuracy scores. Results also suggested that high accuracy scores don't necessarily translate to high real-world performance.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers
LAMDA: A Longitudinal Android Malware Benchmark for Concept Drift Analysis
Evaluating the Robustness of Adversarial Defenses in Malware Detection Systems
Cyber Security Data Science: Machine Learning Methods and their Performance on Imbalanced Datasets
Imbalanced malware classification: an approach based on dynamic classifier selection
CyberLLMInstruct: A New Dataset for Analysing Safety of Fine-Tuned LLMs Using Cyber Security Data
토론
이 논문과 연결된 포럼 글이 아직 없습니다.