HACOE: hierarchical attack classification with outlier exposure

Abstract

Traffic classification is critical for network security, particularly in identifying and mitigating malicious network attacks. With the rapid progress of network technologies, the emergence of new types of network applications (unseen applications) can pose significant challenges to traffic classification methods. Additionally, the increasing prevalence of encrypted traffic due to concerns about privacy and data security further complicates the detection of unprecedented and unseen cyberattacks. Although machine learning-based approaches have demonstrated enhanced accuracy in handling complicated network patterns, identifying unseen attacks primarily relies on unsupervised methods or limited observations of new attack examples. We introduce a novel approach that combines hierarchical traffic classification with outlier exposure techniques (HACOE) to address these challenges. This approach enables the identification of unseen attacks without the need for prior exposure to specific attack data. By enhancing the calibration of neural network confidence through outlier exposure, HACOE distinguishes unseen attacks as a separate class while identifying benign and known attack types. Our experimental results show the effectiveness of HACOE in detecting unseen attacks; HACOE identifies up to 50% of unseen attacks while incorrectly classifying only 4-18% of benign instances as unseen. In addition, under the same setting for the existing zero-day detection baselines, HACOE demonstrates better or comparable performance while providing accurate classification results for known attacks.