Penerapan Algoritma Isolation Forest dan Metode Rule Based untuk Deteksi Serangan Brute Force

Abstract

Brute force attacks remain one of the most common cyber threats targeting network authentication services such as Secure Shell (SSH), File Transfer Protocol (FTP), and web-based login systems. This type of attack is performed by repeatedly attempting various combinations of usernames and passwords until valid credentials are obtained. Brute force activities are often difficult to distinguish from legitimate network traffic because they exhibit communication patterns similar to normal user behavior. Therefore, an automated detection approach is required to identify abnormal patterns from network log data. This study implements a brute force attack detection system based on network log analysis using the Isolation Forest algorithm and a Rule-Based method. The dataset used in this research consists of network traffic logs captured using Wireshark and exported in CSV format. Data preprocessing was conducted to standardize log structure, convert timestamps into numerical values, and extract additional features, including packet count per source IP address. The Isolation Forest algorithm was applied as an unsupervised anomaly detection method, enabling the identification of abnormal network activities without requiring labeled data. Subsequently, a Rule-Based method was employed as a verification stage to classify detected anomalies as brute force attacks based on predefined rules, such as the presence of specific keywords in log information fields, repeated login attempts, and defined time windows. The experimental results indicate that the Isolation Forest algorithm effectively identifies anomalous network activities that deviate from normal traffic patterns. The application of the Rule-Based method further refines the detection results by filtering anomalies that exhibit brute force characteristics. The combination of both methods produces more specific and interpretable detection outcomes, as demonstrated by the identification of repeated failed login attempts originating from the same source IP within a short time interval. Detection results are presented through tabular outputs and visualizations to support further analysis.