Botnet Detection based on Anomaly and Community Detection

Abstract— A novel two-stage approach for the important cyber-security problem of detecting the presence of a botnet and identifying the compromised nodes (the bots), ideally before the botnet becomes active. The first stage detects anomalies by leveraging large deviations of an empirical distribution. We propose two approaches to create the empirical distribution: a flow-based approach estimating the histogram of quantized flows, and a graph-based approach estimating the degree distribution of node interaction graphs, encompassing both Erd˝os-R´enyi graphs and scale-free graphs. The second stage detects the bots using ideas from social network community detection in a graph that captures correlations of interactions among nodes over time. Community detection is done by maximizing a modularity measure in this graph. The modularity maximization problem is non-convex. We propose a convex relaxation, an effective randomization algorithm, and es-tablish sharp bounds on the suboptimality gap. We apply our method to real-world botnet traffic and compare its performance with other methods. A botnet is a network of compromised computers controlled by a “botmaster.” Botnets are typically used for Distributed Denial-of-Service (DDoS) attacks, click fraud, or spamming. < final year projects >