Scalable Analytics for Anomaly Detection in the Cloud
The anomaly detection service contains two analytics engines: Skydive Anomaly detection engine and Attack detection engine. The Skydive Anomaly detection engine monitors data provided by Skydive network probes and applies the network related analytics as illustrated in the table below.
The detected network anomalies are stored in FINSEC Data Tier as instances of “x-event” FINSTIX object, where each detected “x-event” belongs to a specific anomaly type (e.g., “Data leakage” anomaly type). The Attack detection engine monitors events produced by FINSEC probes and analytics. It correlates these events according to the modelled attacks stored in Data Tier using FINSTIX “x-attack” model. The detected attacks are stored to FINSEC Data Tier as instances of “x-attack” FINSTIX object, where each detected “x-attack” belongs to a specific attack type.
The Anomaly Detection service is composed of two main components (see figure below):
- External Anomaly Detection (EAD): this is an external component running outside the FINSEC platform that accesses the FINSEC platform through the Anomaly Detection Service. The main data processing and analysis is done by EAD;
- Internal Anomaly Detection (IAD): this is a service running as part of the FINSEC platform. EAD communicates with IAD to query the input data and to report the analysis results. IAD stores the EAD results using Data Access API and exposes REST API to get access to these results from API Gateway.
Leveraging on an Apache Spark infrastructure for Big Data analytics, the solution is scalable adaptive and provides anomaly detection analytics as a cloud service. These are the main innovative features of the anomaly detection service.