Using Kamanja for Application Monitoring

Data leakage is a major concern for enterprises.  White papers from Cisco and the SANS Institute suggest that insider security breaches may be an even bigger concern than external attacks.  This makes sense.  Those with the greatest access to data have the opportunity to do the greatest damage.  This may be through malice, through negligence, or even through an outsider exploiting stolen credentials.

One major US bank has decided to address this issue with Kamanja.  Previously, they had been operating with a batch-oriented legacy system to monitor application usage.  It took months of manual effort to create and implement new alerts.  They were crippled by the time it took to develop new source system extracts and to assimilate new data.  This reliance on old technology put both PII and NPI at risk, with implications for compliance as well as an obvious financial downside.

Their solution centered on an open-source big data stack, with a modified Lambda architecture that put Kamanja at the heart.  This approach allowed them to migrate to real time data streaming, to deploy new models rapidly, and to generate alerts without manual intervention.

The initial data modelling approach was relatively straightforward.  They would calculate the number of times a given unit of PII and/or NPI was accessed over a specified time period, index that against a normative value, and calculate the risk to generate an alert.

This evolved into machine learning to identify the normal patterns of behaviour, with the first use case focusing on out-of-office hours access.

Kamanja would then trigger automatic alerts when anomalies occurred.  It also allowed for rapid implementation of new models, so they could improve detection as the threats became more sophisticated, without having to rewire the whole system.