Big Data for Security Analytics: Seeking Safe Haven in a Hostile World

Contributed by

5 min read

Is there a case to be made for big data for security analytics? The answer is an unqualified “yes.” In fact CSO Magazine called cyber security “the killer app” for big data analytics. Meanwhile a recent study pegged the compound annual growth rate of security analytics at a robust 27.6%, topping $7 billion four years from now.

The reason is straightforward. In today’s ultra-sophisticated and well funded attack environment, some of the most insidious and potentially devastating threats cannot be detected without very deep insight into networks, data, and usage. Most of the traditional approaches to data security are adept at combating known threats. But traditional approaches can do little if anything for organizations seeking to beef up and grow their security strategies to detect and guard against the more complex evolving threats that are omnipresent.

Traditional solutions are good at creating alerts to changes in documented, “business as usual” patterns. For example, they can send an alert of suspicious activity when they see multiple failed attempts to login to the same account from the same IP address. Or they will generate an alert when it finds excessive data access after hours from a given account.

But guess what? The hackers know these patterns are tracked. So they just alter their ways to avoid the usual patterns. In addition - and this is key - many traditional security solutions simply cannot handle the sheer volumes of data created by today’s networks. They also choke on unstructured data, which far and away is the fastest growing kind of data being accessed and used.

Enter big data for security analytics. These are solutions capable of capturing, filtering and analyzing millions of discrete network events per second. Moreover, these solutions can work with data from an ever-widening variety of data sources, including log and audit files. These sources also stretch into that murky realm of unstructured data, which includes images, email, video, social media, news feeds, and many others. Small wonder the market for big data and analytics topped $125 billion last year. CSO believes that big data for security analytics will be the “first line of defense” when it comes to an integrated approach to sniffing out threats before they strike as well as threat detection, deterrence, and prevention.

Big data for security analytics meets the new requirements today for an efficient means of retaining massive amounts of data for large-scale analytics. This is a key ingredient of a solution that yields optimal visibility into all activities on the network and infrastructure. Organizations get that valued “knowledge of the unknown” with automated, actionable intelligence that sniffs out behavioral anomalies.

Getting started with big data for security analytics can seem complicated. Here are a few first steps I recommend:

  1. Be sure your data platform has a strong focus on authentication, authorization, auditing, and data protection capabilities. Auditing is critical for tracking data accesses and tracing data back to any unforeseen breaches.
  2. Traditional data platforms simply don’t handle unstructured data cleanly and efficiently, if at all. So construct yours using a technology built for this use, namely Apache Hadoop. This will give you a platform for aggregating data from the multiple sources, which is what you need to get the clearest picture of potential threats to your environment.
  3. Organize your internal resources. Be sure that your company’s data scientists and IT security analysts are talking and meeting regularly to discuss the evolving security platform

Remember that the goal in today’s threat environment is predicting an attack before it hits, and protecting all data from hackers seeking any weaknesses in your systems. It is naïve and potentially disastrous to presume off-the-shelf solutions will give you all the protection you need. Ultimately deploying advanced, large-scale analytics using machine learning and anomaly detection will become your most valued tools in the fight against cyber crime.

This blog post was published June 20, 2016.

50,000+ of the smartest have already joined!

Stay ahead of the bleeding edge...get the best of Big Data in your inbox.

Get our latest posts in your inbox

Subscribe Now