RiskIQ Provides Faster and More Advanced Security Threat Detection Using the MapR Platform
July 26, 2016
MapR Technologies, Inc., provider of the industry’s only Converged Data Platform, today announced that RiskIQ, the leader in external threat management, is using the MapR Converged Data Platform to cost-effectively provide faster and more comprehensive threat detection on extremely large datasets.
RiskIQ’s external threat management platform scans, analyzes and stores entire websites, mobile app stores, and social media outlets across the entire internet. As the amount of data and sophistication of security threats grew, RiskIQ wanted to be able to provide their clients with faster and more comprehensive threat detection analysis in an economical way. RiskIQ selected the MapR Platform and has been using it as a distributed storage system for several years. More recently the company is using MapR with Apache Spark, Hive, Parquet, and Oozie and has successfully introduced new threat detections product offerings for their customers.
“The MapR file system architecture is something we appreciated from the start,” said Chris Kiernan, RiskIQ CTO and co-founder. “We knew we could leverage MapR for almost anything we wanted to do. With the node management and the way clustering is done, we always knew it was built in the right way if we needed to do analysis.”
RiskIQ’s 100 web crawlers collect about 10-20 TBs data each day from across the Internet. This number continues to grow as RiskIQ adds depth and new datasets to their crawl data. In order to reduce the size of the data, the company developed a technique to create Parquet files from raw crawl data that are 10x smaller and sent via NFS into their warehouse for analysis. “We can query an entire day’s worth of files in minutes instead of hours,” said Kiernan. “There are all kinds of things we can do, now that we have the data in a compact format. It’s a central part of our architecture.”
MapR architecture increases efficiency and reduces costs
RiskIQ was able to keep costs down by building the new data analysis use case on top of their existing MapR cluster. “We continue to use the cluster as a production file system while, at the same time, we’ve built an entire warehouse using the same infrastructure for a very small price point,” said Kiernan. “We have been able to cut Capex and Opex in half. We would have had to pay twice as much to build a vanilla Hadoop cluster. If we had built this in Cloudera, we would have needed separate clusters for production and analytics. It wouldn’t be a dual-purpose system.”
Comprehensive and reliable platform
“The fact that MapR makes sure that everything is compatible has worked really well. If we want to try a new technology, we can install it and it’s ready to use,” said Adam Hunt, RiskIQ’s chief data scientist. “In my previous job, I worked with vanilla Hadoop, and the cluster did go down. That’s just not acceptable. We’ve never had that issue with MapR. It’s rock solid. We don’t see performance degradation no matter what we do to the cluster and upgrades are seamless.”
New capabilities provide competitive advantage
With the MapR Platform, RiskIQ has developed new product offerings that has allowed them to push new types of data into their application that helps them understand things about websites they may not have understood before. “We have built all new parts of our products based on this new analysis, so it has been absolutely instrumental to our host reputation service. We can now answer all of the ad hoc questions we could never answer before to provide even more advanced detection for our clients. It’s improving the way we run the business,” said Kiernan.
“RiskIQ has cost-effectively added extremely large datasets, developed new products and powerful new analytics for threat detection capabilities utilizing their existing, reliable MapR cluster,” said Jack Norris, senior vice president, data and applications, MapR Technologies. “The MapR Platform has enabled customers like RiskIQ to achieve results never before attainable, proving our system is designed for mission-critical performance that can yield a measurable competitive advantage.”
RiskIQ is a cybersecurity company that helps organizations discover and protect their external facing known, unknown and 3rd party web, mobile and social digital assets. The company’s External Threat Management platform combines a worldwide proxy network with synthetic clients that emulate users to monitor, detect and take down malicious and copycat apps, drive by malware and malvertisements. RiskIQ is being used by leading financial institutions and other companies to protect their web assets and users from external security threats and fraud. We are headquartered in San Francisco, backed by growth equity firms Summit Partners and Battery Ventures.
About MapR Technologies
MapR Technologies is a visionary Silicon Valley software company and creator of the next-generation data platform for AI and analytics, with the scale and reliability required by enterprise-grade, mission-critical deployments. The MapR Data Platform delivers the power of dataware to accelerate data-driven innovation. Forward leaning companies such as Cisco, Philips, and Société Générale, are able to create new data-driven solutions to outperform the competition. Learn more: mapr.com.
MapR is a registered trademark of MapR Technologies, Inc. in the United States and other countries. Other names and brands may be the property of others.