Australian consumers are among the most technologically sophisticated in the world, using a wide range of applications and devices to shop whenever and wherever they choose. They demand to be served quickly, which creates opportunities for highly responsive companies. Above all, consumers value a personalized experience, one with messages, recommendations, and promotions tailored to each individual.
“With the Cisco-MapR platform, Quantium has positioned itself to stay well ahead of our competitors for the foreseeable future."
|-- Alex Shaw, Head of Technology Operations, Quantium|
MapR Distribution for Apache Hadoop and Cisco UCS cut query time by 92 percent, improve accuracy of results.
Such observations are music to the ears of Quantium, a data analytics firm located in Sydney. Quantium develops insights into consumer needs, behaviors, shopping habits, and media consumption by applying rigorous techniques of actuarial science to consumer transaction data. Focused on retail, financial services and media, the firm works with market-leading companies such as Woolworths, National Australia Bank, Westfield, Coca Cola Amatil and Qantas.
During its early years, Quantium performed analytical work using data provided by its clients. While that business model had been highly successful, the firm’s executives saw an opportunity to take a giant step forward by acquiring its own data assets. Doing so would enable a range of innovative, value-added services. For example, Quantium could correlate the client’s internal data with external information about shopping behaviour and deliver a much broader understanding of consumer needs and behaviours1. Armed with these powerful insights, the firm’s clients could offer highly personalized recommendations and promotions that would increase revenues, enhance customer retention, and create a competitive advantage.
However, moving from concept to deployment required overcoming one major obstacle: the legacy analytics platform. As the data analysis expanded in scope and complexity, the firm’s Microsoft SQL Server platform and aging server hardware could not handle the load. To take advantage of the opportunity, Quantium needed a new analytics platform with scalable, reliable, high-performance servers and an enterprise-grade software framework.
Hosted on Cisco infrastructure, MapR Distribution for Hadoop meets Quantium’s strict requirements
To meet its challenges, Quantium assembled a team of data scientists from across the business. The team created a set of requirements and evaluated the available software and hardware solutions on the market. “Decisions about the new platform would affect Quantium’s business for years to come, so we invested a significant amount of time and money in the selection process,” says Alex Shaw, Quantium’s Head of Technology Operations.
Quantium realized that a big data solution was needed, not only because of the data volume but also the heavy analytical requirements. One of the big 4 Australian banks, for example, has more than 2 million customers who generate 14 million transaction a week—more than 5 billion transactions each year. While the team chose Hadoop as the big data software solution, they still needed to choose the best distribution from among the top-tier Hadoop vendors (see figure 1). The first stage of the process, a thorough analysis of features and benefits, narrowed the field to MapR and one other competitor.
To make the final decision, Quantium conducted an extensive proof of concept (POC) over a three-month period. The evaluation team built out and tested key use cases with actual data and real-world queries. MapR, a Cisco Preferred Solution Partner, was the clear winner, according to Shaw. “The POC demonstrated that MapR performs better than the competition. The MapR file system gives us maximum control over how we store information within the data volumes and has good security features.” MapR also incorporates better disaster recovery and data replication features than competing distributions, features that will come into play in the near future.
With the software decision made, the team turned its attention to the hardware. Dell was a trusted vendor at Quantium, so it was automatically considered along with challengers Cisco and an OEM vendor of commodity x86 servers. Performance results were similar across the three vendors, but Cisco Unified Computing System™ (UCS) won based on its lower power consumption and superior centralised management capabilities.
Performance of new platform exceeds targets
During the requirements phase, Quantium had set a target of a ten-fold increase in performance. That goal has been exceeded: Before and after testing shows that the MapR-Cisco platform decreases query processing time by 92 percent, which represents a 12.5X increase in performance (see figure 2)2.
Unique business model outpaces competitors
Performance is vital for Quantium to deliver rapid results to customers such as Woolworths, a retail giant in Australia, who continually demand expanded feedback and analytics on consumer activity and behaviour. “Having access to external data sets to combine with our clients’ data distances us from everybody else in this space,” says Shaw. “For example, a retail chain may have a fairly good picture of customer behaviour in their own stores, but little knowledge about how often their customers shop with competitors. We can provide those kinds of insights because we have the ability to leverage the full value of our data assets by marrying data sets containing in-store and online purchasing information with media consumption behaviour to make advertising more effective.”
Greater innovation, shorter time to market
As the data sets grew over time, and analytical complexity increased, Quantium relied on complex and time consuming sampling methods. Designing and implementing sampling techniques takes time and specialised skills. Now, with the new Cisco-MapR solution, data scientists can design complex queries that run against multiterabyte data sets and get more accurate results in just minutes rather than hours or days. In addition, the more powerful platform drives innovation because scientists can test alternative scenarios quickly and accurately, shortening development timeand improving time to market. “We have a lot of smart people who have been hamstrung by technology and its ability to implement their ideas. Now they have improved ways of executing analytics which opens up the ability to create new and innovative solutions for our clients” says Shaw.
Scaling to accommodate business growth
Quantium’s client base is expanding rapidly, which requires the firm to increase its compute capacity accordingly. The new platform features a clustered architecture, which scales easily with high reliability and a lower total cost of ownership. The clustered approach allows Quantium to finetune performance in a cost-effective way, adding servers to the cluster as necessary to meet service-level agreements instead of replacing servers with more powerful models (see figure 3). Unlike the legacy system, all users benefit from the addition of new servers to the cluster. Cisco’s efficient, intuitive cluster management tools cut the time required to administer the multitenancy system by up to 30 percent compared to competitive offerings.
Multitenancy model safeguards client information
Quantium’s clients purchase subscriptions that allow them to access insights by logging directly into the firm’s products hosted in-house. Quantium supports this capability through a multitenancy model (the data from multiple customers resides on the same cluster). Multitenancy lowers costs and increases hardware utilization but introduces the risk of unauthorized access to information by other users on the cluster. Quantium ensures the safety of its clients’ data using the unique security features of the MapR Distribution for Hadoop. “MapR incorporates data partitioning via the Volumes feature, which allows us to logically segregate individual data sets while optimizing data storage for optimum performance,” says Shaw.
Going forward, Quantium will work to take full advantage of the efficiencies of the Cisco-MapR platform to maintain a strong competitive edge in the marketplace. Development cycles will continue to decrease, reducing costs and reducing the time by which new products are brought to market. Most importantly, Quantium is poised to enter new market segments where its expertise in complex analytical methods and big data analysis can provide tangible business value.
All of these plans and more are made possible by the performance, flexibility, and reliability of the new analytics platform. “We’ve expanded the range of problems that we can solve, enabling our clients to grow their business by interacting with each of their customers as individuals with specific wants and needs,” says Shaw. “With the Cisco-MapR platform, Quantium has positioned itself to stay well ahead of our competitors for the foreseeable future.”