July 23, 2015 | BY Dr. Kirk Borne
Big Data is a big deal in all enterprises everywhere. It is also big concept. In other words, big data is not only about rapidly increasing data volumes. There is also a rapid increase in data-driven business culture, applications, and use cases. As these continue to spiral upward, there is an accompanying increased demand for easy-to-use analytics tools.
Many other factors are also affecting data-driven organizations: the insufficient number of skilled data scientists in the workforce; the huge demand for data analytics solutions from business leaders; the need for traditional business analysts (mostly non-scientists) to apply analytics to their datasets without requiring analysts to have a degree in data analytics; the transformation from a management culture that relies on instinct-based decisions to one that places high value on data-based decisions; and peer pressure (i.e., “everyone else is doing it”).
Help is emerging for the data-drenched organization: Enterprise Analytics, the new EA (symbiotically tied to traditional EA = Enterprise Architecture). This new EA democratizes the data-driven activities across an organization. EA also relieves the IT department from the burden of adding data analytics to their growing list of mission-critical responsibilities (which detracts from their primary IT activities, primarily infosec and cybersecurity).
But, is it sensible to say that Enterprise Analytics will truly democratize analytics functions, such that any person (without math, or statistics, or programming, or scientific training) can truly perform the functions of a data scientist? I think that is the wrong question, for at least two reasons.
First, if an organization is truly experimenting with their data to make significant discoveries, to validate the results, to refine and improve the analytics models, to extract meaningful insights from data, and to implement innovative data-driven processes, then a data scientist is usually required. More specifically, a data science team is required, since no single person can have all of the required skills for successful enterprise-wide data science.
Second, there are some analytics functions that are scientifically or mathematically less intensive that can be carried out by analysts throughout an organization. Some of these functions include data exploration, data quality verification, data transformations, correlation analyses, model-testing, data visualization, reporting, and so on.
All of these data analytics functions justify the emergence of Enterprise Analytics as the right tool for the new data-driven organization. However, the right tool in the wrong hands can lead to problems. At best, it may lead to results that are not statistically significant or interpretable. At worst, it may lead to totally incorrect results: for example, overfitting a model can give the false impression of high accuracy when in fact it is precisely the opposite; or the misapplication of analytic methods (e.g., by overlooking requirements for particular data types or data transformations) can lead to useless results and wasted effort. Fortunately, an analyst without a formal analytics education but with good data literacy, numeracy, curiosity, and problem-solving skills will grow into it, after some tutorials and training.
The trend in Enterprise Analytics toward more "as-a-service" and API offerings will ease the transition to democratization of data within organizations. This service orientation trend is not new to business, but it is new to the advanced data analytics functions of the business (which were usually reserved for the “hero data analysts” of the organization). This trend will only get stronger in the coming years, especially with the advent of the Internet of Things (IOT).
Considering how ubiquitous sensors in the IOT will be emitting streams of data everywhere, and combining that with the huge market value on IOT applications, we can expect that all enterprises will want to get into that action. Consequently, data analytics APIs and Enterprise Analytics tools will become a necessity for IOT entrepreneurs, startup companies, innovators, and (of course) the big incumbents who see the value in enterprise solutions when compared against expensive in-house custom-built investment-intensive solutions. We will eventually see Enterprise Analytics capabilities spread across all enterprises, enabling the brave new world of Analytics of Things.
In order to manage and conquer these enormous Enterprise Analytics challenges, the other EA (Enterprise Architecture) matters! That is where the comprehensive EA solutions from MapR can truly benefit any organization. Whether it is with powerful batch processing capabilities on Hadoop clusters, or in streaming analytics applications using Spark, or querying multiple diverse databases for knowledge discovery using Drill, or addressing large-scale NoSQL data management requirements for Hadoop processing using MapR-DB, the MapR enterprise architecture solution suite delivers a variety of products and services for IOT and big data analytics applications in any enterprise.