4 Hot Business Intelligence Trends: Your Data Presents Endless Possibilities (Wrap-up)

Contributed by

6 min read

Editor's Note: Read Part One, Part Two, Part Three, and Part Four of this four-part series.

The numbers on data growth are mind-blowing.

International Data Corp. has estimated that the size of the digital universe is growing about 40 percent per year. IDC estimates that as of 2013, the world had collected 4.4 zettabytes of data. By 2020, that number is expected to grow tenfold to 44 zettabytes.

4 Hot Business Intelligence Trends: MapR Technologies

_[Click here to download the PDF]_

How big is that? Well, a zettabyte looks like this: 1,000,000,000,000,000,000,000. It’s one sextillion bytes, or one billion terabytes.

This might help you visualize a little easier. If you covered a football field with 32 gigabyte iPhones and kept stacking layers on top of each other, by the time you got to 44 zettabytes, the stack would reach 14.4 miles into the air. That’s 2 1/2 Mt. Everests stacked on top of each other and more than twice peak the altitude of commercial plane flights.

The Internet of Things will drive much of this growth. IDC estimates that by 2020, every human on the planet will generate 1.7 megabytes of new information every second. Machine-generated data will account for 40 percent of the digital universe in 2020, up from 11 percent in 2005.

What are we going to do with all this data? Well, if Moore’s law is any indication, we will only want more of it. But we won’t be able to manage it the way we do today. Data may be growing, but the number of IT staff available to deal with it isn’t.

The only solution is to hand over the management of data to the people who need it. That’s been done before. In the early decades of the 20th century, growth in telephone use presented major capacity problems to telecommunications carriers, since all calls were routed through switchboards staffed by humans. At one point in the 1940s, AT&T employed 350,000 switchboard operators.

People said the growth of telephone use would eventually require everyone in the country to be an operator. And that’s exactly what happened. Innovations in automated switching made self-service dialing possible. Today, there are fewer than 10,000 operators in the U.S. When was the last time you talked to one?

In the same way, PCs absorbed the growth in demand for business computing services and spreadsheets turned everyone into programmers. We manage growth by adapting our skill sets.

The innovations going on today in Hadoop, big data management, and business intelligence tools will make it possible for us to not only handle the growth in data, but actually make better sense of it than we do today. In fact, IDC predicts that the percentage of useful data will grow from 22 percent in 2013 to 37 percent in 2020, even as the volume swells by orders of magnitude. Among the factors driving this improvement are:

  • Low-cost, scale-out storage infrastructure is making it cheaper to store data than to throw it away.
  • Nearly 20 years of search engine development is paying off in tactics and technologies that make unstructured data accessible and useful. Machine learning, semantic analysis, and schemaless databases have unlocked value in what was once useless rivers of text and bits.
  • Machines are getting better at figuring out what’s important. Like a reverse search engine, they pour over data and suggest questions to ask, rather than leaving it to humans to guess.
  • Computer visualization tools make it possible to see patterns in data that are impossible to spot in rows and columns.

The upshot of this information explosion is that data is moving from being an operational to a strategic asset. Companies that innovate in their use of large quantities of data will win.

Consider one darling of the sharing economy: Airbnb. It’s the world’s largest hotelier, despite owning no properties and employing no clerks, housekeepers, bellmen, or desk clerks. Its assets are its database of properties and customers and the technology that connects them in an efficient and positive way.

Or consider Facebook. It has built in an $18 billion annual revenue stream through the masterful management of big data. The ability to serve a personalized experience with millisecond response times to hundreds of millions of users simultaneously is a nearly insurmountable competitive advantage.

You can profit from the data explosion by beginning the process today of turning business users into telephone operators. Build some experimental data lakes and test the tools available to extract knowledge from them. Many are open source or offer community versions at little or no cost. It’s not expensive to get your feet wet.

As users experiment, they’ll come up with their own ideas for what to do with data. Encourage them. Help them find new external sources. Support their use of the tools, but don’t do the work for them. The greatest IT potential of the organization will be unleashed when IT people aren’t the ones doing all the work.

Learn more by downloading the free book: The Definitive Guide to BI and Analytics on a Data Lake.

This blog post was published October 31, 2016.

50,000+ of the smartest have already joined!

Stay ahead of the bleeding edge...get the best of Big Data in your inbox.

Get our latest posts in your inbox

Subscribe Now