How to Modernize Your Enterprise Data Architecture by Tearing Down Data Silos

Contributed by

9 min read

When big data became a phrase, a way to describe large volumes of data coupled with workloads that required more than one server, a promise was made: “no more data silos.” This promise was riding on the shoulders of Apache Hadoop. As the years went by, there were continued attempts to stack more on those shoulders, but failures continued. Apache Hadoop wasn’t up to the enterprise task of getting rid of all the silos. Instead, it created more silos, due to its lack of enterprise flexibility and agility.

As we fast forward, we can see that the concept of digital transformation has put a stranglehold on businesses. There is an urgent need to transform, adapt, and fend of competitors who are capable of quickly leveraging these new technologies.

Digital transformation is essentially a remaking and reforming of how an enterprise serves all its constituencies, including its customers, employees, and business partners. It further defines processes that support continuous operations improvement, fearlessly disrupting existing businesses and entire markets while inventing new business models and even new businesses along the way. And, as the name implies, digital transformation fully leverages digital technologies in a highly strategic, carefully planned way to effect these profound changes. Just consider the examples above as illustrations of the awesome power of digital transformation.

There are both consistent and compelling reasons why digital transformation is, in many cases, the hottest topic in boardrooms, conference rooms, and around corporate lunch tables. Business and IT leaders alike have come to the common conclusion that they must transform the way the enterprise does business or risk going out of business entirely. In the recent “Voice of the Enterprise: Storage” survey from 451 Research, of 500 senior IT decision-makers, two thirds of respondents said their businesses will require moderate to significant transformation in the next five years.

For example, venerable banks must transform to deal with the literally hundreds of well-funded ‘fin techs’–non-traditional, well-funded, low-cost, online-only banks–gaining great favor with millennials and other high-growth demographic groups. Another venerable industry, hospitality, is being rocked to its core by the likes of Airbnb, HomeAway, and others, with their all-digital approach to linking global travelers to rooms. Businesses stood in awe as Uber and Lyft utterly and irreversibly disrupted a century-old industry by leveraging their digital native expertise and the ubiquity of smartphones.

Business imperatives are driving digital transformation. We are now a part of the most significant overhaul and replatforming of the IT infrastructure in the last 30 years. The fast-emerging, newer technologies that will drive the applications and workloads in the transformed enterprise include the likes of big data analytics, containers, Internet of Things (IoT), and hyper-aggressive cloud adoption. The traditional or legacy systems initially put in service in the late 1980s simply cannot support these workloads in an efficient or effective way, if at all. This infrastructure overhaul will be the key on-ramp to digital transformation.

Next Gen Data Platform

1. Fully Leverage All of Your Data

If the business is all about data, then it should be prepared to use all the data. This digital transformation can only be accomplished by fully leveraging all of the data available. As an example, consider one major technology for which those legacy systems were built upon, namely data warehouses. Using archived data that was scrubbed, structured/prepared, and otherwise made ‘warehouse-ready,’ IT would apply various business intelligence reporting tools to view this historical data and mine insights to drive decision-making. This whole process took time–weeks in some cases. Latency is the enemy of agility. The way to achieve digital transformation is to remove as much latency as possible.

Today, businesses compete ferociously to improve time to market of mission-critical products and services. Doing so means working with ever-larger data sets from an expanding universe of data sources, including social media, third parties, government sources, private research sources, IoT sources, and, of course, a constant in-flow of operational data. Increasingly, this data is semi or completely unstructured, and legacy systems choke on both data types. All of this is incompatible with the desire of businesses to marry operational data in real time to the output of business analytics tools from data warehouses. It simply isn’t enough to use the data warehouse to analyze sales and other data from the previous quarter. Rather, businesses need to match those insights with real-time analysis of current operational and other data to gain entirely new insights and make far more timely business decisions.

Analytical and Operational Applications

2. Tear Down Your Data Silos

But this cannot be done when warehouse data resides in one cluster and operational data is in one or more others. Something new is needed: a converged data platform. Think of converged infrastructure as the planned, strategic combination as well as integration of all computing resources–hardware, networking, and software–effectively eliminating silos and allowing the seamless yet secure flow of data between servers, on-premises, in the cloud, both, or even in multiple clouds simultaneously. Enterprises are already shifting spending to infrastructure that will support digital transformation. Over the next four years, companies will experience flat IT spending. But underneath that will be a steady decrease in legacy spending accompanied by a corresponding increase in spending behind next-generation technologies. The key to reducing costs while driving innovation is the data. In fact, the forecast also shows that within four years, 90% of data and workloads will be handled on next-generation technology.

Innovating and Reducing Costs at the Same Time

The bottom line is that enterprises that experience the most success expanding market share will be those not with the most data but with the most data agility, which is the ability to generate the fastest and most appropriate response to changes in customer demand.

3. Shape Your Strategy Around the Cloud

The data on the global movement of workloads to various cloud environments is as unambiguous as it is compelling. According to a survey of more than 900 senior IT decision-makers by 451 Research, nearly 59% of enterprise workloads that were placed in non-cloud environments in 2016 will shrink dramatically to 40.5% by 2018, representing a stunning 31% drop. Over this time, workload deployments to SaaS environments will jump 63%; deployments to hosted private clouds will rise 41%; and deployments to IaaS environments will more than double at 112%.

Perhaps the biggest challenge facing IT with respect to cloud is the obvious realization that there is no one single cloud. Rather, there are many, and most enterprises will deploy applications to several of them simultaneously. The challenge is one of orchestration and integration of data across various clouds.

For example, consider the app/dev environment. Ideally, a lot of prototyping and testing would be done in the public cloud, with its scale-on-demand capabilities that make it easy for developers to get this vital work done without imposing on internal resources. Most organizations still choose to retain their most sensitive data on premises, however. Yet, to complete the app/dev process from development to test to production, data must flow securely and seamlessly among these different environments in what is known as the hybrid IT world (a combination of on-premises and off-premises, public and private cloud as well as use of traditional on-premises non-cloud clusters). What is sorely needed to make all this happen is a distributed processing model that scales easily across all locations and environments. In other words, a converged data platform. I simply cannot reinforce it enough: latency is the antithesis of agility. Moving data takes time; transforming it takes time; agility is the key to digital transformation; and tearing down the silos is the most efficient way to cut out latency, bar none!

This blog post was published February 01, 2018.

50,000+ of the smartest have already joined!

Stay ahead of the bleeding edge...get the best of Big Data in your inbox.

Get our latest posts in your inbox

Subscribe Now