A Practical Guide to Microservices and Containers

by James A. Scott

Typical Problems Facing Enterprise IT on the Road to Digital Transformation

This chapter provides an overview of the many challenges facing organizations today as they initiate or continue their journey to digital transformation, as well as the vital re-platforming of the IT infrastructure. While the focus here is on problems and challenges, specific solutions to these problems will be the chief focus of most of the rest of this book. Whatever specific problems you and your organization may be facing, rest assured you are not alone. These once-in-a-generation infrastructure overhauls are never easy. Not only does IT management have to deal with traditional and legacy infrastructure that is inadequate for the very high scale and low latency requirements of emerging technologies, but it must also come to terms with legacy ‘thinking’ as well as legacy skillsets. It is a multi-faceted challenge, but one that must be overcome for the sake of the long-term viability of the enterprise.

Three emerging technologies merit particular attention here as current challenges, the resolution of which shall be addressed later on. They are:

  • Cloud computing
  • Big data and big data analytics
  • Containers

Cloudy cloud

The data on the global movement of workloads to various cloud environments is as unambiguous as it is compelling. According to a survey of more than 900 senior IT decision makers by 451 Research1, nearly 59% of enterprise workloads that were placed in non-cloud environments in 2016 will shrink dramatically to 40.5% by 2018, representing a stunning 31% drop. Over this time workload deployments to SaaS environments will jump 63%; deployments to hosted private clouds will rise 41%; and deployments to IaaS environments will more than double at 112%.

Perhaps the biggest challenge facing IT with respect to cloud is the obvious realization that there is no one single cloud. Rather there are many, and most enterprises will deploy applications to several of them simultaneously. The challenge is one of orchestration and integration of data across various clouds.

For example, consider the app/dev environment. Ideally a lot of prototyping and testing would be done in the public cloud, with its scale-on-demand capabilities that make it easy for developers to get this vital work done without imposing on internal resources. Most organizations still choose to retain their most sensitive data on premises, however. Yet to complete the app/dev process from development to test to production, data must flow securely and seamlessly among these different environments in what is known as the hybrid IT world (a combination of on-premises and off-premises public and private cloud as well as use of traditional on-premises non-cloud clusters).

What is sorely needed to make all this happen is a distributed processing model that scales easily across all locations and environments. In other words, a converged data platform.

Big Data and Big Data Analytics

One of the biggest challenges organizations have today with big data, which includes IoT data, is that it is increasingly of the semi and unstructured variety. Legacy RDBMS simply cannot aggregate, store and process this data efficiently or effectively, certainly not in high volumes. The data volumes are unprecedented and growing at warp speed, in some cases doubling annually by some estimates. IT is under tremendous pressure to deal with the volume, variety, and velocity of new data, while at the same time pressured to deliver more personalization and better service to the customer base. This complexity of big data environments is illustrated well in Fig. 2-1 below.

The Growing Complexity of Big Data Environments
Figure 2-1.

This environment itself is highly dynamic, with frequent new releases of the many different components that make up the big data world. Other challenges abound:

  • Storing all the data is difficult, with many organizations looking to data lakes to collect and store huge volumes of data in native formats. But constructing them properly can be tricky, at times resulting in repositories where data is collected and seldom retrieved.
  • Big data analytics skills do not grow on trees! Shortages of so-called data scientists have been well documented, and shortages of people with critical skills translates into very high salaries to acquire them.
  • According to a recent major global security report, security worries easily top the list of barriers to more aggressive deployments of big data platforms.
  • Apart from the storage issue, there is the big data conundrum of having systems capable of managing these mega-data streams from so many disparate sources. From within the enterprise the sources frequently cut across multiple departments. And the number of potential outside sources is growing rapidly, including all of social media, third party data, government data, reports and research from the scientific and academic communities, industry data, and on and on.
  • Getting effective ROI from big data efforts are closely related to how quickly you get the results from the big data analytics tools. This process needs to happen at the speed of business, but in many organizations it does not, owing to all the complexities of the big data environment outlined here.


As will be addressed more fully in Chapter 4, containers are one of the fastest growing new technologies, owing to their capability to usher in a new and vastly improved application development environment. You can think of containers as operating system virtualization in which workloads share operating systems (OS) resources. Though they have been around for just a few years, their adoption rate and acceptance is impressive to say the least. In one major global study2 of 1100 senior IT and business executives, 40% of respondents said they are already using containers in production, half of those with mission-critical workloads. Only 13% say they have no current plans to use containers this year, with the remainder all making container usage plans.

Containers can do a lot of what virtual machines (VMs) cannot do in a development environment. They can be launched or abandoned in real time instantly, which VMs cannot. Unlike with VMs, there is no requirement for OS overhead in the container environment. And containers are destined to play a major role in facilitating the seamless transfer of development from one environment or platform to another.

However, there are issues and challenges with containers as well. For one, most containerized applications today are stateless. This could be an issue for stateful applications, but as will be shown, there are workarounds. These workarounds include solutions that provide the reliable storage needed to support stateful applications.

As with big data platforms, containers do provoke data security concerns, but this is not unusual with relatively new technologies deployed in critical areas such as application development.

1451 Research Voice of the Enterprise Cloud survey, Q2 2016, https://451research.com/dashboard/customer-insight/voice-of-the-enterprise/voice-of-the-enterprise_cloud

2Thales 2017 Global Data Threat Report of 1,500 organizations in seven nations. https://www.thalesgroup.com/sites/default/files/asset/document/thales_2017_data_threat_report-global_edition.pdf