10 min read
Editor's Note: This is an excerpt from the book, "A Practical Guide to Microservices and Containers: Mastering the Cloud, Data, and Digital Transformation" – you can download the ebook here.
Containers offer a front-row seat to application development agility, which also explains the very rapid adoption of and enthusiasm for containers. To more fully appreciate the value of application agility, it is important to understand just how completely the application development environment is changing.
This environment has become far more complex and multifaceted in the last decade. The complexity has arisen chiefly due to the great difficulties in dealing with separate clusters or silos of data. This reality, in some organizations, has made application agility almost a misnomer. Clearly, a way is needed to seamlessly move data from one environment to another without having to customize the data for each environment. In fact, it is widely believed that, within a few years, developers leveraging containers will be able to do exactly that–move their test/dev projects into production without needing any major code rewrites to accommodate new data locations, such as on-premises or onto a cloud.
Enterprise applications have traditionally been built with a monolithic, vertically integrated structure. All application logic was contained within a single program, which often took months or even years to develop. Detailed functional specs were required, and extensive testing and code reviews were part of the process. The resulting application may have fit the stated requirements, but there was little latitude for embellishment. Enhancing an application required the preparation of new functional specifications, extensive testing, and more code reviews. Even modest enhancement requests could consume months. The legacy applications used by airlines, banks, and credit card processors are typical of these very large, robust, but inflexible programs.
The rapidly changing business and technology landscape of today no longer tolerates this approach. Year-long development schedules create applications that are often irrelevant by the time they are complete. New technologies like bots, automated assistants, and mobile devices must be accommodated quickly. New payment mechanisms, such as Bitcoin and Ethereum, come seemingly out of nowhere to change the way customers conduct transactions. Organizations that want to expose data or services to business partners have no way to do so because those needs weren't anticipated when the application was built.
Application agility reimagines development and deployment around a modular, loosely coupled, and dynamic architecture. Like Lego bricks, the components of the application can be moved and reassembled as needed. Communication between components is provided by a messaging plane. New or enhanced functionality is provided by revising individual components and slipstreaming them back into the network without disrupting the application as a whole. These components, or services, can also be stored in a library and shared by multiple applications. Services are called only when needed, which reduces program size and complexity while improving performance.
Applications which have been containerized are prime candidates for leveraging orchestration tools like Kubernetes (k8s). Kubernetes enables dynamic allocation of resources allowing quick ramp up or ramp down of those resources to best utilize the needs of the business in that moment. When building applications which are decoupled via an event stream for communication each service talking to an event stream can be easily scaled up or down to meet the business needs on-demand. In addition to orchestrating the containers k8s enables the ability to dynamically plug into your underlying converged data platform, thus enabling full access to persistence across your entire data center for all containerized applications. This has the added benefit of enabling analytics at scale on that data in the same location without having to move it a second time. Until recently, stateless applications were about the only true use case for containers, now any application requiring persistence can be successfully deployed in a container and orchestrated with Kubernetes.
Application agility also changes the nature of the data that applications process. Less and less, it is data warehouse or data mart data finding its way into leading edge applications. The growth area is event-based data. Whether it's collecting machine sensors to predict and prevent failures, or providing key offers to customers, or identifying and preventing fraud before it happens–all such use cases are enabled by event-based data flows and a converged platform.
Consider the example of legacy banking applications. Constructed in the age of mainframes for use with green-screen terminals by data processing professionals, these applications must now be accessible to any customer from any device. What's more, they must be intuitive to use. It's no surprise that many banks have spent years overhauling their application portfolios for the age of self-service banking.
Now consider one of the new breed of mobile payment apps, like Square. Its developers must cope with a constant stream of new devices, payment methods, and customer requests. The company closely monitors social media activity to measure customer satisfaction and identify bugs, which it has a few days to fix, at best. The most popular mobile apps are updated weekly in order to maintain feature parity with their competitors. In many cases, those apps must be synchronized with desktop versions that work on any platform and in any browser.
The frenetic pace of business in the age of digital transformation demands maximum application agility. Thanks to cloud computing, the barriers to entry have fallen, and the only way market leaders maintain their positions is by innovating faster than everyone else. Switching costs are low, and customers have seemingly endless choices.
Application agility enables high-scale, fast, mission-critical applications that can integrate analytics and deep learning to directly impact business. It supports microservices-based architectures for fast development and efficient deployment and processing.
Microservices architectures are increasingly used today as a more agile methodology for building and deploying large-scale applications. In contrast to monolithic applications, microservices consist of a series of individual modular applications—each of which focus on one specific task so that minimized scope leads to advantages around maintainability, testability, deployability, and scalability. They often use a publish-subscribe framework as the lightweight communications layer between them.
When adding new applications or analytical workloads to your infrastructure, you shouldn’t have to increase complexity by adding new data silos. Having one platform based on a single codebase for all of your data—whether in files, database records, or streams, along with your analytics capabilities—means you can gain faster insights while also minimizing the overhead of copying data. You also avoid dealing with distinct administrative and security frameworks across the different silos.
Today’s modern data architectures have a wide array of applications to address business problems. And since not all applications use the same types of data, you need a platform that can support the different types of compute engines that best manage your data. The combination of a distributed file system, an integrated NoSQL JSON document database, and an integrated event streaming engine lets you write almost any application you need.
The open source community is driving many great innovations that organizations should use today. Any platform used for today’s big data deployments needs to incorporate open source tools to round out the platform capabilities. Some of the key technologies from the ecosystem entail SQL querying, advanced analytics, and large-scale processing. Apache Drill, Apache Hadoop, and Apache Spark are leading the way to help organizations derive more value from their data.
SQL continues to be a leading business technology for gaining insights from enterprise data. Support for ANSI SQL in a big data platform is critical for letting organizations use their familiar desktop business intelligence (BI) tools to analyze data. Big data requirements also dictate the ability to query data as soon as possible in a self-service model. By reducing the upfront dependence on IT for data modeling, business analysts can get immediate insights on data as soon as it lands in the platform, while using the tools with which they are most productive. Learn more about Apache Drill.
Protecting data residing in disparate data silos is difficult. You need to make data security easier by minimizing the number of silos in order to remove the redundant administrative effort. Having a consistent, unified security model across different compute engines reduces the chance for error, and helps to ensure that sensitive data is not exposed to unauthorized users.
As always, if you have any questions you can post it in the comments section below or you can reach me on Twitter @kingmesal.
To learn more about the MapR Data Fabric for Kubernetes, go here.
Stay ahead of the bleeding edge...get the best of Big Data in your inbox.