How the Growth in Microservices Fuels the Need for Dataware

Contributed by

6 min read

Editor's Note: This is the 3rd blog post in a 5-part series that describes how modern enterprises are struggling with the handling of data, making it available to applications without creating new silos, and how MapR solves these challenges by introducing a new layer of abstraction called dataware. The previous 2 blog posts can be found here and here.

The growth of microservices has been staggering in recent years. Modern applications are now more dependent on microservices than ever before. To function properly, applications based on microservices must easily share and integrate data across a variety of sources. Because microservice instances grow and contract rapidly and frequently cannot tolerate latency, traditional databases cannot be directly involved. Microservices need to be able to access data faster and in real time in a way that is impeded by going through a database. Yet the data needed by microservices comes from traditional databases and many other sources, so companies are faced with a challenge of how to provide microservices the needed data at a speed fit for the demands of today’s applications.

Microservices are hugely powerful when it comes to making applications responsive and allowing multiple teams to work together to create and evolve complex applications. But the biggest challenge is how companies handle the persistence layer – namely, the data. Companies need to be able to get to and write data quickly and then move on so that the application isn’t delayed.

Developers need a way to assemble all the data, deliver it to the microservices, and then harvest what the microservices have done to send it back to the systems of record. Many developers and organizations are choosing dataware. Dataware is an abstraction layer that allows data to be managed as a first-class enterprise resource decoupled from any other dependencies. Dataware effectively handles the diversity of data types, data access, and ecosystem tools needed to manage data as an enterprise resource, regardless of the underlying infrastructure and location.

In the past, developers have had to put the persistence layer and the management code inside each microservice so the microservice can access a database or other store directly. This has led to massive bottlenecks and complexity, not to mention that doing so renders the microservice heavy thereby requiring more resources than normal. What has become much more popular is using streams as the persistence layer for microservices.

Streams allow you to have a stream of data you can write to and read from, which is just like a database but operationally is more friendly to microservices. The challenge is how you get the data in and out of the stream and then back into persistent repositories underneath the microservices. Putting this plumbing in each microservice is a recipe for disaster. Instead, developers need smart streams, so when inbound data arrives in real time or outbound data needs to be stored in multiple repositories, the intelligence and plumbing to do this is hidden from each microservice. That is exactly what dataware does.

Dataware greatly simplifies the development process. There is no need for a developer to understand where each data source is coming from or for a specialized data stack to support the processing. There is also no need to build restart logic and confirm that the application is consistent across the various processing sources and data. These complexities are eliminated through dataware because the data is available and consistent at all times.

Dataware thus greatly reduces the demands on developers. When working with data for microservices, a developer may want to combine real-time data with historical data. Without dataware, a developer has to keep track of all these sources and access points on her own.

But with dataware, a global namespace is created that allows data to be called wherever it resides. This is hugely liberating for developers.

Dataware allows the presentation of data as a stream but combines access to data at rest, so developers can combine deep learning from historical data with the latest arriving data. Dataware also adds intelligence about how you manage the data flows. This structure can handle the complexity of real-world operations, which makes dataware powerful for microservices. Assuming raw streams can do this on their own is naive. Dataware allows creation of a whole solution to microservice persistence.

This is vital for applications because they are used in so many places. With dataware, replication of data is handled with the global namespace, allowing easier scalability. A user could be using an app to add a forgotten tip for a car ride in London, even while hailing a new car in Moscow. The entire process of using and developing applications with microservices becomes simpler with dataware.

This blog post was published April 18, 2019.

50,000+ of the smartest have already joined!

Stay ahead of the bleeding edge...get the best of Big Data in your inbox.

Get our latest posts in your inbox

Subscribe Now