5 min read
In this week’s Whiteboard Walkthrough, Prashant Rathi, Senior Product Manager at MapR, describes the architecture for fine-grained monitoring in the MapR Data Platform from collection to storage and visualization from a variety of data sources as part of the Spyglass Initiative.
Here's the undedited transcription:
Hi, and welcome to my Whiteboard Walkthrough. I'm Prashant Rathi, Senior Product Manager at MapR, and I'm going to talk about monitoring in the MapR Data Platform. As we all know, managing Hadoop clusters can be very complex, and as you add more data, more use cases, and even more users to your clusters, you always want to rely on a robust monitoring environment.
We at MapR believe in empowering administrators with a robust data collection and storage infrastructure, which allows them to monitor the cluster as they want it, how they want it, and in their own environment.
Let me cover some of the areas that we'll be talking about in our monitoring environment here. With node infrastructure monitoring, you can look at all the memory, CPU, I/O, and all the information disk utilization at each node. You can also look at all the applications running on a per-node basis. In addition, you can look at cluster space utilization where you can monitor volume trends, how data is getting collected in your volumes, and how close you're getting to your cluster space utilization.
For YARN and application monitoring, you can look at the YARN resource manager, node manager metrics, as well as look at queue and application metrics in YARN. For service daemon monitoring, you can look at how you can monitor Drill, Spark, and any other service which runs through the breadth of your Hadoop infrastructure.
Let's go into the details of the architecture of MapR Monitoring. At the very base, we have all the data sources, so as I said, you can monitor all your hosts and all of your MapR-specific core components, like MapR XD and MapR Database, as well as ecosystem components like YARN, Drill, Spark, and so on. At the very heart of the monitoring layer is the collection layer, where we collect all the metrics and all the logs from each of your nodes, where regions run on the nodes, and ship these metrics and logs at a per second, or at a configurable interval to the data storage layer. Collecting a new metric or monitoring a new service is as simple as adding a new plug-in into your metrics collector or log shipper.
At the same time, on the storage layer, we are relying on OpenTSDB and elastic storage for monitoring metrics and logs. OpenTSDB, with its time series database, allows us per-second monitoring, as well as allows us to look at the trends in the past seven days and past 24 hours, and then you can relate that data with the logs from Elasticsearch to see what exactly happened within your cluster at a particular time.
The power of the interface is not just in storage, but also in individualization, where you can now visualize with our chosen components like Grafana and Kibana, and build your own dashboards.
Let me give you an example of the type of dashboard you can build. Say you want to monitor your cluster and you’re only interested in looking at CPU and memory across the cluster. You can add that to a dashboard, and that should be it. But if you want to look at all of the information on a particular node, you can tag that particular node into your system and look at memory, disk, and CPU next to each other for that particular node.
Not only can you build customizable dashboards, but if you have your own environment for monitoring other components of your infrastructure, you can easily integrate it using the APIs that OpenTSDB and Elasticsearch allow you to do. That way, you have single pane of glass to look at everything within your monitoring environment.
I hope you liked this video, and if you have any comments or suggestions, please add them in the comments section.
Stay ahead of the bleeding edge...get the best of Big Data in your inbox.