5 min read
Last year, about six months after shipping our first version of Hadoop 2.x with YARN, multiple customers asked us to consider working with Apache Mesos. Our early response was that of curiosity. Why are multiple customers asking us to work with Mesos when we just released YARN? YARN is a Hadoop cluster resource manager. We were also spending effort making YARN better—adding label-based scheduling, disk as a resource, and generally improving YARN high availability at that time.
Upon further inquiry, it became clear that the need was not necessarily around Hadoop, but beyond Hadoop. There is a need for a global resource manager that helps across entire data centers (or even beyond that).
This is when Santosh Marella (a MapR engineer) got involved. He started conversations with Adam Bordelon from Mesosphere, and hiked up to the Mesosphere San Francisco office for a hackathon. Within a week, they had a working prototype to make Mesos and YARN co-operate and integrate with each other. The idea was to use Mesos as a data center-wide resource manager and continue to use YARN for Hadoop resource management.
Independently, about a week earlier, Mohit Soni (an eBay engineer) had presented a prototype solution for solving the same problem at MesosCon in Chicago.
It was quickly apparent that we were all working on the same problem. Adam connected Santosh and Mohit together to explore the possibility of a collaboration. After a detailed technical pow-wow, they decided it was better to join together, combine ideas from both solutions, and unite the efforts.
Myriad was born out of this (with naming credits going to Mohit).
Benefits of Myriad
Once you allow Mesos and YARN to talk to each other, a new set of possibilities open up.
Let’s focus on the benefits to the Hadoop ecosystem first:
Beyond Hadoop, there are several benefits as well:
A diverse team of engineers has collectively decided to push this effort even further. We welcome any and all input, as well as participation. The early code base is available at github https://github.com/mesos/myriad but it is not production ready.
To Learn More
To learn more about YARN and MESOS:
1) Read/watch our YARN and Mesos Whiteboard Walkthrough
2) Attend one of the Strata presentations or visit our MapR booth:
3) Stop by the MapR Myriad Demo Pod at Strata to meet the technical brains behind the project and discuss your use cases.