MapR 5.0 Documentation : Storm on YARN

This procedure guides you through the process of installing and launching Storm on YARN.
  1. Download Storm from https://github.com/yahoo/storm-yarn/archive/master.zip with the following command:
    wget https://github.com/yahoo/storm-yarn/archive/master.zip

  2. Unzip the downloaded archive, then change to the storm-yarn-master directory:
    # unzip master.zip
    # cd storm-yarn-master

  3. Add the following section to the pom.xml file:
    <repository>
         <id>MAPR-REPO</id>
         <name>MapR Repository</name>
         <url>http://repository.mapr.com/maven/</url>
    </repository>

  4. Replace the <hadoop.version> property in the pom.xml file with the following line:
    <hadoop.version>2.3.0-mapr-4.0.0-fcs</hadoop.version>

  5. Create a work folder to hold working files for Storm. In this example, the work folder is named storm-work-folder:
    # mkdir storm-work-folder

  6. Copy the lib/storm.zip file to the work folder.
    # cp lib/storm.zip /storm-work-folder

  7. Add the storm-0.9.0-wip21 and storm-yarn-master directories to your PATH environment variable:
    # export PATH=$PATH:/storm-work-folder/storm-0.9.0wip21/bin:/storm-yarn-master/bin

  8. Copy the storm.zip file to the MapR-FS layer with a hadoop fs -put command or over an NFS mount point. The following example uses hadoop fs -put:
    # hadoop fs -put storm.zip /lib/storm/0.9.0-wip21

  9. From the storm-yarn-master folder, start Maven:
    # cd storm-yarn-master
    # mvn package -DskipTests

  10. Edit the storm-0.9.0-wip21/conf/storm.yaml file to include the ZooKeeper servers for your cluster. The file uses the following syntax:
    storm.zookeeper.servers:
    - "<ip address of ZooKeeper1>"
    - "<ip address of ZooKeeper2>"
    - "<ip address of ZooKeeper3>"
    storm.zookeeper.port: <port number>

  11. Launch Storm on Yarn with the following command:
    # storm-yarn launch <path to your storm.yaml file>

  12. Use the following command to display the YARN application ID for Storm:
    # yarn application -list

  13. Store the storm.yaml file in the .storm directory to enable Storm to find the configuration file when Storm submits jobs:
    # storm-yarn getStormConfig -appId <application ID> -output ~/.storm/storm.yaml

  14. Use the following command to display the Nimbus host:
    # cat ~/.storm/storm.yaml | grep nimbus.host

  15. Run two sample topologies to verify Storm is working correctly. To run the word count example, use the following command:
    storm jar lib/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.WordCountTopology WordCountTopology -c nimbus.host=<your nimbus host>

  16. To run the Exclamation example, use the following command:
    storm jar lib/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.ExclamationTopology ExclamationTopology -c nimbus.host=<your nimbus host>

  17. Monitor the results of the sample topologies by connecting to the Storm Web UI with a browser. The Storm Web UI is typically at the Nimbus host, on port 7070.

Storm UI

Attachments: