Configure Spark JAR Location (Spark 1.6.1)

By default, Spark on YARN uses a Spark JAR file that is installed locally. The Spark JAR file can also be added to a world-readable location on MapR file system. When you add the JAR file to a world-readable location, YARN can cache it on nodes to avoid distributing it each time an application runs. Complete the following steps to add the Spark JAR file to a world-readable location on MapR file system:
  1. Copy the following JAR file from the local file system to a world-readable location on MapR file system: Substitute your Spark version and specific JAR file name in the command.
    /opt/mapr/spark/spark-<version>/lib/spark-assembly-<version>-hadoop<version>-mapr-<version>.jar
  2. Set the spark.yarn.jar property in the spark-defaults.conf file to point to the world-readable location where you added the Spark JAR file. For example:
    spark.yarn.jar      maprfs:///<path to jar>