Configure Authentication for Spark on YARN

When authentication is enabled, authentication keys are randomly generated for each job.

Note: Starting in MEP 4.0, for secure clusters, you can skip this step. For new installs done through the 6.0 MapR Installer, the installer enables this configuration. For manual installs and upgrades, running configure.sh -R, as the final step in the configuration process, enables these settings.

Complete the following step to manually configure authentication on a non-secure cluster or in versions earlier than MEP 4.0:

  1. Configure the following property in the spark-defaults.conf file on each Spark node:
    spark.authenticate true

    The spark-defaults.conf file is in the following location: /opt/mapr/spark/spark-<version>/conf/

  2. If you have configured an external shuffle service, you must also add the following property to yarn-site.xml on each Spark node:
    <property>
          <name>spark.authenticate</name>
          <value>true</value>
    </property>