Configure Authentication for Spark on YARN

As of Spark 1.5.2-1603 and Spark 1.6.1-1604, when the cluster is secure, authentication via a shared secret is enabled by default between all of the SparkMaster and SparkWorker nodes. When authentication is enabled, authentication keys are randomly generated for each job. When the cluster is not secure or for previous versions of Spark, complete the following step to manually configure authentication.

Configure the following property in the spark-defaults.conf file on each spark node:
spark.authenticate true

The spark-defaults.conf file is in the following location: /opt/mapr/spark/spark-<version>/conf/