Configure SASL Encryption for Spark on YARN

As of Spark 1.6.1-1605, when the cluster is secure, SASL encryption between all the SparkMaster and SparkWorker nodes is enabled by default. When the cluster is not secure or for previous versions of Spark, complete the following steps to manually enable SASL encryption.

  1. Verify that authentication is enabled or configure authentication. SASL encryption uses the same authentication keys.
  2. Configure the following property in the spark-defaults.conf file on each spark node.
    spark.authenticate.enableSaslEncryption true

    The spark-defaults.conf file is in the following location: /opt/mapr/spark/spark-<version>/conf/