MapR 5.0 Documentation : Security for Spark on YARN

Depending on the version of Spark that you installed and your cluster's security configuration, certain Spark security features may be enabled or disabled by default.

Configure Authentication for Spark on YARN

As of Spark 1.5.2-1603 and Spark 1.6.1-1604, when the cluster is secure, authentication via a shared secret is enabled by default between all of the SparkMaster and SparkWorker nodes. When authentication is enabled, authentication keys are randomly generated for each job. 

When the cluster is not secure or for previous versions of Spark, complete the following step to manually enable authentication.

  • In the spark-defaults.conf on each spark node, configure the following property:

    spark.authenticate true
    The spark-defaults.conf file is in the following location: /opt/mapr/spark/spark-<version>/conf/

Configure SSL Encryption for Spark on YARN

As of Spark 1.5.2-1605 and Spark 1.6.1-1605, encryption between all the SparkMaster and SparkWorker nodes is disabled by default. In Spark 1.5.2-1603 and 1.6.1-1604, when the cluster is secure, encryption between all of the SparkMaster and SparkWorker nodes is enabled by default using the TLS v1.2 protocol. 

Complete the following step to manually configure encryption.

  • In the spark-defaults.conf on each spark node, configure the following properties:

    spark.ssl.akka.enabled true
    spark.ssl.fs.enabled true
    spark.ssl.keyPassword mapr123
    spark.ssl.keyStore /opt/mapr/conf/ssl_keystore
    spark.ssl.keyStorePassword mapr123
    spark.ssl.trustStore /opt/mapr/conf/ssl_truststore
    spark.ssl.trustStorePassword mapr123
    spark.ssl.protocol TLSv1.2
    spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA

The spark-defaults.conf file is in the following location: /opt/mapr/spark/spark-<version>/conf/

When you manually configure encryption between the SparkMaster and SparkWorker nodes, configure the same protocol and algorithms for each node. Otherwise, the connection between those components will fail.

Configure SASL Encryption for Spark on YARN

As of Spark 1.6.1-1605, when the cluster is secure, SASL encryption between all the SparkMaster and SparkWorker nodes is enabled by default.

When the cluster is not secure or for previous versions of Spark, complete the following steps to manually enable SASL encryption.

  1. Verify that authentication for Spark on YARN is enabled or configure authentication. SASL encryption uses the same authentication keys. 
  2. Configure the following property in the spark-defaults.conf file on each spark node.
    spark.authenticate.enableSaslEncryption true
    The spark-defaults.conf file is in the following location: /opt/mapr/spark/spark-<version>/conf/