Configure SSL Encryption for Spark on YARN

Starting in MEP 6.0.0, you can remove spark.ssl.keyStorePassword, spark.ssl.trustStorePassword, and spark.ssl.keyPassword from the spark-defaults.conf file for addittional security. These passwords are stored in the /opt/mapr/conf/ssl-client.xml file and Spark can access passwords from this file itself.

Note: If passwords are present in both /opt/mapr/conf/ssl-client.xml and /opt/mapr/spark/spark-2.3.1/conf/spark-defaults.conf files, then the password from the spark-defaults.conf file is used.

Complete the following step to manually configure encryption for the Spark HTTP file and broadcast servers:

In the spark-defaults.conf file on each spark node, configure the following properties:

For Spark 2.0.1 and later:
```
spark.ssl.fs.enabled true
spark.ssl.keyPassword <ssl-keystore-password>
spark.ssl.keyStore /opt/mapr/conf/ssl_keystore
spark.ssl.keyStorePassword <ssl-keystore-password>
spark.ssl.trustStore /opt/mapr/conf/ssl_truststore
spark.ssl.trustStorePassword <ssl-keystore-password>
spark.ssl.protocol TLSv1.2
spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA
```
Note: Starting in MEP 4.0, for secure clusters, you can skip this step. For new installs done through the 6.0 MapR Installer, the installer enables this configuration. For manual installs and upgrades, running configure.sh -R, as the final step in the configuration process, enables these settings.

For Spark 1.6.1:

spark.ssl.akka.enabled true
spark.ssl.fs.enabled true
spark.ssl.keyPassword <ssl-keystore-password>
spark.ssl.keyStore /opt/mapr/conf/ssl_keystore
spark.ssl.keyStorePassword <ssl-keystore-password>
spark.ssl.trustStore /opt/mapr/conf/ssl_truststore
spark.ssl.trustStorePassword <ssl-keystore-password>
spark.ssl.protocol TLSv1.2
spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA

Spark UI SSL is not needed when running Spark on YARN because encryption is provided by the YARN protocol. For versions prior to MEP 4.1.0, to enable users logged in with a normal user account (not mapr or root) to run spark jobs on the cluster, disable Spark SSL for Spark-on-YARN jobs. To disable Spark SSL, add spark.ssl.ui.enabled false to the spark-defaults.conf file on each spark node. The spark-defaults.conf file is in the following location: /opt/mapr/spark/spark-<version>/conf/. Make sure SSL is enabled for the Spark history server.

Note: When you manually configure encryption for Spark, set the same protocol and algorithms for each node. Otherwise, the connection between those components might fail.