Configure SSL Encryption for Spark on YARN
spark.ssl.keyStorePassword
,
spark.ssl.trustStorePassword
, and
spark.ssl.keyPassword
from the spark-defaults.conf
file for addittional security. These passwords are stored in the
/opt/mapr/conf/ssl-client.xml
file and Spark can access passwords
from this file itself. /opt/mapr/conf/ssl-client.xml
and
/opt/mapr/spark/spark-2.3.1/conf/spark-defaults.conf
files,
then the password from the spark-defaults.conf
file is used.
spark-defaults.conf
file on each spark node, configure
the following properties:
- For Spark 2.0.1 and
later:
spark.ssl.fs.enabled true spark.ssl.keyPassword <ssl-keystore-password> spark.ssl.keyStore /opt/mapr/conf/ssl_keystore spark.ssl.keyStorePassword <ssl-keystore-password> spark.ssl.trustStore /opt/mapr/conf/ssl_truststore spark.ssl.trustStorePassword <ssl-keystore-password> spark.ssl.protocol TLSv1.2 spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA
Note: Starting in MEP 4.0, for secure clusters, you can skip this step. For new installs done through the 6.0 MapR Installer, the installer enables this configuration. For manual installs and upgrades, running configure.sh -R, as the final step in the configuration process, enables these settings. - For Spark
1.6.1:
spark.ssl.akka.enabled true spark.ssl.fs.enabled true spark.ssl.keyPassword <ssl-keystore-password> spark.ssl.keyStore /opt/mapr/conf/ssl_keystore spark.ssl.keyStorePassword <ssl-keystore-password> spark.ssl.trustStore /opt/mapr/conf/ssl_truststore spark.ssl.trustStorePassword <ssl-keystore-password> spark.ssl.protocol TLSv1.2 spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA
Spark UI SSL is not needed when running Spark on YARN because encryption is
provided by the YARN protocol. For versions prior to MEP 4.1.0, to enable users logged in with a normal user
account (not mapr or root) to run spark jobs on the cluster, disable Spark
SSL for Spark-on-YARN jobs. To disable Spark SSL, add
spark.ssl.ui.enabled false
to the
spark-defaults.conf
file on each spark node. The
spark-defaults.conf
file is in the following location:
/opt/mapr/spark/spark-<version>/conf/
. Make sure SSL
is enabled for the Spark history server.