Post-Upgrade Steps for Spark

Complete the following steps after you upgrade Spark with or without the MapR Installer.

Post-Upgrade Steps for Spark Standalone Mode

  1. (Optional) Migrate Custom Configurations.
    Migrate any custom configuration settings into the new default files in the conf directory (/opt/mapr/spark/spark-<version>/conf).
  2. If Spark SQL is configured to work with Hive, copy hive-site.xml file into the conf directory (/opt/mapr/spark/spark-<version>/conf).
  3. Run the following commands to configure the slaves:
    1. Copy the /opt/mapr/spark/spark-<version>/conf/slaves.template into /opt/mapr/spark/spark-<version>/conf/slaves.
    2. Add the hostnames of the Spark worker nodes. Put one worker node hostname on each line.
      For example:
      localhost
      worker-node-1
      worker-node-2
  4. Run configure.sh -R.
  5. Restart all the spark slaves as the mapr user:
    /opt/mapr/spark/spark-<version>/sbin/start-slaves.sh spark://<comma-separated list of spark master hostname: port>
  6. Delete the old Spark directory from /opt/mapr/spark. For example, if you upgraded from Spark 2.1.0 to 2.3.1, you need to delete /opt/mapr/spark/spark-2.1.0.
    Starting with the MEP 6.1.0 release, for Spark 2.2.1 and later versions, after an upgrade the old directory is automatically removed. Only the new directory and the directory with the timestamp is present.

Post-Upgrade Steps for Spark on YARN

  1. (Optional) Migrate Custom Configurations.
    Migrate any custom configuration settings into the new default files in the conf directory (/opt/mapr/spark/spark-<version>/conf). Also, if you previously configured Spark to use the Spark JAR file from a location on the MapR Filesystem, you need to copy the latest JAR file to the MapR Filesystem and reconfigure the path to the JAR file in the spark-defaults.conf file. See Configure Spark JAR Location.
  2. If Spark SQL is configured to work with Hive, copy hive-site.xml file into the conf directory (/opt/mapr/spark/spark-<version>/conf).
  3. Run configure.sh -R.
  4. Delete the old Spark directory from /opt/mapr/spark. For example, if you upgraded from Spark 2.1.0 to 2.3.1, you need to delete /opt/mapr/spark/spark-2.1.0.
    Starting with the MEP 6.1.0 release, for Spark 2.2.1 and later versions, after an upgrade the old directory is automatically removed. Only the new directory and the directory with the timestamp is present.