Spark 2.0.1-1707 Release Notes

This section provides reference information, including new features, patches, known issues, and limitations for Spark 2.0.1-1707.

The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 2.0.1 Release Notes.

Spark Version 2.0.1
Release Date August 2017
MapR Version Interoperability See MEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark
GitHub Release Tag 2.0.1-mapr-1707
Maven Artifacts http://repository.mapr.com/maven/
Package Names See Package Names for MapR Expansion Packs (MEPs)
Important:
  • Full support of MapR Streams is available only on MapR 5.2 and later clusters.
  • You cannot submit Spark jobs in YARN mode when the cluster is running in MRv1(classic) mode. This applies even if you have installed and configured YARN in your cluster.
  • When integrating Hive with Spark 2.0.1-1707, use Hive 1.2.-1707, which contains the fix for MAPR-26310.

Hive Support

This version of Spark supports integration with Hive. However, note the following exceptions:

Patches

This MapR release includes the following new patches since the latest MapR Spark 2.0.1 release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Number Date (YYYY-MM-DD) MapR Patch Number and Description
bcb1640 2017/05/22 [MAPR-27519] Improve performance of calculating web UI counters for Kafka-streaming.
ab0a3f2 2017/05/16 [SPARK-19019][PYTHON][BRANCH-2.0] Fix hijacked `collections.namedtuple`. Port cloudpickle changes needed for PySpark to work with Python 3.6.0.
9faac4a 2017/05/05 [MAPR-26414] Fix Spark History Server memory leak.
46b1913 2017/05/04 [SPARK-33] Fix streaming example.
ab1f040 2017/05/04 [SPARK-19146][CORE] Drop more elements when stageData.taskData.size > retainedTasks.
5e61eb7 2017/05/04 [MAPR-26287] Remove unnecessary code from hadoop-version-picker.sh.
2b16ad3 2017/05/04 [MAPR-26414] Fix Spark History Server memory leak.

Known Issues and Limitations

  • MAPR-17271: On secure clusters, the MapR Control System (MCS) does not display links for Spark-Master and Spark-HistoryServer.
  • MAPR-25052: Spark Thrift Server does not start on clusters secured by MapR-SASL.
  • Spark versions up to and including 2.3.0 have the following security vulnerability: CVE-2018-1334 Apache Spark local privilege escalation vulnerability

Resolved Issues

None.