Spark 2.1.0-1710 Release Notes

This section provides reference information, including new features, patches, and known issues for Spark 2.1.0-1710.

The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 2.1.0 Release Notes.

Spark Version 2.1.0
Release Date November 2017
MapR Version Interoperability See MEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark
GitHub Release Tag 2.1.0-mapr-1710
Maven Artifacts http://repository.mapr.com/maven/
Package Names Navigate to https://package.mapr.com/releases/MEP/ and select your MEP and OS to view the list of package names.
Important:
  • Full support of MapR Streams is available only on MapR 5.2 and later clusters.
  • Spark 2.1 can connect to Hive Metastore 2.1. But, features of Hive added after Hive 1.2 are not supported by Spark.
  • Spark Standalone and Spark on YARN can only run on clusters in MRv2 (YARN) mode. They are not supported on clusters in MRv1 (classic) mode.
  • MapR 6.0 and MEP 4.0 introduce "Simplified Security". If you are using these versions and enable security on your MapR cluster, MapR scripts automatically configure Spark security features.

Hive Support

This version of Spark supports integration with Hive. However, note the following exceptions:

New in This Release

  • Simplified Security - Starting in the MapR 6.0 and MEP 4.0 releases, you can use the "Enable Security" check box in the installer to enable security for the core platform and the installed ecosystem components. Alternatively, running configure.sh -R enables Spark security features if you have enabled security on your MapR cluster. See Security with Spark Standalone and Security with Spark on YARN for more information.

Patches

This MapR release includes the following new patches since the latest MapR Spark 2.1.0 release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Date (YYYY-MM-DD) Comment
f1a4e96 2017/08/03 [SPARK-16845][SQL] Fix code generation to prevent `GeneratedClass$SpecificOrdering` from growing beyond 64 KB
13def70 2017/08/07 [MAPR-28339] Fix race condition resulting in delete failures, when Spark SQL queries use "SaveAsTable" API
c64da80 2017/08/09 [MAPR-20331][SQL] Enhance Hive partition pruning predicate pushdown
b27ea82 2017/08/17 [MAPR-28705] Fix late arriving message
c170554 2017/08/17 [MAPR-18971][CORE] Upgrade Netty to 4.0.43.Final version
0de12ba 2017/08/18 [MAPR-28659] Fix issue where executor threads in Spark executor are stuck in a lock state
6a33e6f 2017/08/28 [MAPR-19307][PYSPARK] Ensure user conf is propagated to SparkContext
85004b1 2017/08/29 [MAPR-28460] Fix impersonation when data read from MapR Database via Spark-Hive
a75bbe8 2017/08/29 [SPARK-39] Remove ambiguous dependencies from Spark classpath
c5a87b0 2017/09/06 [MAPR-29052] Use waitForConsumerAssignment() instead of consumer.poll(0) to avoid initialization error in MapR Event Store For Apache Kafka client
cf96fdd 2017/09/11 [MAPR-29106] - Fix unsafe deserialization in Apache Spark launcher API
7942de3 2017/09/18 [SPARK-18991][CORE] Change ContextCleaner.referenceBuffer to use ConcurrentHashMap to make it faster
3d44f64 2017/09/18 [SPARK-45] Move Spark-OJAI connector code to Spark github repo
e2a4e2a 2017/09/25 [SPARK-20358][CORE] Fix executors failing stage on interrupted exception by cancelled tasks
e39c4b7 2017/10/03 [SPARK-69] Fix license problem when reading from JSON and writing to MapR Database
d444627 2017/10/06 [MAPR-29014]Fix message offset for MapR Core 6.0

Known Issues

Resolved Issues

None.