Spark 2.1.0-1803 (MEP 4.1.1) Release Notes

This section provides reference information, including new features, patches, and known issues for Spark 2.1.0-1803.

The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 2.1.0 Release Notes.

Spark Version 2.1.0
Release Date March 2018
MapR Version Interoperability See MEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark
GitHub Release Tag 2.1.0-mapr-1803
Maven Artifacts http://repository.mapr.com/maven/
Package Names Navigate to https://package.mapr.com/releases/MEP/ and select your MEP and OS to view the list of package names.
Important:
  • Spark 2.2 can connect to Hive Metastore 2.1. But, features of Hive added after Hive 1.2 are not supported by Spark.
  • Spark Yarn and Standalone modes are supported only on clusters in MRv2 (YARN) mode. It is not supported on clusters in MRv1 (classic) mode.
  • MapR 6.0 and MEP 4.1 and later introduce built-in security. If you are using these versions and enable security on your MapR cluster, MapR scripts automatically configure Spark security features.

Hive Support

This version of Spark supports integration with Hive. However, note the following exceptions:

New in This Release

  • PAM for Spark Web UIs on secure clusters.

Patches

This MapR release includes the following new patches since the latest MapR Spark 2.2.1 release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Date (YYYY-MM-DD) Comment
0a08b10 2018/01/18 [MAPR-30536] Spark SQL queries on Map column failed after upgrade
f160e85 2018/02/01 MapR [SPARK-143] Added PAM for Spark UIs for secure clusters
46da505 2018/02/07 [MAPR-30583] PartitioningAwareFileIndex changed to getFileBlockLocations in a parallel way
bf7296a 2018/02/08 MapR [SPARK-152] Incorrect date string parsing fixed
4c01f15 2018/03/01 [SPARK-21181] Release byteBuffers to suppress netty error messages
9473a5f 2018/03/07 MapR [SPARK-171] StackOverflowException in equals method in DBMapValue
3b5d608 2018/03/19 MapR [SPARK-190] Spark R NoSuchElementException during start
3e3f07e 2018/03/23 MapR [SPARK-143] Spark History Server does not require login for secured-by-default clusters
44f7810 2018/03/25 MapR [SPARK-194] Redirect to Spark History server

Known Issues

Spark versions up to and including 2.3.0 have the following security vulnerability:

Resolved Issues

Bug 25770: Removed invalid error message displayed at times during the running of Spark application:
ERROR MapRFileSystem: Failed to delete path, error: No such file or directory
          (Tickets: MapR-28620, MapR-25770, MFS-1897)