Spark 2.2.1-1803 (MEP 5.0.0) Release Notes

This section provides reference information, including new features, patches, and known issues for Spark 2.2.1-1803.

The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 2.2.1 Release Notes.

Spark Version 2.2.1
Release Date March 2018
MapR Version Interoperability See MEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark
GitHub Release Tag 2.2.1-mapr-1803
Maven Artifacts http://repository.mapr.com/maven/
Package Names Navigate to https://package.mapr.com/releases/MEP/ and select your MEP and OS to view the list of package names.
Important:
  • Spark 2.2 can connect to Hive Metastore 2.1. But, features of Hive added after Hive 1.2 are not supported by Spark.
  • Starting from Spark 2.2.1 and MEP 5.0.0 Kafka is updated to 1.0.1.
  • MapR 6.0 and MEP 5.0 and later introduce built-in security. If you are using these versions and enable security on your MapR cluster, MapR scripts automatically configure Spark security features.
  • Spark Master port was changed to 8580.

Hive Support

This version of Spark supports integration with Hive. However, note the following exceptions:

New in This Release

Patches

This MapR release includes the following new patches since the latest MapR Spark 2.2.1 release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Date (YYYY-MM-DD) Comment
5430a1d 2018/01/22 MapR [SPARK-16] Spark 2.2.1 porting
b3a4ffa 2018/01/22 [MAPR-30228] Spark example job failed with "/opt/mapr/conf/ssl_keystore (Permission denied)" error on Spark 2.1.X MEP 4.0 - MapR SASL security enabled
aeb9e9d 2018/01/22 MapR [SPARK-144] Add insertToMapRDB method for rdd for Java API
489e21d 2018/01/23 MapR [SPARK-16] Change Spark version in Warden files and configure.sh
548e492 2018/02/01 MapR [SPARK-143] Added PAM for Spark UIs for secure clusters
2499aba 2018/02/06 MapR [SPARK-135] Spark 2.2 with MapR Streams (Kafka 1.0)
1b37b08 2018/02/07 MapR [SPARK-21] Structured Streaming MapR Database Sink created
2d9e466 2018/02/07 [MAPR-30583] InMemoryFileIndex changed to getFileBlockLocations in a parallel way
1a2e864 2018/02/08 MapR [SPARK-152] Incorrect date string parsing fixed
cca410c 2018/02/15 MapR [SPARK-153] Exception in spark job with configured labels on yarn-client mode
6f83937 2018/02/20 MapR [SPARK-159] Added possibility to configure secure ports as part of security by default
f341e85 2018/02/20 MapR [SPARK-155] Changed Spark Master port from 8080
f417a68 2018/02/26 MapR [SPARK-161] Include Kafka Structured streaming jar to Spark package
77ed36f 2018/02/28 MapR [SPARK-164] Update Kafka version to 1.0.1-mapr in Spark Kafka Producer module
5a390c1 2018/03/07 MapR [SPARK-170] StackOverflowException in equals method in DBMapValue
45cea4a 2018/03/19 MapR [SPARK-154] Spark R NoSuchElementException during start
3725eed 2018/03/21 MapR [SPARK-191] Incorrect work of MapR Database Sink 'complete' and 'update' modes fixed
844642e 2018/03/23 MapR [SPARK-143] Spark History Server does not require login for secured-by-default clusters
71cd6d2 2018/03/25 MapR [SPARK-194] Redirect to Spark History server
1c093b1 2018/03/26 MapR [SPARK-188] Could not connect to thrift server via spark beeline on kerberos cluster
4eff5df 2018/04/02 [SPARK-198] Update hadoop dependency version to 2.7.0-mapr-1803 for Spark 2.2.1
d324bfa 2018/04/04 [SPARK-205] Change Kafka dependencies versions

Known Issues

Spark versions up to and including 2.3.0 have the following security vulnerability:

Resolved Issues

None.