Spark 2.2.1-1912 (MEP 5.0.4) Release Notes

This section provides reference information, including new features, patches, and known issues for Spark 2.2.1.

The notes below relate specifically to the MapR Distribution for Apache Hadoop. This release of Spark has backward-compatibility changes, see the open-source Spark 2.2.1 Release Notes for more information.

These release notes contain only MapR-specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.

Spark Version 2.2.1
Release Date December 2019
MapR Version Interoperability See MEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark
GitHub Release Tag 2.2.1-mapr-1912
Maven Artifacts http://repository.mapr.com/maven/
Package Names Navigate to https://package.mapr.com/releases/MEP/ and select your MEP and OS to view the list of package names.
Important:
  • Spark 2.2 can connect to Hive Metastore 2.1, but features of Hive added after Hive 1.2 are not supported by Spark.
  • Starting from Spark 2.2.1 and MEP 5.0.0, Spark uses Kafka version 1.0.1.
  • Spark Yarn and Standalone modes are supported only on clusters in MRv2 (YARN) mode. They are not supported on clusters in MRv1 (classic) mode.
  • MapR 6.0 and MEP 5.0 and later introduce security by default. If you are using these versions and enable security on your MapR cluster, MapR scripts automatically configure Spark security features.

Hive Support

This version of Spark supports integration with Hive. However, note the following exceptions:

New in This Release

None.

Fixes

This MapR release includes the following new fixes since the latest MapR Spark 2.2.1 release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Date (YYYY-MM-DD) Comment
e77ddc4 2019/06/04 MapR [SPARK-545] PySpark streaming package for kafka-0-9 fixed
3bd05f3 2019/06/06 MapR [SPARK-541] Avoid duplication of the first unexpired record
fa252d8 2019/06/14 MapR [SPARK-333] Render application UI init page if driver is not up
d4fde38 2019/07/02 [SPARK-24002][SQL] Task not serializable caused by org.apache.parquet.io.api.Binary$ByteBufferBackedBinary.getBytes
90499d5 2019/07/31 MapR [SPARK-592] Add possibility to use start-thriftserver.sh script with 2304 port
41e68c4 2019/10/15 MapR [SPARK-595] Spark cannot access hs2 through zookeeper
c8111ff 2019/11/12 MapR [SPARK-575] Warning messages in spark workspace after the second attempt to login to job's UI
e17c039 2019/11/12 MapR [SPARK-641] backport SPARK-21357 into mapr-spark-2.2.1

Known Issues

  • None.

Resolved Issues

  • None.