Spark 2.2.1-1901 (MEP 5.0.2) Release Notes

This section provides reference information, including new features, patches, and known issues for Spark 2.2.1.

The notes below relate specifically to the MapR Distribution for Apache Hadoop. This release of Spark has backward-compatibility changes, see the open-source Spark 2.1.0 Release Notes for more information.

These release notes contain only MapR-specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.

Spark Version 2.2.1
Release Date February 2019
MapR Version Interoperability See MEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark
GitHub Release Tag 2.2.1-mapr-1901
Maven Artifacts http://repository.mapr.com/maven/
Package Names Navigate to https://package.mapr.com/releases/MEP/ and select your MEP and OS to view the list of package names.
Important:
  • Spark 2.2 can connect to Hive Metastore 2.1. But, features of Hive added after Hive 1.2 are not supported by Spark.
  • Starting from Spark 2.2.1 and MEP 5.0.0 Kafka version is updated to 1.0.1.
  • Spark Yarn and Standalone modes are supported only on clusters in MRv2 (YARN) mode. It is not supported on clusters in MRv1 (classic) mode.
  • MapR 6.0 and MEP 5.0 and later introduce security by default. If you are using these versions and enable security on your MapR cluster, MapR scripts automatically configure Spark security features.

Hive Support

This version of Spark supports integration with Hive. However, note the following exceptions:

New in This Release

None.

Patches

This MapR release includes the following new patches since the latest MapR Spark 2.2.1 release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Date (YYYY-MM-DD) Comment
e444b4a 2018/10/01 MapR [SPARK-316] Backporting request for SPARK-22062 and SPARK-21475
fc8076b 2018/10/01 [SPARK-22033][CORE] BufferHolder, other size checks should account for the specific VM array size limitations
f1dd6d9 2018/10/17 [MAPR-32263] Seek called on unsubscribed partitions
874aefc 2018/10/24 [MAPR-32290] Spark processing offsets when messages are already ttl in first batch
7ecab19 2018/11/19 [SPARK-357] consumer groups are prepeneded with a "service_" prefix
9cf9c83 2018/12/21 [MAPR-26258] hbasecontext.HBaseDistributedScanExample fails
f0bb7de 2019/01/04 MapR [SPARK-390] Fix for hive version
b346a9f 2019/01/04 MapR [SPARK-311] Spark beeline uses default ssl truststore instead of mapr ssl truststore
22d8192 2019/01/08 MapR [SPARK-382] Codegen issue fixed
66ada4a 2019/01/08 MapR [SPARK-319] Fix for sparkR version check
70fc922 2019/01/22 MapR [SPARK-419] Update hive-maprdb-json-handler jar for spark 2.2.1

Known Issues

  • None.

Resolved Issues

  • None.