Spark 2.3.2.0-1901 (MEP 6.1.0 and MEP 6.0.1) Release Notes

This section provides reference information, including new features, patches, and known issues for Spark 2.3.2.0.

The notes below relate specifically to the MapR Distribution for Apache Hadoop. This release of Spark has backward-compatibility changes, see the open-source Spark 2.3.2.0 Release Notes for more information.

These release notes contain only MapR-specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.

Spark Version 2.3.2.0
Release Date February 2019
MapR Version Interoperability See MEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark
GitHub Release Tag 2.3.2-mapr-1901
Maven Artifacts http://repository.mapr.com/maven/
Package Names Navigate to https://package.mapr.com/releases/MEP/ and select your MEP and OS to view the list of package names.
Important:
  • Starting with MEP 6.0.0, keyStore and trustStore passwords can be removed from the spark-defaults.conf file and can be set in the /opt/mapr/conf/ssl-client.xml file.
  • Starting with MEP 6.0.0, after an upgrade, configuration files of previous versions are saved in the /opt/mapr/spark directory.
  • The MapR 6.1 and MEP 6.0.0 release introduces "Simplified Security". If you are using these versions and enable security on your MapR cluster, MapR scripts automatically configure Spark security features.

Hive Support

This version of Spark supports integration with Hive. However, note the following exceptions:

New in This Release

For a complete list of all new features, refer to the open source documentation.

Patches

This MapR release includes the following new patches since the latest MapR Spark 2.3.1 release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Date (YYYY-MM-DD) Comment
fddc84f 2018/10/01 MapR [SPARK-325] Add examples for work with the MapR-DB JSON connector into the Spark project
a2b88ef 2018/10/16 MapR [SPARK-331] Remove snapshot versions of MapR dependencies from Spark
4f380b4 2018/10/23 [MAPR-32263] Seek called on unsubscribed partitions
e8d59b9 2018/10/24 [MAPR-32290] Spark processing offsets when messages are already TTL in the first batch
58b3115 2018/11/14 MapR [SPARK-352] Spark shell fails with "NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream" if java is not available in PATH
fcee5ab

2018/11/15

MapR [SPARK-350] Deprecate Spark Kafka-09 package
a2a3a0f 2018/11/19 MapR [SPARK-326] Investigate possibility of writing Java example for the MapR-DB OJAI connector
da4e0b3 2018/11/27 MapR [SPARK-356] Merge MapR changes from kafka-09 package into the kafka-10
3608c7b 2018/11/29 MapR [SPARK-319] Fix for SparkR version check
2aaa6a5 2018/12/03 MapR [SPARK-349] Update OJAI client to v3 for Spark MapR-DB JSON connector
71f6153 2018/12/10 MapR [SPARK-137] Analyze the warning during compilation of OJAI connector
f62df7a 2018/12/13 MapR [SPARK-369] Spark 2.3.2 fails with error related to zookeeper
a2bf94f 2018/12/18 [MAPR-26258] hbasecontext.HBaseDistributedScanExample fails
cd75768 2018/12/27 MapR [SPARK-372] Support 4 digits version scheme for Spark
182a7b1 2019/01/08 [MSPARK-24355] Spark external shuffle server improvement to better handle block fetch requests.
6b28052 2019/01/09 MapR [SPARK-381] Kafka version changed to '1.1.1-mapr-SNAPSHOT'
fec2162 2019/01/11 MapR [SPARK-374] Spark Hive example fails when we submit job from another(simple) cluster user
609c0a8 2019/01/11 MapR [SPARK-363] Hive requests fail with ClassNotFoundException
e6a8a6b 2019/01/17 MapR [SPARK-410] All submitted jobs fail for Spark 2.3.2.0
f032694 2019/01/18 MapR [SPARK-363] Hive version changed to '1.2.0-mapr-spark-MEP-6.0.0'
2686b34 2019/01/21 MapR [SPARK-373] Unexpected behavior during job running in standalone cluster mode
b991202 2019/01/22 MapR [SPARK-419] Update hive-maprdb-json-handler jar for spark 2.3.2.0
63acd83 2019/01/23 MapR [SPARK-374] Spark Hive example fails when we submit job from another (simple) cluster user

Known Issues

  • None.

Resolved Issues

  • None.