Spark 1.6.1-1703 Release Notes

The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 1.6.1 Release Notes.

Spark Version 1.6.1
Release Date April 2017
MapR Version Interoperability See MEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark
GitHub Release Tag 1.6.1-mapr-1703
Maven Artifacts http://repository.mapr.com/maven/
Package Names See Package Names for MapR Expansion Packs (MEPs)
Note: For some important Spark limitations, See "Known Issues and Limitations" later in this release note.

New in This Release

This version of Spark supports integration with Hive. However, note the following exceptions:

Patches

This MapR release includes the following new patches since the latest MapR Spark 1.6.1 release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Date (YYYY-MM-DD) MapR Patch Number and Description
9275605 2017/03/16 Fixed gap handling.
dbf3b16 2017/03/10 Removed overridden methods count, countApprox, isEmpty from kafkaRDD.
3014abc 2017/03/07 [MAPR-26040] Kafka streaming module was refactored (#96).
62def95 2017/03/06 [BUILD][1.6] Fix compilation.
e092b35 2017/02/23 Merge pull request #90 from mapr/spark-1.6.3-porting.
04cbcb0 2017/02/23 [SPARK-17696][SPARK-12330][CORE] Partial backport of to branch-1.6.
9be6167 2017/02/13 [SPARK-14357][CORE] Properly handle the root cause being a commit denied exception.
566a5a9 2017/02/13 [SPARK-16182][CORE] Utils.scala -- terminateProcess() should call Process.destroyForcibly() if and only if Process.destroy() fails.
31843fc 2017/02/13 [SPARK-17618] Fix invalid comparisons between UnsafeRow and other row formats.
2781e0b 2017/02/13 [SPARK-8428][SPARK-13850] Fix integer overflows in TimSort.

Known Issues and Limitations

  • To integrate Spark 1.6.1 with MapR Event Store For Apache Kafka, you must install the latest Kafka 0.9.0.0 package.
  • Full support of MapR Event Store For Apache Kafka is available only on clusters with MapR 5.2 and later.
  • MAPR-17271: On secure clusters, the MapR Control System (MCS) does not display links for Spark-Master and Spark-HistoryServer.
  • MAPR-19761: On a secure cluster, MapR software does not support the Spark SQL Thrift JDBC server. When the cluster is secure, the Spark Thrift server will not start.
  • Spark versions up to and including 2.3.0 have the following security vulnerability: CVE-2018-1334 Apache Spark local privilege escalation vulnerability

Resolved Issues

None.