Spark 1.6.1-1707 Release Notes

This section provides reference information, including new features, patches, known issues, and limitations for Spark 1.6.1-1707.

The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 1.6.1 Release Notes.

Spark Version 1.6.1
Release Date August 2017
MapR Version Interoperability See MEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark
GitHub Release Tag 1.6.1-mapr-1707
Maven Artifacts http://repository.mapr.com/maven/
Package Names See Package Names for MapR Expansion Packs (MEPs)
Important:
  • To integrate Spark 1.6.1 with MapR Streams, you must install the latest Kafka 0.9.0.0 package.
  • Full support of MapR Streams is available only on MapR 5.2 and later clusters.
  • When integrating Hive with Spark 2.0.1-1707, use Hive 1.2.-1707, which contains the fix for MAPR-26310.

Hive Support

This version of Spark supports integration with Hive. However, note the following exceptions:

Patches

This MapR release includes the following new patches since the latest MapR Spark 1.6.1 release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Date (YYYY-MM-DD) MapR Patch Number and Description
c352e23 2017/05/16 [MAPR-27339] Fix failures when Spark jobs write to Hive tables.
0fdd46e 2017/05/15 [SPARK-16664][SQL] Fix persist calls on DataFrames with more than 200 columns.

Known Issues and Limitations

  • MAPR-17271: On secure clusters, the MapR Control System (MCS) does not display links for Spark-Master and Spark-HistoryServer.
  • MAPR-19761: On a secure cluster, MapR software does not support the Spark SQL Thrift JDBC server. When the cluster is secure, the Spark Thrift server will not start.
  • Spark versions up to and including 2.3.0 have the following security vulnerability: CVE-2018-1334 Apache Spark local privilege escalation vulnerability

Resolved Issues

None.