Spark 1.6.1-1611 Release Notes

The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 1.6.1 Release Notes.

Spark Version 1.6.1
Release Date December 9, 2016
MapR Version Interoperability See MEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark
GitHub Release Tag 1.6.1-mapr-1611
Package Names The following packages are associated with this release:
  • mapr-spark-1.6.1.201612010646-1.noarch.rpm
  • mapr-spark-master-1.6.1.201612010646-1.noarch.rpm
  • mapr-spark-historyserver-1.6.1.201612010646-1.noarch.rpm
  • mapr-spark_1.6.1.201612010646_all.deb
  • mapr-spark-master_1.6.1.201612010646_all.deb
  • mapr-spark-historyserver_1.6.1.201612010646_all.deb

New in This Release

This version of Spark supports integration with Hive and MapR Event Store For Apache Kafka. However, note the following exceptions:

  • Hive on Spark is not supported.
  • Spark-SQL is supported, but it is not fully compatible with Hive; for details, see the Apache Spark documentation and the MapR Spark documentation.
  • If you want to integrate Spark 1.6.1 with MapR Event Store For Apache Kafka, you must install the latest Kafka 0.9.0.0 package.
  • Full support of MapR Event Store For Apache Kafka is available only on MapR 5.2 clusters.

Patches

This MapR release includes the following new patches since the previous release of MapR Spark 1.6.1. For details, refer to the commit log for this project in GitHub.

GitHub Commit Number Data (YYYY-MM-DD) MapR Patch Number and Description
50fbc6b 2016-09-23 [SPARK-17210][SPARKR] The sparkr.zip archive is not distributed to executors when running sparkr in RStudio.
bba4dfa 2016-09-29 MAPR-24602: Spark RStudio jobs failed with a java.net.SocketTimeoutException.
e837cec 2016-10-10 [MAPR-24635] The Spark History Server takes a long time to display the stage details.
847bc6f 2016-10-24 [MAPR-24870] Reverted changes that were made in MAPR-23243.
3edeef6 2016-10-28 [MAPR-24415] SPARK_JAVA_OPTS is deprecated.
175c9b5 2016-11-04 [MAPR-13522] [CORE] Executor should kill itself when it is unable to send a heartbeat to the driver more than n times.
92e042e 2016-11-04 [MAPR-13522] [CORE] Fixed the exit log place for heartbeat.
26990db 2016-11-11 [MAPR-25223] SparkR did not work.

Known Issues and Limitations

  • MAPR-17271: On secure clusters, the MapR Control System (MCS) does not display links for Spark Master and Spark History Server.
  • MAPR-19761: On a secure cluster, MapR does not support the Spark SQL Thrift JDBC server. When the cluster is secure, the Spark Thrift Server will not start.
  • Spark versions up to and including 2.3.0 have the following security vulnerability: CVE-2018-1334 Apache Spark local privilege escalation vulnerability

Resolved Issues

None.