Spark 2.0.1-1611 Release Notes

The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 2.0.1 Release Notes.

Spark Version 2.0.1
Release Date December 9, 2016
MapR Version Interoperability See MEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark
GitHub Release Tag 2.0.1-mapr-1611
Maven Artifacts http://repository.mapr.com/maven/
Package Names See Package Names for MapR Expansion Packs (MEPs)

New in This Release

This version of Spark supports integration with Hive. However, note the following exceptions:

Note: For the API changes in this version, see Spark 2.0.1 API.

Patches

This MapR release includes the following new patches on top of the Apache Spark 2.0.1 release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Number Data (YYYY-MM-DD) MapR Patch Number and Description
bed0ba9 2016-11-14 [MINOR] Fixed the spark-jars.sh script.
287a9b2 2016-11-09 [MINOR] Removed the unused config template.
b05ddad 2016-11-09 [MAPR-24603] Could not launch the Beeline shell after starting the Spark Thrift Server.
b64908b 2016-11-07 Fixed a syntax error in V09DirectKafkaWordCount example (#75).
b6788d0 2016-11-01 Spark 2.0.1 MapR-Streams Python API (#73).
b1dff21 2016-10-31 [MAPR-24415] SPARK_JAVA_OPTS was deprecated (#71).
361effd 2016-10-26 [MAPR-24863] Added spark-defaults template for installer (#69).
a1e0492 2016-10-20 [MAPR-25002] FileNotFoundException during SparkHiveExample (#68).
7357889 2016-10-12 Added the Kafka streaming producer (#66).
d0c05d4 2016-10-07 [SPARK-17707][WEBUI] The Web UI prevented the spark-submit application from finishing.
23f9305 2016-09-08 [SPARK-15487][WEBUI] Spark Master UI to reverse proxy Application and Workers UI.
d99c601 2016-10-05 Fixed Scala style for SparkHiveExample.
fb327ad 2016-10-05 Changed the Kafka 0.9 version to 2.0.1.
cd374ce 2016-10-05 Changed version to 2.0.1-mapr-SNAPSHOT.
73a298a 2016-09-20 [MAPR-24491] The HBase classpath might contain Hive libraries.
f010164 2016-09-14 Minor fix for previous commit.
f3ec15a 2016-09-14 Added script for MAPR-24374.
4292173 2016-09-13 Some minor changes to spark-defaults.conf.
36e847d 2016-09-13 Changed the default HBase version to 1.1.1 in compatibility.version.
d721e09 2016-09-12 Refactored a streaming example.
9aba5a1 2016-09-12 [MAPR-24470] HiveFromSpark test failed in yarn-cluster mode.
70809c9 2016-09-08 Changed the Hive execution version to 1.2.0.
acabece 2016-09-01 Added spark streaming integration with Kafka 0.9 and Mapr-Streams.
641faa3 2016-08-31 Added MapR Repo.
1a2e25e 2016-08-22 [MAPR-23559] Spark PID in /opt/mapr/pid.
690788c 2016-05-25 [MAPR-22940] Failed to connect Spark Beeline (after Spark Thrift Server is started) on Kerberos cluster.
fe2c1ce 2016-05-03 Removed Hive jars from generated classpath for Hive.
85c45d1 2016-05-03 Fixed hardcoded Hive library path.
1124353 2016-04-29 [MAPR-23203] Removed derby jars from the generated Hive classpath.
3bc0cd4 2016-04-13 [MAPR-23068] Spark samples failed in Hue 3.9.
506bce9 2016-03-16 [MAPR-18865] Unable to submit Spark apps from Windows client.
9c2fda0 2015-10-20 Skip maven clean task on the parent module.
b699ada 2015-07-30 New: Issue with running Hive commands in Spark.
49d106a 2015-07-28 Spark warden.services.conf should have a dependency on the CLDB.
d7425bd 2015-05-20 Removed DFS shuffle settings.
5ed864c 2015-04-19 Fixed bugs in the logic to avoid SSH for localhost.
ad12fcf 2015-04-14 Copied every file in the conf directory into the distribution package.
4b15fb8 2015-04-10 Created spark-defaults.conf for MapR.
e9e4ab1 2015-04-08 Avoided SSH to localhost when stopping slaves.
c8405cd 2015-02-25 Added htrace jar to Spark classpath for HBase 0.98.
c502093 2015-02-25 Added the Scala library.
254aecf 2015-02-25 Supported HBase classpath computation in util script.
f6319e4 2015-02-24 Created ext-util.
c850a32 2015-02-24 Added external conf and scripts.
93279f6 2015-02-06 Enabled SPARK_HIVE mode while building.
c851ac8 2015-10-12 Built Spark on MapR.
7e1b86c 2015-11-25 Spark Master failed to start in HA mode.
bcf7665 2015-02-04 Updated the datanucleus jar in Spark for Bug 21228.
fbaf081 2015-08-22 Changed the dependencies for MapR and increased the Hadoop version from 2.2.0 to 2.7.0.

Known Issues and Limitations

Known issues:

  • MAPR-17271: On secure clusters, the MapR Control System (MCS) does not display links for Spark-Master and Spark-HistoryServer.
  • MAPR-25052:The Spark Thrift Server does not start on clusters secured by MapR-SASL.
  • Spark versions up to and including 2.3.0 have the following security vulnerability: CVE-2018-1334 Apache Spark local privilege escalation vulnerability
Limitations:
  • Spark 2.0.1 Standalone mode is supported only on clusters in MRv2 (YARN) mode.
  • Full support of MapR Event Store For Apache Kafka is available only on clusters with MapR 5.2 and later.

Resolved Issues

None.