Spark 2.1.0-1808 (MEP 3.0.4 and MEP 4.1.2) Release Notes

This section provides reference information, including new features, patches, and known issues for Spark 2.1.0-1808.

The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 2.1.0 Release Notes.

Spark Version 2.1.0
Release Date September 2018
MapR Version Interoperability See MEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark/tree/2.1.0-mapr-1808
GitHub Release Tag 2.1.0-mapr-1808
Maven Artifacts http://repository.mapr.com/maven/
Package Names Navigate to https://package.mapr.com/releases/MEP/ and select your MEP and OS to view the list of package names.
Important:
  • Spark 2.2 can connect to Hive Metastore 2.1. But, features of Hive added after Hive 1.2 are not supported by Spark.
  • Spark Yarn and Standalone modes are supported only on clusters in MRv2 (YARN) mode. They are not supported on clusters in MRv1 (classic) mode.

Hive Support

This version of Spark supports integration with Hive. However, note the following exceptions:

Patches

This MapR release includes the following new patches since the latest MapR Spark 2.2.1 release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Date (YYYY-MM-DD) Comment
6fb5a8d 2018/04/25 [MAPR-31202] Spark History Server bug fixed. Redundant 'jsr311-api' artifact excluded
40d7890 2018/05/02 [MAPR-31274] Spark SQL Thrift Server cannot access hive-maprdbjson table
244a07c 2018/05/26 MapR [SPARK-226] Spark - pySpark Security Vulnerability
43d2314 2018/05/30

MapR [SPARK-214] Hive-2.1 poperties can't be read from a hive-site.xml as Spark uses Hive-1.2

a6b5a92 2018/06/13 [MAPR-31632] RM UI showing broken page for Spark jobs
232dbad 2018/07/18 [WEBUI] Avoid possibility of script in query param keys
455a796 2018/08/02 MapR [SPARK-300] Update Hive dependencies for spark 2.1.0
ec8bd03 2018/08/08 [SPARK-302] Local privilege escalation
2a1db27 2018/08/28 [SPARK-16986][WEB-UI] Converter Started, Completed and Last Updated to client time zone in history page
55195ae 2018/08/30 MapR [SPARK-279] Cannot connect to Spark Thrift Server with new Spark and Hive packages

Known Issues

  • You cannot connect to a Spark Thrift Server on a Kerberos-secured cluster as Kerberos and SSL are not compatible.

    Workaround: Modify the hive.server2.use.SSL to false in the hive-site.xml file.

  • When you install a secure (MapR-SASL) cluster using the MapR Installer, the configure.sh script configures Hive after Spark. As a result, Spark copies the wrong hive-site.xml file and the Spark and Hive integration may not work correctly and you may have problems connecting to Spark beeline.

    Workaround: Check the hive-site.xml file in the Spark home directory, and, if needed, rerun the configure.sh script or copy the hive-site.xml file from your Hive home directory and restart services.

  • Spark versions up to and including 2.3.0 have the following security vulnerability:

Resolved Issues

None.