What's New in MEP 6.0.0

Provides a summary of the new functionality in MapR Expansion Pack (MEP) 6.0.0.

MEP 6.0.0 provides you with support for MapR 6.1.0 and new features in MapR Database JSON; Amazon S3 API; and Apache Hadoop YARN, Kafka, Spark, Hive, Tez, Hue, Livy, Flume, Oozie, and Sqoop.

New Features in MapR Database JSON

Node.js and Python OJAI Clients

With MEP 6.0.0, you can use the Node.js and Python OJAI clients to write MapR Database JSON applications. These clients provide lightweight libraries that support the OJAI API and alternatives to the Java OJAI client. You can connect to MapR Database JSON from middleware components, and add, update, and query documents in a MapR Database JSON table. Both clients tailor their features to developers of those respective languages.

The following topics provide more information about these clients:

New Features in Apache Kafka

Support for Apache Kafka Streams

Kafka Streams is a programming library that enables you to create Java or Scala streaming applications and, specifically, building streaming applications that transform input topics into output topics.

Kafka Streams enables you to build moderately complex operational streaming applications faster by offloading common functions such as failure recovery, joins and enrichment, and aggregations and windowing.

See Kafka Streams and Kafka Streams 1.1-1808 Release Notes for more information.

Support for Apache Kafka KSQL

Streaming SQL for Apache Kafka (KSQL) is an open source streaming SQL engine that implements continuous, interactive queries. KSQL enables you to query, read, write, and process data in real time and scale, using SQL commands. KSQL interacts directly with the Kafka Streams API, removing the requirement of building a Java application. See KSQL and KSQL 4.1.1-1808 Release Notes for more information.

New Ecosystem Component

Support for MapR Object Store with S3-Compatible API

The MapR Object Store with S3-Compatible API (MapR Object Store) provides you with a REST interface compatible with the Amazon S3 API standard to store and retrieve data from the MapR platform object store in the form of files. For more information, see MapR Object Store with S3-Compatible API.

New Features in YARN

YARN Resource Calculation Based on Labels

MEP 6.0.0 implements correct steady and instantaneous fair shares, headroom, and maximum resource calculation for queues with label-based scheduling (LBS). For more information, see YARN Resource Calculation Based on Labels.

Support for Azure Data Lake Store

You can use Azure Data Lake Store (ADLS) as an input source or an output destination for all applications. For more information, see Support for ADLS.

New Features in Spark

New Spark Version

MEP 6.0.0 includes support for Spark 2.3.1. For more information, see Spark 2.3.1-1808 (MEP 6.0.0) Release Notes.

New Features in Hive and Tez

New Hive and Tez Versions

MEP 6.0.0 includes support for Hive 2.3 and Tez 0.9. For more information, see Hive 2.3 and Tez 0.9. You can also refer to Hive 2.3.3-1808 (MEP 6.0.0) Release Notes and Tez 0.9.1-1808 (MEP 6.0.0) Release Notes.

Support for UPDATE, INSERT INTO, and MERGE statements in Hive
  • You can use the UPDATE statement to update primitive, complex, and complex nested data types in MapR Database JSON tables, using the Hive connector.

    For more information, see Understanding the UPDATE Statement.

  • You can use the INSERT INTO statement to insert or overwrite rows in nested MapR Database JSON tables, using the Hive connector.

    For more information, see Understanding the INSERT INTO Statement.

  • You can use the MERGE statement to efficiently perform record-level INSERT and UPDATE operations within Hive tables.

    For more information, see Understanding the MERGE Statement.

Support for Splitting Hive Log Files

You can split Hive log files into HiveServer2 and Metestore log files by process ID. For more information, see Splitting Hive Logs into HiveServer2 and Metastore logs by Process ID.

Support for SQL Standards-Based Hive Authorization

You can configure SQL standards-based authorization to enable fine-grained access control with SQL commands. For more information, see SQL Standards-Based Hive Authorization.

Support for Auto-Generated PEM files for WebHCat REST API on a MapR-SASL cluster

For more information, see Requirements for Using Automatically Generated PEM Files.

Support for Configuring JDBC Connection String with SSL Encryption Enabled or Disabled

For more information, see Configuring JDBC Connection String with SSL Encryption Enabled or Disabled.

Support for Zero RM HA for the new Tez UI

For more information, see Hive-on-Tez User Interface.

New Features in Hue and Livy

New Hue and Livy Versions

MEP 6.0.0 includes support for Hue 4.2.0 and Livy 0.5.0. For more information, see Hue 4.2.0-1808 (MEP 6.0.0) Release Notes.

New Features for Livy Security

For more information, see Configure Livy with Security.

New Feature in Flume

Support for SSL By Default On Secure Clusters for Flume Thrift Source and Sink

For more information, see Configure SSL for Flume Thrift Source and Sink.

New Feature in Oozie

Support for Encrypting the Oozie Database User Password

For more information, see Encrypt the Oozie Database User Password.

New Features in Sqoop

New Sqoop Version

MEP 6.0.0 includes support for Sqoop 1.4.7. The Teradata Connector for Hadoop (TDCH) version is updated to v1.5.4. For more information, see Sqoop 1.4.7-1808 (MEP 6.0.0) Release Notes.

New Features in Upgrade

Support for Preserving User Configuration

With MEP 6.0.0 you can preserve user configuration in the Hive, Tez, Hue, Impala, Sentry, Oozie, Sqoop, Kafka, Spark, Livy, and HttP-FS components. For more information, see Preparing to Upgrade the Expansion Pack.