Drill 1.11.0-1710 Release Notes

This section provides reference information, including new features, patches, known issues, and limitations for Drill 1.11.0-1710.

The following release notes apply to the 1.11.0 version of the Apache Drill component included in the MapR Converged Data Platform.

Version 1.11.0
Release Date November 2017
MapR Version Interoperability See MEP Components and OS Support, Interoperability Matrix, and Drill Support Matrix.
Package Names Package Names for MapR Expansion Packs (MEPs)

New in this Release

  • Secondary index support on MapR Database for JSON tables.
  • MapR default security configuration that secures the cluster and ecosystem components using MapR security (mapr tickets) to provide authentication, authorization, and encryption.
  • Client to Drillbit Encryption using SSL/TLS.
  • Spill to disk for the Hash Aggregate operator (in addition to the Sort operator).
  • Throttle-based resource assignment (queuing feature).
  • Drill integration with Zeppelin.
In addition to the features mentioned above, Drill 1.11 also includes the following features and improvements:
MapR Tracking Numbers Features
MD-2490 Addition of an internal options table.
MD-1813 Multiplexing session over a connection.
MD-1655 Provide an option to persist Query Profiles in a Drillbit server's memory
MD-1245 MapR-SASL Plugin for ODBC/Drill (Authentication & Encryption)
MD-536 SASL support between Drillclient and Drillbit
MapR Tracking Numbers Improvements
MD-2962 Add a UDF to decode encoded fieldpath
MD-2871 Restricted scan (for non-covering) gets over-parallelized due to scan range size
MD-2799 DRILL-5811 Large number of "Failure finding Drillbit" messages when using MapR Filesystem
MD-2789 Fix correct batch size for merge back during non-covering query.
MD-2693 Drill should use `com.mapr.db.index.IndexDesc.getNullOrdering()` to decide if an index can be used for order by
MD-2656 Drill needs to return more specific error for DB
MD-2647 Enable UTF-8 support in query string by default
MD-2506 Standardize on JDBC driver that gets shipped with Drill and sqlline
MD-2422 Drill performance - doing less than 40 queries/sec
MD-2416 RestrictedJsonTableGroupScan should support projection pushdown
MD-2403 Drillbit fails to start when only keystore path is provided without keystore password.
MD-2339 Additional log events for debugging index selection
MD-2319 Query Containing Wildcard in String Matching Filter Runs 4x Slower Than Impala
MD-2301 Support Impersonation without authentication for REST API
MD-2278 Include flags controlling secure Drill ZK ACLs in the secure installer
MD-2225 Review the names of index planning config options
MD-2196 DRILL-4264: Allow field names to include dots
MD-2192 Provide log tracing of a query submitted by an OJAI client for user debugging
MD-2074 Memory Fragmentation Fundamentals Completion
MD-2067 Handle the case of order-by with no filter conditions
MD-2002 Operator Specific Handle for Empty Batches
MD-1987 Drill should read MapR Database tablets in parallel
MD-1962 Create configuration file for distribution specific configuration
MD-1836 Leverage hash index when doing secondary index planning
MD-1823 Add logging information for listing candidate indexes for a given query
MD-1815 Leverage MapR Database batch gets for better performance.
MD-1789 Cast function support pull request and code review process
MD-1786 Push down limit on MapR Database table scan
MD-1752 Throttling-based per-query memory assignment
MD-1673 Add Statistics support for MapR Database
MD-1665 Provide query level option to pick specific indexes
MD-1643 Disable table caching by default
MD-1498 Plan queries against MapR Database JSON faster
MD-1318 Improve Low Latency Performance of Drill
MD-1209 Configuring ZooKeeper Access Controls for Drill on a Secure Cluster

Patches

None

Known Issues and Limitations

This release of Drill includes the following known issues:
MapR Tracking Numbers Known Issues
MD-3171 Drillbit down due to fs client assertion in Java_com_mapr_fs_jni_MapRTableTools_GetScanner
MD-3162 DRILL-5946: Document maprdb format plugin options
MD-3160 Hit ExpressionParsingException: Expression has syntax error! line 1:0:no viable alternative at input 'TIME'
MD-3155 Simba JDBC driver dependency issues with third party applications
MD-3131 Memory leak by HashPartitionSender
MD-3108 planner.memory.max_query_memory_per_node should be default to 4GB
MD-3089 Some queries with join on two or more tables only pick non-covering index plans
MD-3071 Queries using hash index plans (intermittently) fail with SYSTEM ERROR: RpcException: Data not accepted downstream.
MD-3047 Apache Drill JDBC driver from mapr-drill package not works
MD-3041 Identify closed jdbc connection
MD-3040 need a better error message - RpcException: Drillbit (demo-179.lab) does not require auth, but auth is enabled.
MD-3037 Performance issues with queries against MapR Database JSON tables (full table scans)
MD-3031 Duplicate columns projected from MapR Database Scan for large IN condition
MD-2979 Memory leak by ParquetRowGroupScan
MD-2950 Full outer join on two empty hive tables throws IllegalArgumentException: Full outer join not currently supported
MD-2948 SI: Non-covering queries hang after drillbit restart
MD-2848 IteratorValidatorBatchIterator is throwing exception when assertions are enabled.
MD-2838 After upgrading from MapR Drill 1.10.0 to 1.11.0, drillbit hangs on startup
MD-2837 Queries on MapR Database 5.x tables hang
MD-2809 Scalar replacement failures
MD-2791 Cancelled query using index plan logs several errors
MD-2790 Queries fail with various CONNECTION ERRORs
MD-2695 Test Framework - Test fails with DATA_READ_ERROR : Failed to load table into HBase
MD-2589 Drillbit startup has errors: "Can't load log handler java.util.logging.FileHandler"
MD-2544 drillbit jvm dumps core when querying a table with SI
MD-2528 Parquet int_64 causes UnsupportedOperationException in Drill
MD-2515 Evaluate Codegen Compilers for performance overhead
MD-2513 Querying last column in CSV causes IndexOutOfBoundsException
MD-2455 Query with INTERVAL in predicate does not return any rows
MD-2420 External sort with higher memory takes longer to finish vs. smaller memory
MD-2419 Query fails with OOM with queuing enabled and exchanges disabled
MD-2415 Resource leak during Hash Agg spill to disk longevity run
MD-2400 Random Jenkins unit test failure due to ".crc" files in spill directory
MD-2397 Zeppelin cannot connect to Drill through the JDBC driver on a secure MapR cluster when Zeppelin has Kerberos authentication enabled.
MD-2285 Filter with interval was not fully pushed down
MD-2218 Direct memory fragmentation cause OOM while running OJAI queries
MD-2141 Predicates on indexed field, joined by OR, do not return the correct results for time and timestamp data types
MD-2120 Order by queries on short and byte datatypes fail
MD-2112 Query with LIKE Predicate on Non-Varchar Fields Returns No Results
MD-2060 Hash Partitioner Sender taking unusual amount of time
MD-2041 Potential wrong result from spillable hash agg
MD-1964 OOM when Hash Aggr outputs large aggregate data types
MD-1923 Simple query failed when planner.use_simple_optimizer=true
MD-1867

SYSTEM ERROR: CannotPlanException when query MDB Json table

Resolved Issues

This release of Drill includes the following resolved issues:
MapR Tracking Numbers Resolved Issues
MD-2825 initcap function returning incorrect result
MD-2769 Default value of security.admin.user_groups and security.admin.users is "true"
MD-2738 Memory Limit Exception in External Sort
MD-2621 Fix XSS vulnerabilities in Drill
MD-2617 NullPointerException: Querying maprdb json tables
MD-2531 Error connecting to Drill from Data Stage through Simba ODBC driver
MD-2436 Issue with drill-config.sh checking java version
MD-2216 DRILL-5660: Drill 1.10 queries fail due to Parquet Metadata "corruption" from DRILL-3867
MD-2197 DRILL-4511: refresh over empty folder results in error, we need a better error message
MD-2195 DRILL-4755: StringIndexOutOfBoundsException seen with CONVERT_FROM function
MD-2146 DRILL-4970: Wrong results when casting double to bigint or int
MD-2144 DRILL-4720: MINDIR() and IMINDIR() functions return no results with metadata cache
MD-2130 Drill wrong old date-time values displaying
MD-2125 Queries fail with "Failed to create schema tree." when Impersonation is enabled and logins are anonymous
MD-1988 Spillable Hash Agg : OOM while constructing the hash table
MD-1980 DRILL-5130: UNION ALL difference in results
MD-1970 ArrayIndexOutOfBoundException occurs when querying MapR Database JSON with Drill
MD-1946 DRILL-5140: Fix CompileException in run-time generated code when record batch has large number of fields
MD-1945 DRILL-4139: Fix parquet partition pruning for BIT, INTERVAL and DECIMAL types
MD-1922 Use indexDesc instead of indexFid in getIndexTable API to support HashPartitioned Indexes
MD-1890 Drill Explorer on the mac does not connect when started as an app
MD-1848 Drill not able to read MapR Database when FQDN is longer than 64 characters
MD-1828 DRILL-5507 Millions of "Failure finding Drillbit running on host" info messages in foreman logs
MD-1776 Issue with column alias for nested table calculated column when using sum ()
MD-1671 Have to restart drillbits whenever hive metastore is restarted.
MD-1661 A query with limit 0 on MapR Database binary table not allowing filter to push past project
MD-1657 Random Error : Flatten does not support inputs of non-list values
MD-1640 Drill fails to compare multi-byte characters from hive table(DRILL-3250)
MD-1633 Query remains in CANCELLATION_REQUESTED state after "java.io.IOException: Broken pipe"
MD-1618 Moving a directory which contains a cache file causes subsequent queries to fail
MD-1607 Reverse a String: IndexOutOfBoundsException
MD-1597 simba and mapr jdbc driver differ in implementation
MD-1586 Pam authentication with Centrify doesn't work by JPam restriction
MD-1576 JBOSS Drill Connectivity - NullPointerException
MD-1524 linux ODBC is ALWAYS returning 64K as the length of a varchar or char
MD-1515 Drill planner/optimizer pushes down the Filter columns into Scan even after Filter is eliminated
MD-1473 Query failed with error Statement "break OrOP2" is not enclosed by a breakable statement with label "OrOP2"
MD-1455 Hash aggregate does not support schema changes
MD-1382 Use simple optimizer for low latency operational queries
MD-1361 Out of heap running CTAS against text delimited
MD-1359 Wrong results after set planner.enable_nljoin_for_scalar_only=false
MD-1339 Query text show as empty string in profile when queries are executed using PreparedStatements
MD-1256 unset MAPR_TICKETFILE_LOCATION for sqlline
MD-1238 Wrong result with Drill displaying date fields with parquet files generated by spark
MD-1237 Cancelled query stuck into CANCELLATION_REQUESTED forever
MD-1205 Simple query with only one join condition failed with "Hash join does not support schema changes"
MD-1168 Drill Web UI Page Source Has Links To External Sites
MD-1115 DRILL-5464: HashAgg throws a SchemaChangeException when atleast 1 scan fragment reads 0 rows
MD-965 UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL].
MD-853 Query on TPC-H SF100 dataset fails with "Hash join does not support schema changes" [MapR Database JSON Tables]
MD-752

Remove mapr jars from Drill packages and source them from $MAPR_HOME/lib