Known Issues at Release (MapR 6.1.0)

You may encounter the following known issues after upgrading to Version 6.1. This list is current as of the release date. For a dynamic list of all known issues for all MapR product releases, refer to MapR Support List of all known issues

Where available, the workaround for an issue is also documented in this topic. MapR regularly releases maintenance releases and patches to fix issues. We recommend checking the release notes for any subsequent maintenance releases to see if one or more of these issues are fixed.

Installation and Configuration Issues

You can see generic installation issues here: MapR Installer Known Issues.

Upgrade Issues

MAPRYARN-129
Issue: For some clusters that are upgraded from MapR 6.0.1 to 6.1.0, using any Hadoop command generates the following java.lang.ClassNotFoundException:
hadoop fs -ls /
log4j:ERROR Could not instantiate class [org.apache.hadoop.yarn.WorldWritableLogAppender].
java.lang.ClassNotFoundException: org.apache.hadoop.yarn.WorldWritableLogAppender
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
at org.apache.log4j.Logger.getLogger(Logger.java:104)
at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:262)
at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:108)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.commons.logging.impl.LogFactoryImpl.createLogFromClass(LogFactoryImpl.java:1025)
at org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:844)
at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
at org.apache.hadoop.fs.FsShell.<clinit>(FsShell.java:43)

This issue only affects clusters that use scheduler debug logging as described in 31038 of Known Issues at Release (MapR 6.0.1). The issue occurs because the hadoop-yarn-common-2.7.0-mapr-1808.jar in MapR 6.1.0 does not contain the org.apache.hadoop.yarn.WorldWritableLogAppender Java class.

Workaround: Remove the following lines from the /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/log4j.properties file:
yarn.debug.controller.logger = INFO, ED
yarn.debug.controller.log.file=yarn-mapr-scheduling-debug.log
yarn.debug.controller.log.maxfilesize=256MB
yarn.debug.controller.log.maxbackupindex=20

log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.scheduler.
DebugController=${yarn.debug.controller.logger}
log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.scheduler.
DebugController=false
 
log4j.appender.ED=org.apache.hadoop.yarn.WorldWritableLogAppender
log4j.appender.ED.File=${hadoop.log.dir}/${yarn.debug.controller.log.file}
log4j.appender.ED.MaxFileSize=${yarn.debug.controller.log.maxfilesize}
log4j.appender.ED.layout=org.apache.log4j.PatternLayout
log4j.appender.ED.layout.ConversionPattern=%d{ISO8601} : %m%n 
30422
Issue: Using the configure.sh -R command after an upgrade from 5.2.x to 6.0 or later can return error messages for some MapR Monitoring components. For example:
/opt/mapr/server/configure.sh -R 
Configuring Hadoop-2.7.0 at /opt/mapr/hadoop/hadoop-2.7.0 Done configuring Hadoop Node setup configuration: drill-bits fileserver hbinternal nfs nodemanager opentsdb Log can be found at: /opt/mapr/logs/configure.log 
ls: cannot access /opt/mapr/hbinternal: No such file or directory Configuring opentsdb 
getopt: unrecognized option '--unsecure' 
getopt: unrecognized option '-EC' 
usage: /opt/mapr/opentsdb/opentsdb-2.3.0/bin/configure.sh [-nodeCount <cnt>] [-nodePort <port> -nodeZkCount <zkCnt>] [-nodeZkPort <zkPort>] [-R] -OT "ip:port,ip1:port," -Z "ip:port,ip1:port," 
 
/opt/mapr/server/configure.sh -R 
Configuring Hadoop-2.7.0 at /opt/mapr/hadoop/hadoop-2.7.0 
Done configuring Hadoop 
Node setup configuration: drill-bits fileserver grafana hbinternal historyserver nfs nodemanager spark-historyserver 
Log can be found at: /opt/mapr/logs/configure.log 
ls: cannot access /opt/mapr/hbinternal: No such file or directory 
ls: cannot access /opt/mapr/spark-historyserver: No such file or directory 
Configuring grafana 
getopt: unrecognized option '--unsecure' 
getopt: unrecognized option '-EC' 
usage: /opt/mapr/grafana/grafana-4.1.2/bin/configure.sh [-nodeCount <cnt>] [-nodePort <port>] [-grafanaPort <port>] [-secureCluster] [-loadDataSourceOnly] [-R] -OT "ip:port,ip1:port,"

These errors are generated because a newer version of the server configure.sh uses certain options that do not exist with older ecosystem components.

Workaround: None. The upgrade can succeed in spite of the errors, and the current ecosystem configuration is left unchanged. The error messages can be ignored.

Indexing and MapR Drill

MD-2218 (previously 28041)
Running multiple queries in sequence can cause Drillbits to run out of direct memory. Limiting records in large batches to 5KB may prevent this issue from occurring. Should Drill return an out of memory exception, issue the following command to restart the Drillbits:
$ maprcli node services -name drill-bits -action restart -nodes <node host names separated by a space>
28096
If you use the Google Guava library in your application and encounter the following error:
Caused by: java.lang.NoSuchMethodError: com.google.common.base.Stopwatch.createStarted()Lcom/google/common/base/Stopwatch;
You must use version 18.0 of the library. If you use Maven, you can use the following dependency:
            <dependency>
              <groupId>com.google.guava</groupId>
              <artifactId>guava</artifactId>
              <version>18.0</version>
              </dependency>       

MapR Database

MAPRDB-1375
If you have a MapR Database JSON table with multiple column families and the table has a secondary index, after you perform a series of update operations (for example: set, increment, decrement, replace, append, etc.) on the table, the index may have duplicate rows.
MAPRDB-1520
If you have an OR condition that references a container field path more than once, and MapR Database uses an index to process the query, then the query may return incorrect results. You can workaround this problem by forcing MapR Database to not use indexes. See Forcing Secondary Index Usage in OJAI for details.
27057
When you issue a maprcli table changelog add command, during the initial sync when the existing data is copied to the change log, the end-to-end latency during the initial 5 minutes can be high. After the initial sync, latency will be small, specifically 2-4 seconds.

MapR Client

23624
The same MapR client can access both secure and nonsecure clusters; however, a MapR client that is configured to access a secure cluster can access a nonsecure cluster only if these conditions are met:
  • The secure cluster must be listed first in mapr-clusters.conf.
  • A user must obtain a ticket for the secure cluster even if the user wants to access only the nonsecure cluster.

MapR Control System

MON-4891
Description: The MCS displays an Oops error on the Admin > User Settings > SMTP page when the mapr.smtp.provider is not any of the 3 allowed options (Gmail, Office 365, SMTP). This impacts all upgrade from 5.x to 6.x where your SMTP provider setting is "other".
Resolution: Run the following command to resolve this error if your SMTP provider is not Gmail or Office 365:
maprcli config save -values {"mapr.smtp.provider":"smtp"}
If your SMTP provider is Office 365, run the following command:
maprcli config save -values {"mapr.smtp.provider":"office"}
MON-4788
Description: When running large number of table (get, scan, index) operations, the API server may become unresponsive and you may see following error: GC overhead limit exceeded.
Resolution: Check the heap size and increase it based on available system resources or set up a load balancer to distribute the HTTP requests (load) to other API servers. To increase heap size:
  1. Open /opt/mapr/apiserver/bin/mapr-apiserver.sh file.
  2. Modify the value for JVMXMS and JVMXMX (under JVM tuning).

    For example, modify the following in the file:

    # JVM tuning
    JVMXMS=${JVMXMS:-1024m}
    JVMXMX=${JVMXMX:-1024m}
  3. Save and close the file.
  4. Restart apiserver for the changes to take effect.

    For example, run the following command to restart the apiserver:

    maprcli node services -action restart -name apiserver -nodes `hostname`
MON-4776
Description: You cannot mount or unmount multiple volumes at the same time by selecting them from different pages in the Volumes pane (of the Data > Volumes page).
Workaround: Increase the number of rows, if possible, in the Volumes pane and select the volumes from the same page to mount or unmount.
MON-4701
Description: The table metrics list view displays metric values calculated using endTime-startTime. If you select a time range like Today, This Week, This Month, where the end time is in the future, no data is displayed in the columns.
Workaround: Choose Today so far, This week so far, This month so far instead to view the most current data.
MON-3817
Description: The Node Health by Topology pane does not reflect the most recent topology of the node if the node was just moved to a different/new topology.
Workaround: Navigate to another tab or page and then return to the page to view the updated topology for the node in the Node Health by Topology pane.
MON-3709
Description: The list of services in the Services page do not refresh per the configured refresh rate settings.
Workaround: To view the latest status for the services, reload the page in the browser.
MON-3476
Issue:In the MapR Control System, table permissions can only be modified by the owner/creator of the table.
Workaround: User, who is not the owner but has table admin permissions, must use the CLI to modify table permissions.
MON-3452
Issue: After selecting the Include System Volumes checkbox to display system volumes in the Volumes list pane, if you set quota (using the link in the Quota column), the system volumes are no longer displayed in the list of volumes even though the system volumes checkbox is still selected.
Workaround: Deselect and then re-select the Include System Volumes checkbox to display the system volumes in the list of volumes.
MON-3438
Issue: You may experience some latency in the Admin > Cluster Settings > Alarms page when you click the name of an alarm to display the Alarm Settings window for the alarm.
MON-3425
Issue: When you filter the list of alarms in the Alarms summary page, the page does not refresh and display the correct list of alarms if there are muted alarms.
MON-3407
Issue: If you are upgrading from a pre-6.0 version, the MapR Control System might display 0 instances of webserver running in the Services page. This is because starting from v6.0, the UI static files are in mapr-webserver and the mapr-apiserver runs the server that sends the queries. For more information on webserver in v6.0, see Setting Up the MapR Control System.
MON-3404
Issue: The topic search field in the Stream (detail) > Topic tab does not display correctly in Firefox.
Workaround: Use Chrome (v58 and later) to perform a topic search.
MON-3247
Issue: The MapR Control System does not display NFS edge nodes and POSIX client nodes that are currently connected to the cluster when you sort the list of nodes by topology.
Workaround: Use the node list command to retrieve the list of client nodes currently connected to the cluster.
MON-2409
Issue: When you try to navigate to another cluster by clicking on the cluster name from the dropdown list at the top of the page, the Overview page for the current cluster displays and it takes a few minutes to display the MCS login page for the other cluster.
MON-2374
Issue: If a node has multiple IP addresses, the log viewer link for the services in the nodes information page is broken because the Kibana URL contains multiple IP addresses.
Workaround: Remove all irrelevant IP addresses in the URL and use the modified URL to view logs for the service.
MON-2321
Issue: If Elasticsearch fails, the Services page does not show the nodes on which Elasticsearch has failed.
MON-2219
Issue: If you add Streams license separately, you may not be able to view the Streams page.
Workaround: Contact MapR Support.
MON-2218
Issue: When you preserve snapshots, there is no notification (success message) or indication (Expires On column still does not show No Expiration) even though the operation succeeded.
Workaround: Refresh the page after preserving a snapshot to see the No Expiration setting in the Expires On column.
MON-2076
Issue: When you select topics to remove, deselect all the topics in the Remove Topic confirmation window and click Cancel, then select a topic in the page by clicking on the checkbox, the Remove Topic window opens again even if you did not click on the Remove Topic button.
Workaround: Refresh the page.
MON-2018
Issue: In the Add Change Log page, after entering field to add in the Publish Selected Field Path text field, the Add Field button is disabled.
Workaround: Click outside the text field to enable the Add Field button.
MON-2011
Issue: When adding or editing the compression setting on a replica, selecting Inherited does not have any effect. The default value is lz4.
Workaround: Do not select Inherited as the compression setting on a replica.
MON-2009
Issue: When modifying replica of a binary table with three or more column families, if you select different set of columns to replicate for different column families, the operation might result in an error.
MON-1994
Issue: If you adjust the system time, the time stamp associated with the alarm in the Active Alarms pane might show negative value.
MON-1968
Issue: In the User Disk Usage page, you cannot remove the email address after setting an email address for an entity by editing the properties.
MON-1929
Issue: The Save Changes button in the Access Control Expression builder page is disabled after entering user, group, or role name in text field.
Workaround: Click outside the field to enable the Save Changes button.
MON-1810
Issue: When editing multiple volumes, you cannot undo changes to Auditing and Accountable Entity fields by clicking .
MON-1755
Issue: When creating a mirror volume, after entering the name of the source volume on the remote cluster and selecting the remote cluster on which the volume resides, you may see an error that the source volume does not exist.
Workaround: Re-enter the name of the source volume on the remote cluster.
MON-1221
Issue: In the Alarm Settings window, if you click the cursor in the Additional email recipients text field and then press the return/enter key in the keyboard without entering an email address, the Cluster Settings page displays once again.

MapR Monitoring

OTSDB-72
Issue: The memory allocated to OpenTSDB can be insufficient, resulting in empty graphs and out-of-memory or GC overhead limit exceeded errors.
Workaround: Increase the default memory for OpenTSDB by making the following changes on all OpenTSDB nodes:
  1. Edit the /opt/mapr/conf/conf.d/warden.opentsdb.conf file to change:
    service.heapsize.max=2000
    service.heapsize.min=2000
    to
    service.heapsize.max=6000
    service.heapsize.min=6000
  2. Edit the /opt/mapr/opentsdb/opentsdb-*/etc/init.d/opentsdb file to change:
    $
    {JVMXMX:=-Xmx2000m -Xss1m -XX:MaxMetaspaceSize=128m}
    to
    $
    {JVMXMX:=-Xmx6000m -Xss1m -XX:MaxMetaspaceSize=128m}
  3. Restart the OpenTSDB service:
    maprcli node services -name opentsdb -nodes <space-separated list of OpenTSDB nodes> -action restart
SPYG-994
Issue: The time-to-live for MapR Event Store For Apache Kafka metrics is fixed at 30 days and cannot be reduced to lower disk space usage.
SPYG-934
Issue: On secure Ubuntu 14.04 or 16.04 clusters, Elasticsearch fails to generate a keystore password if the uuid-runtime package is not installed. uuid-runtime is one of the MapR Installer prerequisites.
Workaround: Install the uuid-runtime package from the default Ubuntu repository:
apt-get install uuid-runtime

MapR PACC

See MapR PACC Known Issues.

NFSv4

32364
Issue: Any running IO on NFSv4 mount (with kerberos) is stuck if the krb5 ticket expires for the current user. The mount point also hangs and becomes inaccessible.
Workaround: Restart rpcgssd service with the new ticket to make the mount point accessible and re-trigger the IO to proceed.

Oozie

29634
Issue: On a secure cluster, Oozie jobs can fail when the timeline server for the Hive-on-Tez user interface is configured with MapR-SASL.
Workaround: Restart the resource manager and timeline server. For more information, see Configuring the Timeline Server to Use the Hive-on-Tez User Interface.

PJDTest Suite Compliance

28912
Issue: When trying to open a FIFO on a FUSE mounted filesystem, permissions to perform the operation are not checked.
29234
Issue: The FUSE-based POSIX can only trace at millisecond level and the test suite utimensat in pjdtestsuite fails because the test suite sets time in nanoseconds.
29237
Issue: Any user can set time using touch -t for any file on a FUSE mounted filesystem.

Timeline Server

See Hive-on-Tez User Interface Known Issues.