Troubleshooting Metrics

  • Verify that the mapr-metrics package is installed on all JobTracker and webserver nodes.
  • Verify that all nodes have the correct configuration information in the /opt/mapr/conf/db.conf and /opt/mapr/conf/hibernate.cfg.xml files.
  • Check the /opt/mapr/logs/adminuiapp.log file for database-related messages or errors similar to this example:

    2012-09-26 12:41:38,740 WARN  com.mchange.v2.resourcepool.BasicResourcePool [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2]: com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@663f3fbd -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
    java.sql.SQLException: Access denied for user 'maprmetrics'@'10.10.80.116' (using password: YES)
  • Verify that the hoststats service is running:

    # ps -ef | grep hoststats
    root      8411  8049  0 11:01 pts/0    00:00:00 grep hoststats
    root     26607     1  0 Sep25 ?        00:03:04 /opt/mapr/server/hoststats 5660 /opt/mapr/logs/TaskTracker.stats
  • Check the /opt/mapr/logs/hoststats.log file for database-related errors.
  • Verify that the /opt/mapr/hadoop/hadoop-0.20.2/conf/hadoop-metrics.properties file exists and contains the following entries:

    maprmepredvariant.class=com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContext
    maprmepredvariant.period=10
    maprmapred.class=com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContextFinal
    maprmapred.period=10

    Add these entries if they are absent, then restart the JobTracker.

  • Verify that the /opt/mapr/conf/warden.conf file has the following entries:

    rpc.drop=false
    hs.rpcon=true
    hs.port=1111
    hs.host=localhost
  • Check the JobTracker logs for errors or entries related to the string com.mapr.job.mngmnt.hadoop.metrics.