MapR 4.0.x Documentation : Setting up the MapR Metrics Database

To use MapR Metrics, set up a MySQL database to log metrics data. The MapR distribution for Apache Hadoop does not include MySQL. Download and install MySQL separately and then perform the configuration steps to enable the MapR Metrics database.

Requirements

  • Enable remote access on the MySQL server. All JobTracker and webserver nodes in the cluster must have access to the MySQL server.
  • Use MySQL version 5.1 or greater.
  • Install the mapr-metrics package on all JobTracker and webserver nodes.

Installation

Install mysql server from the EPEL repository.

The MapR Job Metrics service depends on version 16 of the MySQL client library. SUSE 11 systems have version 15 by default. On SUSE systems, manually install version 16 of the MySQL client library by following these steps:

  1. Set up the repository by creating a file named openSUSE-11.4-11.4-0.repo in the /etc/zypp/repos.d directory with the following text:

    [openSUSE-11.4-11.4-0]
    name=openSUSE-11.4-11.4-0
    enabled=1
    autorefresh=0
    baseurl=http://download.opensuse.org/distribution/11.4/repo/oss/
    type=yast2

  2. Issue the following commands to install the library:
    zypper clean
    zypper refresh
    zypper install libmysqlclient16

# yum install mysql-server

Initial Configuration

  1. Start the MySQL server:

    # /etc/init.d/mysqld start 

     

  2. Set a password for the MySQL root user.

    # mysqladmin -u root password <new password>

    MySQL passwords cannot contain the reserved character &.

  3. Log into MySQL and create a new database and schema for use with MapR Metrics.

    # mysql -u root -p 
    Enter password:
    mysql> CREATE DATABASE metrics;
    mysql> SHOW DATABASES;
  4. Create a 'maprmetrics' user, grant the user privileges, and verify that the user has the granted privileges.

    For remote SQL servers, provide the following grants:

    mysql> CREATE user 'maprmetrics'@'%' IDENTIFIED BY 'mapr';
    mysql> grant all privileges on metrics.* to 'maprmetrics'@'%' with grant option;
    mysql> show grants for 'maprmetrics'@'%';

    For SQL servers on a local node with the metrics role installed, provide the following grants:

    mysql> CREATE user 'maprmetrics'@'%' IDENTIFIED BY 'mapr';
    mysql> CREATE user 'maprmetrics'@'localhost' IDENTIFIED BY 'mapr';
    mysql> grant all privileges on metrics.* to 'maprmetrics'@'%' with grant option;
    mysql> grant all privileges on metrics.* to 'maprmetrics'@'localhost' with grant option;
    mysql> show grants for 'maprmetrics'@'%';
    mysql> show grants for 'maprmetrics'@'localhost';
  5. Create the maprmetrics schema.

    # mysql -u maprmetrics -h <hostname or IP where the SQL server is running> -p -vvv < /opt/mapr/bin/setup.sql > /opt/mapr/logs/setup_sql_results.txt
    
  6. Restart the hoststats service.

    # maprcli node services -name hoststats -action restart -filter '[csvc==hoststats]'
    

The MapR Metrics database uses the InnoDB table storage engine by default. The Metrics database uses transactional tables, which require a storage engine that supports locking to prevent data corruption. To change the storage engine, edit the following line in the setup.sql script:
SET storage_engine='InnoDB';

To display a list of the available storage engines, issue the SHOW ENGINES command from the mysql> prompt.

Client Installation

Install the MySQL client on JobTracker and webserver nodes in your cluster.

# yum install mysql 
  1. Test the MySQL connection from the JobTracker and webserver nodes to the MySQL server:

    # mysql -u maprmetrics -h <hostname or IP where sql server is running> -p 
    Enter password:	

Troubleshooting the Client

  • Check that the port used by the MySQL server is set to the default of 3306. To use a non-default port, specify the port number with the -P option for the mysql command.
  • Check the /etc/my.cnf file to verify that the following lines are not present:

    # skip-networking tells mysql server to not listen for tcp/ip connections at all
    skip-networking
    # bind-address should be 0.0.0.0 which tells mysql server to listen on all interfaces, not only localhost (which is what "bind-address = 127.0.0.1" does)
    bind-address = 127.0.0.1
    
  • Restart the MySQL server after any changes to the /etc/my.cnf:

    # /etc/init.d/mysqld restart

Specify MySQL parameters to MapR

  1. On each node in the cluster that has the mapr-metrics package installed, specify your MySQL database parameters in one of the following ways:
  • To specify the MySQL database parameters from the command line, run the configure.sh script:

    # configure.sh -R -d <host>:<port> -du <database username> -dp <database password> -ds metrics
    
  • To specify the MySQL database parameters from the MapR Control System (MCS), click Navigation > System Settings > Metrics to display the Configure Metrics Database dialog. In the URL field, enter the hostname and port of the machine running the MySQL server. In the Username and Password fields, enter the username and password of the MySQL user. Schema is set to metrics by default.

The username you provide must have full permissions when logged in from any node in the cluster.
When you change the Metrics configuration information from the initial settings, you must restart the hoststats service on each node that reports Metrics data. You can restart the hoststats service with the command maprcli node services -name hoststats -action restart.

Customizing the Database Name

You can change the name of the database from the default name of metrics by editing the setup.sql script before sourcing the script.

Troubleshooting Metrics

  • Verify that the mapr-metrics package is installed on all JobTracker and webserver nodes.
  • Verify that all nodes have the correct configuration information in the /opt/mapr/conf/db.conf and /opt/mapr/conf/hibernate.cfg.xml files.
  • Check the /opt/mapr/logs/adminuiapp.log file for database-related messages or errors similar to this example:

    2012-09-26 12:41:38,740 WARN  com.mchange.v2.resourcepool.BasicResourcePool [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2]: com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@663f3fbd -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
    java.sql.SQLException: Access denied for user 'maprmetrics'@'10.10.80.116' (using password: YES)
    
  • Verify that the hoststats service is running:

    # ps -ef | grep hoststats
    root      8411  8049  0 11:01 pts/0    00:00:00 grep hoststats
    root     26607     1  0 Sep25 ?        00:03:04 /opt/mapr/server/hoststats 5660 /opt/mapr/logs/TaskTracker.stats
    
  • Check the /opt/mapr/logs/hoststats.log file for database-related errors.
  • Verify that the /opt/mapr/hadoop/hadoop-0.20.2/conf/hadoop-metrics.properties file exists and contains the following entries:

    maprmepredvariant.class=com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContext
    maprmepredvariant.period=10
    maprmapred.class=com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContextFinal
    maprmapred.period=10
    

    Add these entries if they are absent, then restart the JobTracker.

  • Verify that the /opt/mapr/conf/warden.conf file has the following entries:

    rpc.drop=false
    hs.rpcon=true
    hs.port=1111
    hs.host=localhost
    
  • Check the JobTracker logs for errors or entries related to the string com.mapr.job.mngmnt.hadoop.metrics.

Attachments: