MapR 5.0 Documentation : Perform All Required Kerberos Admin Tasks

Create a Kerberos Principal and a keytab File for the CLDB

To use Kerberos to generate MapR tickets for users, enable Kerberos on the CLDB by creating a Kerberos identity on the Kerberos server used by the cluster and distributing that identity to the other CLDB nodes in the cluster.

If you are using strong encryption with Kerberos with the Oracle JDK, you will require a new Java Cryptography Extension (JCE) policy file.

MapR clusters do not provide Kerberos infrastructure. This section assumes you have a functioning Kerberos realm and your systems have the Kerberos client installed. The tips in this section assume a Linux-based Kerberos environment, and the specific commands for your environment may vary. Please consult with your Kerberos administrator for assistance.

Creating a Kerberos Identity for the CLDB

The CLDB requires a Kerberos server identity, but no other nodes do. By default, this identity takes the form mapr/<cluster name>. You can use configure.sh or edit the mapr-clusters.conf file to change this default. Use the following commands in a Linux-based Kerberos environment to set up the identity:

kadmin
	: addprinc -randkey mapr/my.cluster.com
	: ktadd -k /opt/mapr/conf/mapr.keytab mapr/my.cluster.com

Copy the resulting mapr.keytab file to the same location on every CLDB node. The mapr.keytab file must be owned and readable only by the mapr user. You can specify the location of the mapr.keytab file in the conf/mapr.login.conf file. The default location for mapr.keytab is /opt/mapr/conf.

Updating the keytab File

You can use the kadmin tool to update the server keys that are stored in the keytab file. Because the server tickets used to authenticate to the CLDB use the new keys immediately, you must copy the new keytab file to all the CLDB servers in the cluster immediately after updating the server keys.

To update the keytab file with a new key, run the following command:

kadmin
	: ktadd -k /opt/mapr/conf/mapr.keytab mapr/my.cluster.com

The CLDB automatically detects changes to the keytab file on systems that use Java 7 or later. Systems that use Java 6 require a CLDB restart to detect changes to the keytab file.

Starting with the 4.0.1 release of the MapR Distribution for Hadoop, Java 6 is deprecated in favor of Java 7 and Java 8.

Kerberos Command Summary

  • kinit: Creates a Kerberos ticket. Prompts the user for userid and password. After validating, Kerberos creates a ticket file in /tmp that is owned by the user. Use the -R option to renew an existing ticket. Kerberos credentials expire in 8-10 hours. Expired credentials must be renewed or replaced. By default, tickets can be renewed for up to 24 hours.
  • klist: Lists the contents of the user's ticket file.
  • kdestroy: Destroys the contents of the user's ticket file. The user is no longer authenticated.
  • kadmin: Used to administer Kerberos. The login for this command is implicitly <userid>/admin, since administrator ids typically end in /admin.
  • ktutil: Kerberos keytab maintenance utility. Used to combine, or alter Kerberos keytabs.

Configure SPNEGO to Secure Web UIs in the Cluster

MapR uses the Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO) to secure several Web UIs in the cluster, as well as the REST calls to the MapR Control System (MCS). The following procedure configures SPNEGO support for the web server nodes on your cluster.

  1. On each node in the cluster that will receive inbound SPNEGO traffic, generate a Kerberos principal with the user name HTTP, of the form HTTP/<webserver name>.

    Use the fully qualified domain name as the name in the principal. Although you could also use a short name or the IP address for the principal name, using the fully qualified domain name keeps the name consistent with principal names that configure.sh generates and includes in the mapr.login.conf file. Whatever you use as the principal name is what users will have to match exactly in a browser to access the web pages that are protected.

    Note that several services and components in a MapR cluster handle SPNEGO traffic, including the MCS, JobTracker, TaskTracker, and HBase, among others.

    You can name the keytab file mapr.keytab if that file does not already exist. If the mapr.keytab file already exists, generate the new principal to a different file name and merge it to the mapr.keytab file using the ktutil tool:

    ktutil
    	: addprinc -randkey HTTP/<webserver name>
    	: ktadd -k /opt/mapr/conf/mapr.keytab HTTP/<webserver name>


  2. Verify that the /opt/mapr/conf/mapr.login.conf file lists the correct principal in the MAPR_WEBSERVER_KERBEROS section.

To enable SPNEGO for MapR Control System (MCS) REST calls, on all nodes with the webserver role, add the following line to the /opt/mapr/conf/web.conf file:

mapr.rest.auth.methods=kerberos,basic

Restart MCS to make the change take effect.

Testing SPNEGO With curl

This example tests that the MCS is using GSS for REST calls made with curl.

Use the following command to verify that your version of curl supports SPNEGO - under the “Features" header, the output of the command should show either GSS-Negotiate or SPNEGO - under the “Features” header, the output of the command should show either GSS-Negotiate or SPNEGO:
# curl --version
curl 7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtmp rtsp smtp smtps telnet tftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP

Verify that you have a valid Kerberos ticket-granting-ticket with the kinit -p <user> command, then test curl with the following command:
curl --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt https://<web server node>:8443/rest/<API call> -k -v

This command returns HTTP/1.1 200 OK when curl is working correctly with SPNEGO.


Create a Kerberos Principal and a keytab File for JobTracker

In order to use JobTracker with Kerberos, you need to create a Kerberos principal and a keytab file. A Kerberos principal is a unique identity that represents a user or service in a Kerberos system. The user obtains a ticket for a principal name (through the kinit utility) and this ticket authenticates the user to the Kerberos server.

The keytab file contains principal names and their corresponding encrypted keys, or tickets.

Creating Kerberos Principals

Use the addprinc command at the kadmin console prompt to create two principals in the same realm as the MapR cluster:

  • a JobTracker user principal (used by SPNEGO webservers)

  • an HTTP user principal (for nodes that handle SPNEGO traffic)

For example, if the JobTracker service and the webserver service are running on a node called perfnode153.perf.lab and the realm is called dev-maprtech, the commands to add the JobTracker principal and the HTTP principal are:

kadmin: addprinc -randkey mapr/perfnode153.perf.lab@dev-maprtech
  addprinc -randkey HTTP/perfnode153.perf.lab@dev-maprtech

Creating a keytab File

Keytabs are created or appended to by extracting keys from the KDC database using the ktadd command inside the kadmin console prompt.

To create a keytab file for the JobTracker principal,  you use the same procedure that you use to create the keytab for the MapR-FS or mapred principal for a specific host.

  1. Create the keytab file for the JobTracker principal. Name this file mapr.keytab and put it in the directory /opt/mapr/conf on the machine running the JobTracker server, as shown:

    kadmin: ktadd -k /opt/mapr/conf/mapr.keytab mapr/perfnode153.perf.lab
  2. Set read-only permissions on the mapr.keytab file.

    $ sudo chmod 600 mapr:mapr /opt/mapr/conf/mapr.keytab
  3. Set the file's owner to the user running the JobTracker server (usually mapr).

    $ sudo chown mapr:mapr /opt/mapr/conf/mapr.keytab
  4. Copy the mapr.keytab file to all the nodes running Hue, Hive, httpfs, and Oozie services.

Create a Kerberos Principal and a keytab File for Flume

Skip this task if you are not using Flume.

On each node where Flume is installed, use the following commands in a Linux-based Kerberos environment to set up the identity and update the keytab file:

 

# kadmin
    : addprinc -randkey username/<FQDN@REALM>
    : ktadd -k /opt/mapr/conf/flume.keytab username/<FQDN@REALM>

The flume.keytab file must be owned and readable only by the mapr user.
 

Create a Kerberos Principal and a keytab File for HBase

Skip this task if you are not using HBase.

On all HBase nodes, perform the following steps:

  1. Install the krb5 packages and configure the Kerberos client as per the configuration for your environment.
  2. Set up the HBase Kerberos principal mapr/<fqdn>@<realm>. Each node requires a unique keytab file and Kerberos identity.
  3. Create an hbase.keytab file with the HBase Kerberos principal with the same process used to generate the CLDB keytab.
  4. Copy the hbase.keytab file to the /opt/mapr/conf directory.
  5. Use the chown command to change the keytab file's ownership to mapr:mapr.
  6. Use the chmod command to set the file's permissions to 600.
  7. Update the hbase-site.xml file by adding the following section:

    <property>
       <name>hbase.security.authentication</name>
       <value>kerberos</value>
     </property>
     <property>
      <name>hbase.security.authorization</name>
       <value>true</value>
     </property>
    <property>
       <name>hbase.rpc.engine</name>
       <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
     </property>
     <property>
       <name>hbase.regionserver.kerberos.principal</name>
       <value>mapr/_HOST@<KERBEROS_REALM></value>
     </property>
     <property>
       <name>hbase.master.kerberos.principal</name>
      <value>mapr/_HOST@<KERBEROS_REALM></value>
     </property>
  8. Replace the ${SIMPLE_LOGIN_OPTS} value of the MAPR_HBASE_SERVER_OPTS property with ${KERBEROS_LOGIN_OPTS} and the value of the  MAPR_HBASE_CLIENT_OPTS property with ${HYBRID_LOGIN_OPTS}. Also remove the default -Dzookeeper.sasl.client=false option from the definition of MAPR_HBASE_CLIENT_OPTS.

    These properties are located in the /opt/mapr/conf/env.sh file. 
     

Create a Kerberos Principal and a keytab File for Hive Metastore

Skip this task if you are not using Hive.

On each node where a Hive Metastore is installed, create a Kerberos server identity and add it to a keytab file. 

You can use the following commands in a Linux-based Kerberos environment to set up the identity and update the keytab file:

# kadmin
    : addprinc -randkey username/<FQDN@REALM>
    : ktadd -k /opt/mapr/conf/hive.keytab username/<FQDN@REALM>

The hive.keytab file must be owned and readable only by the mapr user.
 

Create a Kerberos Principal and a keytab File for HiveServer2

Skip this task if you are not using Hive.

On each node where HiveServer 2 is installed, create a Kerberos Identity and keytab.

You can use the following commands in a Linux-based Kerberos environment to set up the identity and update the keytab file:

# kadmin
    : addprinc -randkey username/<FQDN@REALM>
    : ktadd -k /opt/mapr/conf/hive.keytab username/<FQDN@REALM>

The hive.keytab file must be owned and readable only by the mapr user.
 

Create a Kerberos Principal and a keytab File for WebHCat

Skip this task if you are not using WebHCat.

To enable WebHCat to use Kerberos, complete the following steps on the node where WebHCat is installed.

  1. Create the principal HTTP/<FQDN@REALM> for WebHCat and add the principal to the keytab file.
    For example: 

    kadmin: addprinc -randkey HTTP/<FQDN@REALM>
    kadmin: xst -k /opt/mapr/HTTP.keytab HTTP/<FQDN>
  2. Verify the following:
    • The principal was added to the /opt/mapr/conf/HTTP.keytab file and that the file is only readable by the mapr user. For example:  chown mapr /opt/mapr/conf/HTTP.keytab
    • The node where the WebHCat server is running has an HTTP user with a valid maprlogin password.

Create a Kerberos Principal and a keytab File for HttpFS

Skip this task if you are not using HttpFS.

Each node running the httpFS service must have a keytab file (/opt/mapr/conf/mapr.keytab) and these two principals:

  • HTTP/<fully.qualified.domain.name>

  • mapr/<fully.qualified.domain.name>

For complete instructions on generating a Kerberos principal and keytab file, see Configuring Kerberos User Authentication.

To check whether the keytab already exists, and if it contains the two necessary principals, run the klist command with the -k (keytab keys), -e (encryption type) and -t (timestamp) options.

$ klist -ket /opt/mapr/conf/mapr.keytab

The output from this command displays the following information:

  • KVNO (key version number)

  • Timestamp (the time the key was generated)

  • Principal names

  • Encryption types

If the keytab file does not exist, or does not contain both principals, generate them by following these steps.

  1. Generate a Kerberos principal for the mapr user. The principal is of the form mapr/<fully.qualified.domain.name>@<your-realm>.com, where <fully.qualified.domain.name> is unique for each httpFS node. 
    In the following example, perfnode153.perf.lab@dev-maprtech.com is used for the <fully.qualified.domain.name>@<your-realm>.com.

    $ kadmin
    kadmin: addprinc -randkey mapr/perfnode153.perf.lab@dev-maprtech.com
  2. Generate a Kerberos principal for HTTP/<fully.qualified.domain.name>. This is required for Kerberos authentication of the httpFS server using HTTP SPNEGO. 

    $ kadmin
    kadmin: addprinc -randkey HTTP/perfnode153.perf.lab@dev-maprtech.com
  3. If the current node does not already have a keytab file created for another service, create one and name it mapr.keytab.

    kadmin: ktadd -k /opt/mapr/conf/mapr.keytab mapr/perfnode153.perf.lab

    Note that each node references the same keytab file (usually located at /opt/mapr/conf/mapr.keytab), and each keytab file can have multiple principals.

  4. Change the owner of the keytab file from the root user (the default) to the mapr user.

    $ chown mapr:mapr /opt/mapr/conf/mapr.keytab
  5. Set read-only permissions on the mapr.keytab file.

    $ chmod 600 mapr:mapr /opt/mapr/conf/mapr.keytab
  6. Verify Credentials in the keytab File

    To test that the credentials in the mapr.keytab file work, run the klist command with the -k (keytab keys), -e (encryption type) and -t (timestamp) options.

    $ klist -ket /opt/mapr/conf/mapr.keytab

    Verify that the output lists only one key version number (KVNO) for each principal name. If you see the same principal listed more than once with a different key version number, this could indicate a problem. The latest version number is used, which means you might not be able to log in to the node and authenticate with your user credentials. 

    Sample output for a node that has the httpFS and CLDB services installed is shown below.

    Keytab name: FILE:/opt/mapr/conf/mapr.keytab
    KVNO Timestamp         Principal
    ---- ----------------- --------------------------------------------------------
      2 07/18/14 18:50:07 mapr/perfnode153.perf.lab@dev-maprtech (aes256-cts-hmac-sha1-96)
      2 07/18/14 18:50:07 mapr/perfnode153.perf.lab@dev-maprtech (arcfour-hmac)
      2 07/18/14 18:50:08 mapr/perfnode153.perf.lab@dev-maprtech (des3-cbc-sha1)
      2 07/18/14 18:50:08 mapr/perfnode153.perf.lab@dev-maprtech (des-cbc-crc) 
    
      2 07/18/14 18:50:26 HTTP/perfnode153.perf.lab@dev-maprtech (aes256-cts-hmac-sha1-96)
      2 07/18/14 18:50:26 HTTP/perfnode153.perf.lab@dev-maprtech (arcfour-hmac)
      2 07/18/14 18:50:26 HTTP/perfnode153.perf.lab@dev-maprtech (des3-cbc-sha1)
      2 07/18/14 18:50:26 HTTP/perfnode153.perf.lab@dev-maprtech (des-cbc-crc) 
    
      6 07/18/14 18:50:56 mapr/my.cluster.com@dev-maprtech (aes256-cts-hmac-sha1-96)
      6 07/18/14 18:50:56 mapr/my.cluster.com@dev-maprtech (arcfour-hmac)
      6 07/18/14 18:50:56 mapr/my.cluster.com@dev-maprtech (des3-cbc-sha1)
      6 07/18/14 18:50:57 mapr/my.cluster.com@dev-maprtech (des-cbc-crc)

    In the example, the following principals are listed for the node perfnode153.perf.lab:

      • mapr/perfnode153.perf.lab@dev-maprtech (for authenticating to the httpFS service)

      • HTTP/perfnode153.perf.lab@dev-maprtech (for communicating securely over HTTP)

      • mapr/my.cluster.com (for authenticating to the CLDB service)

  7. Verify that context.xml.jpamLogin exists in this location:

    /opt/mapr/httpfs/httpfs-1.0/share/hadoop/httpfs/tomcat/webapps/webhdfs/META-INF/context.xml.jpamLogin

    This file may have been renamed to context.xml to configure PAM authentication for HttpFS. However, to configure Kerberos for HttpFS, rename the file back to context.xml.jpamLogin.

    mv /opt/mapr/httpfs/httpfs-1.0/share/hadoop/httpfs/tomcat/webapps/webhdfs/META-INF/context.xml /opt/mapr/httpfs/httpfs-1.0/share/hadoop/httpfs/tomcat/webapps/webhdfs/META-INF/context.xml.jpamLogin 

Create a Kerberos Principal and a keytab File for Hue and Enable Kerberos Tickets for MapReduce v. 1

Skip this task if you are not using MapReduce v. 1

Creating a Kerberos Principal

Set up the Kerberos principal and keytab file as shown in Create a Kerberos Principal and a keytab File for JobTracker

Using Kerberos Tickets for Hue

Using the keytab and principal you created in the previous step, complete the following steps:

  1. Extract the Kerberos ticket from the keytab file.
  2. Optionally, enable the Kerberos ticket renewer.

Extracting the Kerberos Ticket from the keytab File

To extract the ticket from the keytab file, run the following command (substitute your host and realm for perfnode181.perf.lab@dev-maprtech):

kinit -k -t /opt/mapr/conf/mapr.keytab -c /tmp/hue_krb5_ccache mapr/perfnode181.perf.lab@dev-maprtech

This command extracts the ticket from mapr.keytab and copies it to the path to the Kerberos ticket file used by Hue.

Enabling the Kerberos Ticket Renewer

Kerberos tickets have a default expiration time of 7 days. If you plan to use the Hue Kerberos ticket renewer in your cluster, enable this functionality by making changes to these two files:

  • kdc.conf (add the max_renewable_life parameter)
  • krb5.conf (add the renew_lifetime parameter)

Create a Kerberos Principal and a keytab File for Hue and Enable Kerberos Tickets for YARN

Skip this task if you are not using YARN.

Creating a Kerberos Principal

Set up the Kerberos principal and keytab file as shown in Create a Kerberos Principal and a keytab File for JobTracker

Using Kerberos Tickets for Hue

Using the keytab and principal you created in the previous step, complete the following steps:

  1. Extract the Kerberos ticket from the keytab file.
  2. Optionally, enable the Kerberos ticket renewer.

Extracting the Kerberos Ticket from the keytab File

To extract the ticket from the keytab file, run the following command (substitute your host and realm for perfnode181.perf.lab@dev-maprtech):

kinit -k -t /opt/mapr/conf/mapr.keytab -c /tmp/hue_krb5_ccache mapr/perfnode181.perf.lab@dev-maprtech

 This command extracts the ticket from mapr.keytab and copies it to the path to the Kerberos ticket file used by Hue.

Enabling the Kerberos Ticket Renewer

Kerberos tickets have a default expiration time of 7 days. If you plan to use the Hue Kerberos ticket renewer in your cluster, enable this functionality by making changes to these two files:

  • kdc.conf (add the max_renewable_life parameter)
  • krb5.conf (add the renew_lifetime parameter)

Create a Kerberos Principal and a keytab File for Sqoop2

Skip this task if you are not using Sqoop2.

Note the following items when you complete the configuration steps:

  • Replace <FQDN> with the FQDN of the server. To determine this value, run “hostname -f” in the command line.
  • Replace <REALM> with the realm name in krb5.conf file which is generated when you install the KDC server on the cluster.

Follow these steps to create Kerberos principals and keytab files for Sqoop2:

  1. Using the kadmin program, run the following commands to create principals for Sqoop 2:

    addprinc -randkey HTTP/<FQDN>@<REALM>
    addprinc -randkey mapr/<FQDN>@<REALM>

    Kerberos uses the principal HTTP/<FQDN>@<REALM> for communication between Sqoop2 client and Sqoop2 server. The principal mapr/<FQDN>@<REALM> is the Sqoop2 user that communicates between Sqoop2 server and MapR-FS.
     

  2. Using the kadmin program, run the following commands to create keytabs for the principals:

    xst -k /opt/mapr/conf/mapr.keytab HTTP/<FQDN>@<REALM> 
    xst -k /opt/mapr/conf/mapr.keytab mapr/<FQDN>@<REALM>


Comments:

Since all the services run as "mapr" user you just need a single keytab that includes "mapr" principal - no need to have this.keytab and that.keytab for different services like hive, flume and what not.

Posted by docs at Oct 28, 2015 10:04

Got confirmation from Yuliya that change needed was simply to replace every component-specific .keytab file name with "mapr.keytab" and made that change in this topic. Then, preparing to make the same change globally in the doc set, I noticed dependencies - some of the .keytab file names are defined in one topic and expected to already exist in other topics, and that multi-keytab organization may be intentional by design and important for function. 

So I confirmed w/Yuliya that the component-specific .keytab file names *do* work OK, and reverted those changes in this topic. I've emailed Padma & Bridget about this issue - we'll have to look into each component-specific .keytab file name further, to verify that it's OK to consolidate into mapr.keytab.

Posted by jwolley at Oct 29, 2015 16:50