Release Notes : Unable to Establish a Secure Connection

Recent versions of Safari and Chrome web browsers have removed support for older certificate cipher algorithms, including those used by some versions of MapR. Because of this, users of these new browser versions may lose the ability to log into MapR Control System (MCS).

A fix for this issue is available in MapR Versions 4.0.2 and later. Existing clusters can be patched to workaround this issue. Information and installation instructions for this patch are found later in this document. For additional fixes that you may also want to apply at this time, see Web Browser Security Issues.

This document contains the following sections: 

Affected Versions

To determine whether you will be affected, your MapR version must be in the range listed in the MapR section below, and you must be accessing MCS using a browser version listed in either the Safari or Chrome sections.

MapR

Versions 3.1, 3.1.1, 4.0.0, and 4.0.1.
Safari
Versions 7.0 and higher.
Chrome
Versions 39.0 and higher.

Symptoms

The screenshots and error messages below show what a user might see if they encounter the issue:
ChromeSafari

Screenshot:

Screenshot:
Error Message:
SSL connection error. Unable to make a secure connection to the server. This maybe a problem with the server, or it may be requiring a client authentication certificate that you don't have. Error code: ERR_SSL_PROTOCOL_ERROR
Error Message:
Safari can't open the page <URL> because Safari can't establish a secure connection to the server <server name>.

Patch Information

The steps to implement the fix for a secure cluster (cluster with wire-level security) differ from the steps to implement the fix on a non-secure cluster.  However, in both cases, you will use the fixssl script to generate new versions of the ssl_keystore and ssl_truststore.

While you are implementing the fix on a non-secure cluster, the webserver will experience a brief downtime. The impact on a secure cluster will be greater, as more services will need to be restarted for the patch to take effect.

How to Install the Patch

Complete the steps to implement the fix for your cluster type. You have a secure cluster if you use wire-level security to encrypt data transmission between the nodes in your cluster.

Patching a Non-secure cluster

  1. Determine which nodes in the cluster run the webserver role.
    For example:

    $maprcli node list -columns configuredservice -filter '[configuredservice==webserver]'
    hostname  configuredservice                                    ip
    centos21  webserver,nodemanager,cldb,fileserver,resourcemanager,hoststats  10.10.82.21 

     

  2. Perform the following steps on each webserver node: 

    1. Download the script from the following location: http://package.mapr.com/scripts/mcs/
      For example:

      wget http://package.mapr.com/scripts/mcs/fixssl
    2. Run the following command to update the permissions on the file: 
      chmod 755 fixssl
    3. Run the following command to run the script: 
      sudo ./fixssl
      Once you run the script, the following is displayed:

      “Webserver restarted. Issue should be resolved” 

The fix is complete. No further action is required. You can now access the MCS and other web interfaces, such as the JobTracker UI and the ResourceManager UI  

Patching a Secure Cluster

  1. Perform the following steps on any cluster node:
    1. Download the script from the following location: http://package.mapr.com/scripts/mcs/
      For example:

      wget http://package.mapr.com/scripts/mcs/fixssl
    2. Run the following command to update the permissions on the file: 
      chmod 755 fixssl
    3. Run the following command to run the script: 
      sudo ./fixssl
      Once you run the script, the following is displayed: 

      Creating 10 year self signed certificate with subjectDN='CN=*.us-west-2.compute.internal'
       
      Certificate stored in file </tmp/tmpfile-mapcert.3743>
       
      Certificate was added to keystore
       
      *****************************************************************************************
       
      * In order for your cluster to work, please copy the following files in /opt/mapr/conf  *
       
      * to all the nodes in the cluster, to the same directory: ssl_keystore ssl_truststore   *
       
      * After copying the files to the other nodes, please restart CLDB, Webserver, and any   *
       
      * other service that utilizes https (Jobtracker, tasktracker)                           *
       
      * (See doc for more details if you do not wish to have downtime in your cluster)        *
       
      *****************************************************************************************

       

  2. On each node in the cluster, back up existing certificates and copy the certificates to all other nodes in the cluster.

    For example:

    $ maprcli node list -columns ip
    hostname ip 
    ip-172-31-18-196.us-west-2.compute.internal 172.31.18.196 
    ip-172-31-18-197.us-west-2.compute.internal 172.31.18.197 
    ip-172-31-18-198.us-west-2.compute.internal 172.31.18.198 
    ip-172-31-18-199.us-west-2.compute.internal 172.31.18.199 
    ip-172-31-18-200.us-west-2.compute.internal 172.31.18.200
    
    $ ssh 172.31.18.200 "mv /opt/mapr/conf/ssl_keystore /opt/mapr/conf/ssl_keystoreold"
    
    $ ssh 172.31.18.200 "mv /opt/mapr/conf/ssl_truststore /opt/mapr/conf/ssl_truststoreeold"
    
    $ scp /opt/mapr/conf/ssl_keystore /opt/mapr/conf/ssl_truststore mapr@172.31.18.200:/opt/mapr/conf
  3. Restart the CLDB slave services.  To do this, first you determine which cluster nodes are running the CLDB service and then determine which node is running the master CLDB.  The slaves are the non-master CLDB nodes.
    For example:

    $ maprcli node list -columns configuredservice -filter '[configuredservice==cldb]'
    hostname                                     configuredservice                                   ip             
    ip-172-31-18-198.us-west-2.compute.internal  webserver,cldb,fileserver,nfs,hoststats,jobtracker  172.31.18.198  
    ip-172-31-18-199.us-west-2.compute.internal  webserver,cldb,fileserver,nfs,hoststats,jobtracker  172.31.18.199  
    ip-172-31-18-200.us-west-2.compute.internal  webserver,cldb,fileserver,nfs,hoststats,jobtracker  172.31.18.200  
    
    $ maprcli node cldbmaster
    cldbmaster                                                                           
    ServerID: 8868598593037642491 HostName: ip-172-31-18-199.us-west-2.compute.internal  
     
    $maprcli node services -cldb restart -nodes 172.31.18.198 172.31.18.200
  4. Restart half of the TaskTracker and Nodemanager services.  

    1. List all TaskTracker or NodeManager Hosts.
      For example:
       

      $ maprcli node list -columns configuredservice -filter '[configuredservice==tasktracker]or[configuredservice==nodemanager]'
      hostname                                     configuredservice                     ip             
      ip-172-31-18-196.us-west-2.compute.internal  fileserver,tasktracker,nfs,hoststats  172.31.18.196  
      ip-172-31-18-197.us-west-2.compute.internal  fileserver,tasktracker,nfs,hoststats  172.31.18.197
    2. Restart TaskTracker and NodeManager services on half of the nodes that run those services.  
      For example, the following command will restart both TaskTracker and NodeManager services on all nodes specified.  If either service is not configured on that node, it will ignore it.

      $ maprcli node services -multi '[{ "name": "tasktracker", "action": "restart"}, { "name": "nodemanager", "action": "restart"}]' -nodes 172.31.18.196
      ERROR (10002) -  Service: nodemanager is not configured on node: ip-172-31-18-196.us-west-2.compute.internal
  5. Restart JobTracker and ResourceManager services.

    1. List all nodes running JobTracker or ResourceManager.
      For example:

      $ maprcli node list -columns configuredservice -filter '[configuredservice==jobtracker]or[configuredservice==resourcemanager]'
      hostname                                     configuredservice                                   ip             
      ip-172-31-18-198.us-west-2.compute.internal  webserver,cldb,fileserver,nfs,hoststats,jobtracker  172.31.18.198  
      ip-172-31-18-199.us-west-2.compute.internal  webserver,cldb,fileserver,nfs,hoststats,jobtracker  172.31.18.199  
      ip-172-31-18-200.us-west-2.compute.internal  webserver,cldb,fileserver,nfs,hoststats,jobtracker  172.31.18.200
    2. Restart JobTracker and ResourceManager services.  
      For example, the following command will restart both JobTracker and ResourceManager services on the specified nodes.  If either service is not configured on that node, it will ignore it.

      $ maprcli node services -multi '[{ "name": "jobtracker", "action": "restart"}, { "name": "resourcemanager", "action": "restart"}]' -nodes 172.31.18.198 172.31.18.199 172.31.18.200
      ERROR (10002) -  Service: resourcemanager is not configured on node: ip-172-31-18-199.us-west-2.compute.internal
      ERROR (10002) -  Service: resourcemanager is not configured on node: ip-172-31-18-200.us-west-2.compute.internal
      ERROR (10002) -  Service: resourcemanager is not configured on node: ip-172-31-18-198.us-west-2.compute.internal 
  6. Restart remaining TaskTracker and NodeManager services.
    For example, the following command will restart both TaskTracker and NodeManager services on the specifed nodes.  If either service is not configured on that node, it will ignore it.

    $ maprcli node services -multi '[{ "name": "tasktracker", "action": "restart"}, { "name": "nodemanager", "action": "restart"}]' -nodes 172.31.18.197
    ERROR (10002) -  Service: nodemanager is not configured on node: ip-172-31-18-197.us-west-2.compute.internal
  7. Restart additional secure services (Oozie, HistoryServer, Webserver, HiveServer2, Hue).
    For example, the following command can be run with the IPs or hostnames of all nodes in the cluster, as it will only restart the services that it finds:

    $ maprcli node services -multi '[{ "name": "hue", "action": "restart"}, { "name": "historyserver", "action": "restart"}, { "name": "webserver", "action": "restart"}, { "name": "oozie", "action": "restart"}, { "name": "hs2", "action": "restart"}]' -nodes 172.31.18.198 172.31.18.199 172.31.18.200 172.31.18.196 172.31.18.197 
  8. Restart CLDB master service.
    For example:

    $ maprcli node cldbmaster
    cldbmaster                                                                           
    ServerID: 8868598593037642491 HostName: ip-172-31-18-199.us-west-2.compute.internal  
    
    $ maprcli node services -cldb restart -nodes 172.31.18.199

The fix is complete. No further action is required. You can now access the MCS and other web interfaces, such as the JobTracker UI and the ResourceManager UI. 

How the Script Works

The script behaves differently on secure and non-secure clusters.

Behavior in a Non-secure Cluster

The fixssl script performs the following steps on a node in a non-secure cluster:

  1. Updates manageSSLKeys.sh to use the new certificate cipher algorithm.
  2. Backs up the existing certificates so that new versions can be generated with the new cipher algorithm: 
    /opt/mapr/conf/ssl_keystore is renamed to /opt/mapr/conf/ssl_keystore_old
    /opt/mapr/comf/ssl_truststore is renamed to /opt/mapr/comf/ssl_truststore_old
  3. Runs /opt/mapr/server/configure.sh -R to generate new versions of the keystore and truststore files
  4. Restarts the webserver.

Behavior in a Secure Cluster

The fixssl script performs the following steps on a node in a secure cluster: 

  1. Updates manageSSLKeys.sh to use the new certificate cipher algorithm.
  2. Backs up the existing certificates so that new versions can be generated with the new cipher algorithm: 
    /opt/mapr/conf/ssl_keystore is renamed to /opt/mapr/conf/ssl_keystore_old
    /opt/mapr/comf/ssl_truststore is renamed to /opt/mapr/comf/ssl_truststore_old
  3. Runs the following command to generate new versions of the keystore and truststore files:
    /opt/mapr/manageSSLKey.sh  create -N <clustername> -ug <maprusername>:<maprgroup>
    The cluster name is retrieved from /opt/mapr/conf/mapr-clusters.conf.
    The mapr user and mapr group is retrieved from /opt/mapr/conf/daemon.conf.


 

Attachments:

SSL_safarieError.png (image/png)
SSL_chromeerror.png (image/png)