MapR 5.0 Documentation : Rolling MapR Core Upgrade Using a Script

Overview 

This section explains how to run scripted rolling upgrades. The rolling upgrade is supported for upgrades from 4.0.x or 4.1 to 5.0  If you are upgrading from an earlier version, see Offline MapR Core Upgrade Using Manual Steps. The rolling upgrade process is designed to upgrade the core packages on all nodes on the cluster in a single operation. Packages are propagated and installed across the nodes while the cluster remains online and operational. (Do not try to use this type of upgrade to upgrade specific nodes only.) 

You can monitor the progress of the whole upgrade process by running the MCS on any node that is not currently in maintenance mode (being upgraded).

The upgrade utility is extracted from a setup file and requires a tar file that contains all of the packages. You do not have to have Internet connectivity to run the upgrade if the setup file and the tar file are already downloaded. You do not need to configure a repository for these files.

SSH Mode

Rolling upgrades run only in SSH mode, using either key-based (passwordless) authorization or password-based access for remote access to all of the nodes. The upgrade utility copies the archive file to the other nodes on the cluster that you are upgrading and runs similar commands on each node, using a root or sudo user account specified in the upgrade command.

See Preparing Each Node for more information about setting up passwordless SSH.

Before You Upgrade

You can run the same rolling upgrade process on secure and non-secure clusters; however, on a secure cluster, the MapR Admin user must have a security ticket created before running the upgrade. Otherwise the upgrade commands will not run.

The cluster continues to run at the highest capacity possible during the upgrade process.

If your cluster uses RM HA and the standby RM is on a CLDB node, the active RM node will be upgraded before the standby RM node. To reduce the impact on any jobs that are running during the upgrade, set the following parameters to a number that is greater than the number of ResourceManagers in the cluster:

  • yarn.resourcemanager.am.max-attempts
  • mapreduce.am.max-attempts

See ResourceManager Configuration Properties for details about these parameters.

You do not need to shut down any services before running a rolling upgrade. As nodes are upgraded, they go into maintenance mode and warden is shut down, which shuts down other services, including any ecosystem components. Rolling upgrades do not upgrade packages for Hadoop ecosystem components. Upgrading the ecosystem is a separate upgrade step. Only MapR core packages are upgraded; however, when an upgraded node comes back online, warden restarts ecosystem components that are compatible with the new release. Incompatible ecosystem components will fail to come up until they are upgraded separately.

See Hadoop ecosystem components for details about upgrading those components. See Packages and Dependencies for MapR Software for a list of the MapR packages and Hadoop ecosystem packages.

Rolling Upgrade Procedure

To run a rolling upgrade from Version 4.0.x to Version 5.0 or from Version 4.1.0 to Version 5.0:

  1. Download the mapr-setup package and the archive file (the tar file that contains all of the core packages). Download the archive file to a server on the cluster that you want to upgrade. For example, download the files to a upgrade directory:

    [root@centos24 upgrade]# wget http://package.mapr.com/releases/v5.0.0/redhat/mapr-setup
    ...
    Saving to: "mapr-setup"
    [root@centos24 upgrade]# wget http://package.mapr.com/releases/v5.0.0/redhat/mapr-v5.0.0GA.rpm.tgz
    ...
    Saving to: "mapr-v5.0.0GA.rpm.tgz"


  2. As root (or by using sudo), run the following command to extract the mapr-setup file, which includes the upgrade utility:

    By default, the downloaded mapr-setup file does not have execute permissions. Permissions need to be provided (for example, chmod +x mapr-setup) before running the script.


    [root@centos24 upgrade]# sudo bash ./mapr-setup
    ======================================================
    Self Extracting Install/Patch/Upgrade Utility for MapR
    ======================================================
    
    Extracting installer.......
    Copying setup files to "/opt/mapr-installer"......
    Installed to "/opt/mapr-installer"
    ====================================
    
    To upgrade MapR packages, run "/opt/mapr-installer/bin/upgrade" as the root user or with sudo.
    To apply a patch, run "/opt/mapr-installer/bin/patch" as the root user or with sudo.
    To install MapR packages, run "/opt/mapr-installer/bin/install" as the root user or with sudo. (Note: This option is deprecated. Use the MapR Installer
    instead)



  3. Run the upgrade utility from the extracted /opt/mapr-installer/bin/upgrade directory and provide the path to the archive file. For example:

    [root@centos24 upgrade]# /opt/mapr-installer/bin/upgrade --archive-file /root/upgrade/mapr-v5.0.0GA.rpm.tgz
     
                Current Information (Please verify if correct)
                ==============================================
                Upgrade Settings
                ================
                    MapR Administration User: "mapr"
                    Archive File Location: "/root/upgrade/mapr-v5.0.0GA.rpm.tgz"
                    Upgrade Version: "5.0.0"
                
    (c)ontinue with install, (m)odify options, or save current configuration and (a)bort? (c/m/a) [c]:

    This example presents a simple case where the upgrade is run as root, and the root username and password are requested interactively. Examples with other credentials are shown later in this section.

    The upgrade utility has several options; see Upgrade Command Options. When you run the upgrade, the output describes all of the steps in the upgrade process, and it identifies the groups of nodes where the upgrade will run before it upgrades them. See Rolling MapR Core Upgrade Using Manual Steps for information about the order in which groups of nodes and packages are upgraded. 

    The output shown here captures the upgrade operations on the first node in the cluster to be upgraded: centos24.

    You can modify the options (MapR user account and archive file location), stop the upgrade, or continue.

     

  4. Enter c to continue the upgrade process, then enter the SSH username and password:

    SSH Username: root
    SSH password:      

    Alternatively, you can specify username and password as part of the upgrade command. The upgrade process begins on the first node:

    * 15:22:49 Checking Connection
    * - 15:22:53 Group by OS Family
    Total Nodes in Zookeeper Servers: 0
    Total Nodes in File Servers: 0
    Total Nodes in Task Tracker Servers: 0
    Total Nodes in Node Manager Servers: 0
    Total Nodes in Job Tracker Servers: 0
    Total Nodes in Resource Manager Server: 0
    Total Nodes in HBase Region Servers: 0
    Total Nodes in HBase Master Servers: 0
    Total Nodes in CLDB Servers: 1
    Now running on CLDB Servers: centos24
    * 15:22:53 Initiating Upgrade
    * - 15:22:55 Determining MapR Version
    * - 15:22:58 Group by OS Family
    * 15:22:58 Commencing Upgrade on RedHat/CentOS
    * - 15:23:00 Determining if Node has a Tasktracker Role
    * - 15:23:02 Determining if Node has a CLDB Role
    * - 15:23:06 Determining if Node has Warden Present
    * - 15:23:11 Determining if Node has Zookeeper Present
    * - 15:23:13 Cleaning up Existing Package Archive Directory
    * - 15:23:15 Creating Archive Directory, If needed
    * - 15:23:18 Copying and Extracting MapR Package Archive File
    * - 15:25:21 Determining if a Patch has been applied before
    * - 15:25:23 Determining Hadoop Mode (Yarn vs Classic)
    * - 15:25:27 Blacklisting TaskTracker. New Jobs will not be assigned to the node
    * - - 	 [Skipping]
    * - 15:25:27 Waiting for the Node to be available for Upgrade
    * - - 	 [Skipping]
    * - 15:25:27 Setting Node in Maintenance
    * - - 	 [Skipping]
    * - 15:25:27 Notifying CLDB about Begining Upgrade Process on the Node
    * - 15:25:32 Stopping all MapR Services
    * - 15:26:07 Removing Patch, if Exists
    * - - 	 [Skipping]
    * - 15:26:07 Upgrading Packages
    * - 15:28:51 Configuring Node
    * - 15:28:59 Starting Zookeeper Service
    * - 15:29:02 Starting all MapR Services
    * - 15:29:11 Waiting for CLDB to come online
    * - 15:34:50 Bringing Node back from Maintenance
    * - - 	 [Skipping]
    * - 15:34:50 Instructing CLDB about Upgraded Version
    * - 15:34:57 Notifying CLDB about Completion of Upgrade Process on the Node
    * - 15:35:02 Waiting for Containers to Sync
    * - 15:37:05 Removing Node from Tasktracker Blacklist
    * - - 	 [Skipping]
    MapR Rolling Upgrade Process has been completed successfully on centos24.
    ...

The upgrade process continues on the next node. When the rolling upgrade process is complete on all nodes, configure the new version as usual.

Upgrade Command Options

Run upgrade -h to display the usage for the upgrade utility:

# /opt/mapr-installer/bin/upgrade -h
 usage: upgrade.py [-h] [-s] [-U SUDO_USER] [-u REMOTE_USER]
                  [--private-key PRIVATE_KEY_FILE] [-k] [-K]
                  [--password REMOTE_PASS] [--sudo-password SUDO_PASS]
                  [--wait-for-running-tasks-to-complete]
                  [--mapr-admin ADMIN_USER] [--archive-file ARCHIVE_FILE]
                  [--upgrade-to UPGRADE_TO] [--ssh-port SSH_PORT]

The utility supports the following options:

OptionDescription
-h or --helpDisplay the syntax and usage for the utility.
-s or --sudoRun operations as sudo, without submitting a password. This option means that after initial login, the upgrade process should use sudo to run commands. The -U option identifies the specific sudo user for the upgrade script to run as.
-U SUDO_USERRun the upgrade as a user with sudo privileges. The default account is sudo.
--sudo-password SUDO_PASS
Specify the remote sudo password. All nodes must accept the same username and password.
-u REMOTE_USERConnect with a remote user account that exists on all of the nodes.
--password REMOTE_PASS
Specify the remote SSH password. All nodes must accept the same username and password. Specifying a remote password and specifying a private key file are mutually exclusive options. Do not attempt to run the upgrade command with both sets of credentials.
--private-key PRIVATE_KEY_FILE 
Use key-based SSH on clusters that require or are configured for key-based authentication across nodes, such as cloud-based deployments. Specifying a remote password and specifying a private key file are mutually exclusive options. Do not attempt to run the upgrade command with both sets of credentials.
-k or --ask-passPrompt for the SSH password. All nodes must accept the same username and password.
-K or --ask-sudo-passPrompt for the sudo password. All nodes must accept the same username and password.
--wait-for-running-tasks-to-complete
Delay the upgrade until all running MRv1 jobs are complete.
--mapr-admin ADMIN_USER
Specify the MapR Admin user account on the cluster if it is not mapr (the default).
--archive-file <path-to-file>

Full path to the tar file that you downloaded, which contains the upgrade packages. For example:

/root/upgrade/mapr-v4.0.2GA.rpm.tgz

You must specify the full path to this file.

--upgrade-to

Version of MapR software that you are upgrading to (only 4.0.2 and 4.1.0 are accepted).

The --upgrade-to value and the version of the packages in the downloaded archive file must match.

--ssh-portRemote SSH port. If your SSH ports are set to non-22 values for security reasons, specify the SSH port. All nodes in the cluster must be set to the same SSH port number for upgrade to succeed.

Credential Input

The rolling upgrade script offers two choices for SSH user authorization: private-key-based (passwordless) authorization or password-based access.

The upgrade process supports various combinations of login and remote user credentials and the ability to provide these credentials either on the command line or interactively (via prompts). This section describes a few examples.

Upgrade as the root user

This is a simple example of running the upgrade as root. The root username and password are specified in the command line:

# /opt/mapr-installer/bin/upgrade -u root --password ******** --archive-file ....

Upgrade as a non-root user

This example specifies the remote user account (-u) and the root user (-U) on the command line. This example assumes that a password is required for the sudo user.

$ /opt/mapr-installer/bin/upgrade -u bobr_ubuntu -U root -s --sudo-password ******** --archive file ....

Upgrade using a private key

This example uses a private key instead of a password to access the nodes.

$ /opt/mapr-installer/bin/upgrade -u bobr_amazon -U root -s --sudo-password ******** --private-key .... --archive file ....

Rolling Upgrade Logs

 The rolling upgrade process logs its output to the following file: 

/opt/mapr-installer/var/mapr-upgrade.log

The log includes the regular output of the upgrade utility as well as other detailed information that is provided for support use. 

After you Upgrade 

After you complete the scripted rolling upgrade, complete the following steps:

  1. Update the warden configuration files, which are located in /opt/mapr/conf/ and /opt/mapr/conf/conf.d.

    • Manually merge new configuration settings from the /opt/mapr/conf.new/warden.conf file into the /opt/mapr/conf/warden.conf file.

    • Manually merge new configuration settings from the files in the /opt/mapr/conf/conf.d.new/ directory to the files in the /opt/mapr/conf/conf.d/ directory.
       

  2. For upgrades from 4.0.1, any script that points to files within the /opt/mapr/hadoop-2.4.1 directory must be updated to point to the new hadoop 2.x directory.

 

When you upgrade hadoop common, a new directory is created for the new hadoop common version and the configuration files in the existing /opt/mapr/hadoop/hadoop-2.x.x directory are automatically copied into the active directory associated with the new hadoop 2.x.x directory. For example, when you upgrade from 4.0.1 to 5.0, configuration files that were in the hadoop-2.4.1 directory are copied into the hadoop-2.7.1 directory.

 

 


 --> Back to Upgrading Without the MapR Installer