Remove and Replace Disks using the MapR Command-Line Interface

  1. On the node with failed disks, determine which disk to replace by examining Disk entries in the /opt/mapr/logs/faileddisk.log file. In the sample log file above, the failed disk is /dev/sdd.
  2. Use the disk remove command to remove the disk.
    Warning: Removing a disk in the storage pool that contains Container ID 1 will stop your cluster. Container ID 1 contains CLDB data for the master CLDB. From the command-line, run the maprcli disk remove command without the -force 1 option first and examine the warning messages to make sure you aren't removing the disk with Container ID 1. To safely remove such a disk, perform a CLDB Failover to make one of the other CLDB nodes the primary CLDB, then remove the disk as normal.
    Run the following command, substituting the hostname or IP address for <host> and a list of failed disks for <disks>:
    [user@host] maprcli disk remove -host <host> -disks <disks>

    The disk removal process can take several minutes to complete.

  3. Examine the screen output in response to this command:
    # maprcli disk remove -host `hostname -f` -disks /dev/sdd
    message   host   disk
    removed.  host1  /dev/sdd
    removed.  host1  /dev/sde
    removed.  host1  /dev/sdf
    Note the additional disks removed when /dev/sdd is removed. The disks /dev/sde and /dev/sdf are part of the same storage pool and therefore removed along with the failed disk.
  4. Confirm that removed disks do not appear in the disktab file.
  5. Replace the failed disks on the node or nodes, following correct procedures for your hardware.
  6. Remove the failed disk log file from the /opt/mapr/logs directory. These log files are typically named in the pattern
  7. Use the disk add command to add the replacement disk (or disks) along with other disks from the same storage pool or pools.
    Note: This step reformats the disks. Any data on these disks will be lost.
    For example, to add the removed disks shown in step 3, the command is:
    # maprcli disk add -host `hostname -f` -disks /dev/sdd,/dev/sde,/dev/sdf
    message  host               disk
    added.   host1  /dev/sdd
    added.   host1  /dev/sde
    added.   host1  /dev/sdf

    Note that when the failed disk has been replaced, it should be added back to MapR-FS along with the other disks from the same storage pool that were previously removed. If you add only the replacement disk to MapR-FS, this will result in a non-optimal storage pool layout which can lead to degraded performance.

    Once the disks are added to MapR-FS, the cluster allocates properly-sized storage pools automatically. For example, if you add ten disks, MapR allocates two storage pools of three disks each and two storage pools of two disks each.