MapR 5.0 Documentation : ulimit

ulimit is a command that sets limits on the user's access to system-wide resources. Specifically, it provides control over the resources available to the shell and to processes started by it.

The mapr-warden script uses the ulimit command to set the maximum number of file descriptors (nofile) and processes (nproc) to 64000. Higher values are unlikely to result in an appreciable performance gain. Lower values, such as the default value of 1024, are likely to result in task failures.

MapR's recommended value is set automatically every time warden is started.

Depending on your environment, you might want to set limits manually rather than relying on Warden to set them automatically using ulimit. The following examples show how to do this, using the recommended value of 64000.

Setting resource limits on Centos/Redhat
  1. Edit /etc/security/limits.conf and add the following line:
    <MAPR_USER> - nofile 64000
    
  2. Edit /etc/security/limits.d/90-nproc.conf and add the following line:
    <MAPR_USER> - nproc 64000
    
  3. Check that the /etc/pam.d/system-auth file contains the following settings:
       #%PAM-1.0
    
       auth            sufficient      pam_rootok.so
    
       # Uncomment the following line to implicitly trust users in the "wheel" group.
    
       #auth           sufficient      pam_wheel.so trust use_uid
    
       # Uncomment the following line to require a user to be in the "wheel" group.
    
       #auth           required        pam_wheel.so use_uid
    
       auth            include         system-auth
    
       account         sufficient      pam_succeed_if.so uid = 0 use_uid quiet
    
       account         include         system-auth
    
       password        include         system-auth
    
       session         include         system-auth
    
       session         required        pam_limits.so
    
       session         optional        pam_xauth.so
Setting resource limits on Ubuntu
  1. Edit /etc/security/limits.conf and add the following lines:
    <MAPR_USER> - nofile 64000
    <MAPR_USER> - nproc 64000
    
  2. Edit /etc/pam.d/su and uncomment the following line:
    session    required   pam_limits.so

Use ulimit to verify settings:

  1. Reboot the system.
  2. Run the following command as the MapR user (not root) at a command line:
    ulimit -n

The command should report 64000.