This document describes how to use label-based scheduling for MapReduce (MRv1) jobs.

You can run jobs on specified nodes or groups of nodes using label-based scheduling – assigning labels to various groups of nodes and then using the labels to specify where jobs run. This feature is used in conjunction with the Fair Scheduler. The labels are mapped to nodes using the node labels file, a file stored in MapR-FS. When you run jobs, you can place them on specified nodes individually or at the queue level.

When using label-based job placement, you cannot use the Fair Scheduler with preemption or task prefetch. For details on prefetch, see parameter mapreduce.tasktracker.prefetch.maptasks on page mapred-site.xml (MapReduce v1).

The Node Labels File

The node labels file defines labels for cluster nodes, to identify them for the purpose of specifying where to run jobs. Each line in the node label file consists of an identifier that specifies one or more nodes, along with one or more labels to apply to the specified nodes, separated by whitespace.

<identifier> <labels>

Tabs cannot be used for whitespace. If tabs are used, the node label list will be empty.

  • The labels are a comma-delimited list of labels to apply to the nodes matched by the identifier. Labels that contain whitespace should be enclosed in single or double quotation marks.

Sample node label file

The following example shows both glob identifiers and regular expression identifiers.

/perfnode200.*/ big, "Production Machines"
/perfnode203.*/ big, 'Development Machines'
perfnode15* good
perfnode201* slow
perfnode204* good, big

If glob identifiers overlap, so that patterns from multiple lines match a particular node, only the last line in the file that matches is applied.

The file path and name are specified in the parameter mapreduce.jobtracker.node.labels.file in mapred-site.xml (MapReduce v1). If no file is specified, jobs can run on any nodes in the cluster.

You can use hadoop job -showlabels to view the labels of all active nodes.

The parameter mapreduce.jobtracker.node.labels.monitor.interval in mapred-site.xml (MapReduce v1) determines how often the JobTracker should poll the node label file for changes (the default is two minutes). You can also use hadoop job -refreshlabels to manually tell the JobTracker to re-load the node label file.

Placing Jobs

The placement of jobs on nodes or groups of nodes is controlled by labels applied to the queues to which jobs are submitted, or to the jobs themselves. A queue or job label is an expression that uses logical operators OR, AND, NOT to specify the nodes described in the node labels file:

  • queue label specifies the node or nodes that will run all jobs submitted to a queue.
  • job label specifies the node or nodes that will run a particular job

Job and queue labels specify nodes using labels corresponding to the node labels file, with the operators || (OR) and && (AND).

Examples:

  • "Production Machines" || good — selects nodes that are in either the "Production Machines" group or the good group.
  • 'Development Machines' && good — selects nodes only if they are in both the 'Development Machines' group and the good group.

If a job is submitted with a label that does not include any nodes, the job will remain in the PREP state until nodes exist that meet the criteria (or until the job is killed). For example, in the node labels file above, no nodes have been assigned to both the 'Development Machines' group and the good group. If a job is submitted with the label 'Development Machines' && good, it cannot execute unless there are nodes that exist in both groups. If the node labels file is edited so that the 'Development Machines' group and the good group have nodes in common, the job will execute as soon as the JobTracker becomes aware — either after the mapreduce.jobtracker.node.labels.monitor.interval or when you execute the 
hadoop job -refreshlabels command.

Queue Labels

Queue Labels are defined using the mapred.queue.<queue-name>.label parameter in mapred-site.xml (MapReduce v1). The corresponding parameter mapred.queue.<queue-name>.label.policy specifies one of the following policies that determine the precedence of queue labels and job labels:

  • PREFER_QUEUE — always use label set on queue
  • PREFER_JOB — always use label set on job
  • AND (default) — job label AND queue label
  • OR — job label OR queue label

You can set a default queue policy using mapred.queue.default.label.

Example: Setting a policy on the default queue

The following excerpt from mapred-site.xml (MapReduce v1) shows the PREFER_QUEUE policy set on the default queue.

<property>
<name>mapred.queue.default.label</name>
<value>big || good</value>
</property>
<property>
<name>mapred.queue.default.label.policy</name>
<value>PREFER_QUEUE</value>
</property>

Job Labels

There are three ways to set job labels:

Examples

The following examples show the job placement policy behavior in certain scenarios, using the sample node labels file above.

Job Label

Queue Label

Queue Policy

Outcome

good

big

PREFER_JOB

The job runs on nodes labeled "good" (hostnames match perfnode15* or perfnode204*)

good

big

PREFER_QUEUE

The job runs on nodes labeled "big" (hostnames match /perfnode200./ or /perfnode203./)

good

big

AND

The job runs on nodes only if they are labeled both "good" and "big" (hostnames match perfnode204*)

good

big

OR

The job runs on nodes if they are labeled either "good" or "big" (hostnames match /perfnode200./, /perfnode203./, perfnode15*, or perfnode204*)