mapred-site.xml

MapReduce is one type of application that can run on the Hadoop 2.x framework. MapReduce configuration options are stored in the /opt/mapr/hadoop/hadoop-2.x.x/etc/hadoop/mapred-site.xml file and are editable by the root user. This file contains configuration information that overrides the default values for MapReduce parameters. Overrides of the default values for core configuration properties are stored in the MapR Parameters file.

To override a default value for a property, specify the new value within the <configuration> tags, using the following format:

<property>
 <name> </name>
 <value> </value>
 <description> </description>
</property>

Configurations for MapReduce Applications

Parameter Value Description
mapreduce.framework.name yarn Execution framework set to Hadoop YARN.
mapreduce.map.memory.mb 1024 Larger resource limit for maps.
mapreduce.map.java.opts -Xmx1024M Larger heap-size for child jvms of maps.
mapreduce.reduce.memory.mb 3072 Larger resource limit for reduces.
mapreduce.reduce.java.opts -Xmx2560M Larger heap-size for child jvms of reduces.
mapreduce.task.io.sort.mb 512 Higher memory limit while sorting data for efficiency.
mapreduce.task.io.sort.factor 100 More streams merged at once while sorting files.
mapreduce.reduce.shuffle.parallelcopies 50 Higher number of parallel copies run by reduces to fetch outputs from very large number of maps.

Configurations for MapReduce JobHistory Server

Parameter Value Description
mapreduce.jobhistory.address MapReduce JobHistory Server host:port Default port is 10020.
mapreduce.jobhistory.webapp.address MapReduce JobHistory Server Web UI host:port Default port is 19888.
mapreduce.jobhistory.intermediate-done-dir /mr-history/tmp Directory where history files are written by MapReduce jobs.
mapreduce.jobhistory.done-dir /mr-history/done Directory where history files are managed by the MapReduce JobHistory Server.

Sample Hadoop 2.x mapred-site.xml File

The following mapred-site.xml file defines values for two job history parameters.

<configuration>
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>__HS_IP__:10020</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>__HS_IP__:19888</value>
  </property>
</configuration>