Hadoop YARN - 如何限制requestedMemory？

Question

试图从中运行PI示例hadoop-mapreduce-examples-2.2.0.jar,我得到以下异常:

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException): Invalid resource request, requested memory < 0, or requested memory > max configured, requestedMemory=1536, maxMemory=512

不知道1536来自哪里,但512是我在子任务中设置的最大堆大小mapred-site.xml:

<property>
  <name>mapreduce.map.memory.mb</name>
  <value>512</value>
</property>
<property>
  <name>mapreduce.map.java.opts</name>
  <value>-Xmx410m</value>
</property>
<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>512</value>
</property>
<property>
  <name>mapreduce.reduce.java.opts</name>
  <value>-Xmx410m</value>
</property>

确定map/reduce任务大小的正确方法是什么？

Answer 1

512是yarn.scheduler.maximum-allocation-mb in 的默认值yarn-site.xml,1536是yarn.app.mapreduce.am.resource.mb参数in 的默认值 mapred-site.xml.

确保allocation-mb> app.mapreduce.am.resource.mb,它会没问题.