Hadoop MapReduce 作业卡住,因为 auxService:mapreduce_shuffle 不存在

JCh*_*hao 2 hadoop mapreduce

我已经用相同的问题检查了多个帖子,解决方案总是将以下内容添加到 yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->
    <property>
        <name>yarm.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>
Run Code Online (Sandbox Code Playgroud)

我在配置中介绍了这两个属性并重新启动了纱线。问题仍然存在。

错误是:

17/02/15 15:43:34 INFO mapreduce.Job: Task Id : attempt_1487202110321_0001_m_000000_2, Status : FAILED
Container launch failed for container_1487202110321_0001_01_000007 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
    at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Run Code Online (Sandbox Code Playgroud)

我希望打错字,但似乎找不到。试过直接将stackoverflow上的内容复制到xml文件中,还是不行。

我还能尝试什么?

编辑:

由于错误说 aux_service 应该是auxService,我根据修改了 yarn-site.xml,将所有更改aux-serviceauxService,但它仍然无法正常工作。

编辑2:

如果有人感兴趣,我称这个命令

hadoop jar hadoop-streaming-2.7.1.jar \
       -input /user/myfolder/input1/* \
       -output /user/myfolder/output1 \
       -mapper <path>/<to>/<mapper>/mapper.py \
       -reducer <path>/<to>/<reducer>/reducer.py
Run Code Online (Sandbox Code Playgroud)

虽然我已经在 /usr/local/cellar/hadoop/2.7.1/libexec/share/hadoop/tools/lib/

编辑 3:

我是个笨蛋。校对脚本的家伙!

fra*_*ijo 5

更新yarn-site.xmlas 中的属性名称yarn.nodemanager.aux-services

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
Run Code Online (Sandbox Code Playgroud)