/ bin/bash:/ bin/java:没有这样的文件或目录

Question

我试图在Mac OS X EL Captain 10.11上使用Java 1.7 SDK和Hadoop2.7.1运行一个简单的wordcount MapReduce程序,我在容器日志"stderr"/ bin/bash:/ bin/java中收到以下错误消息:没有相应的文件和目录

申请日志 -

5/11/27 02:52:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/11/27 02:52:33 INFO client.RMProxy: Connecting to ResourceManager at /192.168.200.96:8032
15/11/27 02:52:34 INFO input.FileInputFormat: Total input paths to process : 0
15/11/27 02:52:34 INFO mapreduce.JobSubmitter: number of splits:0
15/11/27 02:52:34 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1448608832342_0003
15/11/27 02:52:34 INFO impl.YarnClientImpl: Submitted application application_1448608832342_0003
15/11/27 02:52:34 INFO mapreduce.Job: The url to track the job: http://mymac.local:8088/proxy/application_1448608832342_0003/
15/11/27 02:52:34 INFO mapreduce.Job: Running job: job_1448608832342_0003
15/11/27 02:52:38 INFO mapreduce.Job: Job job_1448608832342_0003 running in uber mode : false
15/11/27 02:52:38 INFO mapreduce.Job:  map 0% reduce 0%
15/11/27 02:52:38 INFO mapreduce.Job: Job job_1448608832342_0003 failed with state FAILED due to: Application application_1448608832342_0003 failed 2 times due to AM Container for appattempt_1448608832342_0003_000002 exited with  exitCode: 127
For more detailed output, check application tracking page:http://mymac.local:8088/cluster/app/application_1448608832342_0003Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1448608832342_0003_02_000001
Exit code: 127
Stack trace: ExitCodeException exitCode=127:
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
    at org.apache.hadoop.util.Shell.run(Shell.java:456)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 127
Failing this attempt. Failing the application.
15/11/27 02:52:38 INFO mapreduce.Job: Counters: 0

命令我正在运行

hadoop jar wordcount.jar org.myorg.WordCount /user/gangadharkadam/input/ /user/gangadharkadam/output/

我的ENV变量是 -

export   JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home
export HADOOP_HOME=/usr/local/hadoop/hadoop-2.7.1
export HADOOP_MAPRED_HOME=/usr/local/hadoop/hadoop-2.7.1
export HADOOP_COMMON_HOME=/usr/local/hadoop/hadoop-2.7.1
export HADOOP_HDFS_HOME=/usr/local/hadoop/hadoop-2.7.1
export YARN_HOME=/usr/local/hadoop/hadoop-2.7.1
export HADOOP_CONF_DIR=/usr/local/hadoop/hadoop-2.7.1/etc/hadoop
export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:HADOOP_HOME/sbin:$M2_HOME/bin:$ANT_HOME/bin:$IVY_HOME/bin:$FB_HOME/bin:$MYSQL_HOME/bin:$MYSQL_HOME/lib:$SQOOP_HOME/bin

问题似乎是因为YARN正在使用不同的JAVA可执行文件路径,而不是你的操作系统."stderr"中失败任务的本地日志显示 - /bin/bash:/ bin/java:没有这样的文件或目录

我试图创建一个从$ JAVA_HOM/bin/java到/ bin/java的软链接,但在El Captian OS X中却不允许创建软链接.新的OS X EL上尉好有一个无根的登录名和用户不能创造像/斌/某些受限制的文件夹东西.任何关于这个问题的工作都非常感谢.谢谢.

Answer 1

此答案适用于Hadoop 2.6.0及更早版本.禁用SIP并创建符号链接确实提供了一种解决方法.更好的解决方案是修复hadoop-config.sh,以便正确地获取JAVA_HOME

在HADOOP_HOME/libexec/hadoop-config.sh寻找下面的if条件# Attempt to set JAVA_HOME if it is not set

删除导出JAVA_HOME行中的额外括号,如下所示.改变这个

if [ -x /usr/libexec/java_home ]; then
    export JAVA_HOME=($(/usr/libexec/java_home))
else
    export JAVA_HOME=(/Library/Java/Home)
fi

至

if [ -x /usr/libexec/java_home ]; then
    // note that the extra parentheses are removed
    export JAVA_HOME=$(/usr/libexec/java_home)
else
    export JAVA_HOME=/Library/Java/Home
fi

完成此更改后重新启动纱线.

更详细的信息可以在https://issues.apache.org/jira/browse/HADOOP-8717找到,似乎Hadoop 3.0.0-alpha1是第一个带有修复的版本.