相关疑难解决方法(0)

我试图将简单的字数作为MapReduce作业运行.在本地运行时一切正常(所有工作都在名称节点上完成).但是,当我尝试使用YARN(添加mapreduce.framework.name= yarnmapred-site.conf)在集群上运行它时,作业挂起.

我在这里遇到了类似的问题: MapReduce作业陷入了Accepted状态

工作输出:

*** START ***
15/12/25 17:52:50 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
15/12/25 17:52:51 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/12/25 17:52:51 INFO input.FileInputFormat: Total input paths to process : 5
15/12/25 17:52:52 INFO mapreduce.JobSubmitter: number of splits:5
15/12/25 17:52:52 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1451083949804_0001
15/12/25 17:52:53 INFO impl.YarnClientImpl: Submitted application application_1451083949804_0001
15/12/25 17:52:53 INFO …

提前感谢您的帮助

我正在运行以下版本:

Hadoop 2.2 zookeeper 3.4.5 Hbase 0.96 Hive 0.12

当我转到http://:50070时,我能够正确地看到2个节点正在运行.

问题是,当我转到http://:8088时,它显示0个节点正在运行.

据我所知:8088反映了资源管理器并显示了运行的nodemanagers的数量.守护进程全部启动,但似乎节点管理器没有连接到资源管理器.

这是日志文件:

2013-12-16 20:55:48,648 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /127.0.0.1:8031
2013-12-16 20:55:49,755 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8031. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2013-12-16 20:55:50,756 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8031. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2013-12-16 20:55:51,757 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8031. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) …

我做的时候无法看到我的节点yarn node -list,即使我已经配置/etc/hadoop/conf/yarn-site.xml了正确的属性(在我看来,至少根据这个问题,奴隶节点不在Yarn ResourceManager中).

这是我到目前为止所做的:

• 安装resourcemanager在主人身上
• 安装nodemanager在奴隶上

• yarn-site.xml在所有节点上检查:

  <property> <name>yarn.resourcemanager.hostname</name> <value>master-node</value> </property>

• 修改配置文件后,重新启动resourcemanager并nodemanager分别在主服务器和从服务器上.

但是,当我这样做时,yarn node -list我只能看到

Total Nodes: 0 
Node-Id       Node-state    Node-Http-Address      Number-of-Running-Containers

在我的节点,我查看了.out文件/var/log/hadoop-yarn/,我在其中看到:

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 244592
max locked memory       (kbytes, -l) 64
max memory …