火花上的火花,在/0.0.0.0:8032连接到ResourceManager

Question

我在我的开发机器上写了一个火花程序,这是一个mac.hadoop的版本是2.6,spark的版本是1.6.2.hadoop集群有3个节点,当然都在linux机器上.我在spark独立模式的IDE中运行spark程序,它运行成功.但现在,我将其更改为纱线客户端模式,它无法正常工作,并给出如下消息:

...
2017-02-23 11:01:33,725-[HL] INFO main org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at /0.0.0.0:8032
2017-02-23 11:01:34,839-[HL] INFO main org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-02-23 11:01:35,842-[HL] INFO main org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-02-23 11:01:36,847-[HL] INFO main org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-02-23 11:01:37,854-[HL] INFO main org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
...

我已经将相应的配置文件添加到项目的资源目录中.如果我把它作为一个jar包并spark-submit用来运行这个程序,它就没问题了.现在,我想在IDE中运行这个程序作为yarn-client模式,就像spark独立模式一样.我该如何解决这个问题？谢谢.

Answer 1

确保以纱线模式运行时，Spark可以使用YARN配置。这些文件添加core-site.xml，hdfs-site.xml以及yarn-site.xml文件的conf火花目录。
还要确保，其中yarn-site.xml包含资源管理器的地址

<property>
   <name>yarn.resourcemanager.address</name>
   <value>resource_manager_ip:8032</value>
</property>