用于Hadoop EC2安装的Spark脚本:IPC客户端连接被拒绝

Question

我试图通过EC2的spark脚本在EC2集群设置上使用distcp在Hadoop和Amazon S3之间进行复制

[root]# bin/hadoop distcp s3n://bucket/f1 hdfs:///user/root/

我得到的错误是

INFO ipc.Client: Retrying connect to server: .. Already tried n time(s).

Copy failed: java.net.ConnectException: Call to ..my_server failed on connection excep\
tion: java.net.ConnectException: Connection refused

Answer 1

Spark EC2脚本可能是故意的,不会启动JobTracker和TaskTracker服务.因此,在运行Spark EC2安装脚本后,为了启动服务,我跑了

{HADOOP_HOME}/bin/start-all.sh

参考:感谢Brock Noland,网址为https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/H6wAfdeLIJo