我想用我的两个虚拟机将Spark Standlone模式安装到Cluster.
使用spark-0.9.1-bin-hadoop1的版本,我在每个vm中成功执行spark-shell.我按照官方文档制作一个vm(ip:xx.xx.xx.223)作为Master和Worker,并将另一个(ip:xx.xx.xx.224)作为Worker.
但是224-ip vm无法连接223-ip vm.接下来是223(Master)的主日志:
[@tc-52-223 logs]# tail -100f spark-root-org.apache.spark.deploy.master.Master-1-tc-52-223.out
Spark Command: /usr/local/jdk/bin/java -cp :/data/test/spark-0.9.1-bin-hadoop1/conf:/data/test/spark-0.9.1-bin-hadoop1/assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop1.0.4.jar -Dspark.akka.logLifecycleEvents=true -Djava.library.path= -Xms512m -Xmx512m org.apache.spark.deploy.master.Master --ip 10.11.52.223 --port 7077 --webui-port 8080
log4j:WARN No appenders could be found for logger (akka.event.slf4j.Slf4jLogger).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
14/04/14 22:17:03 INFO Master: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/04/14 22:17:03 INFO Master: Starting Spark master at spark://10.11.52.223:7077
14/04/14 22:17:03 INFO MasterWebUI: Started Master web UI at http://tc-52-223:8080 …我正在学习Spark,并希望运行由两台物理机器组成的最简单的集群.我已完成所有基本设置,似乎没问题.自动启动脚本的输出如下所示:
[username@localhost sbin]$ ./start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /home/username/spark-1.6.0-bin-hadoop2.6/logs/spark-username-org.apache.spark.deploy.master.Master-1-localhost.out
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /home/sername/spark-1.6.0-bin-hadoop2.6/logs/spark-username-org.apache.spark.deploy.worker.Worker-1-localhost.out
username@192.168.???.??: starting org.apache.spark.deploy.worker.Worker, logging to /home/username/spark-1.6.0-bin-hadoop2.6/logs/spark-username-org.apache.spark.deploy.worker.Worker-1-localhost.localdomain.out
所以这里没有错误,似乎主节点正在运行以及两个Worker节点.但是,当我在192.168.?????:8080打开WebGUI时,它只列出一个工作人员 - 本地工作人员.我的问题与此处描述的类似:Spark Clusters:工作者信息不会显示在Web UI上,但我的/ etc/hosts文件中没有任何内容.它包含的全部是:
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
我错过了什么？两台机器都运行Fedora Workstation x86_64.