如何使用Zookeeper为HA设置Spark？

Question

我想配置Apache spark master来连接Zookeeper

我已经安装了它们并运行Zookeeper.

在spark-env.sh中,我添加了2行:

-Dspark.deploy.recoveryMode=ZOOKEEPER

-Dspark.deploy.zookeeper.url=localhost:2181

但是当我使用./sbin/start-all.sh启动Apache spark时

它显示错误

/home/deploy/spark-1.0.0/sbin/../conf/spark-env.sh: line 46: -Dspark.deploy.recoveryMode=ZOOKEEPER: command not found

/home/deploy/spark-1.0.0/sbin/../conf/spark-env.sh: line 47: -Dspark.deploy.zookeeper.url=localhost:2181: command not found

我想知道如何在spark-env.sh上添加Zookeeper设置

Answer 1

很可能你已经将这些行直接添加到文件中,如下所示:

export SPARK_PREFIX=`dirname "$this"`/..
export SPARK_CONF_DIR="$SPARK_HOME/conf"
...
-Dspark.deploy.recoveryMode=ZOOKEEPER
-Dspark.deploy.zookeeper.url=localhost:2181

当start-all.sh调用时,bash会抱怨这些-Dspark...命令不是有效命令.请注意,这spark_config.sh是一个bash脚本,应该包含有效的bash表达式.

继在配置指南高可用性,您应该设置SPARK_DAEMON_JAVA_OPTS与选项:spark.deploy.recoveryMode,spark.deploy.zookeeper.url,和spark.deploy.zookeeper.dir.

使用您的数据,您需要添加一行,spark-conf.sh如下所示:

export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=localhost:2181"