是否可以将Apache Livy配置为与Spark Standalone一起运行？

Question

在我安装Apache Livy的机器上(在Ubuntu 16.04上):

(a)是否可以在Spark Standalone模式下运行它？

我正在考虑使用预装为Hadoop 2.6的Spark 1.6.3,可从https://spark.apache.org/downloads.html下载

(b)如果是,我该如何配置？

(c)HADOOP_CONF_DIR应该用于Spark Standalone？链接https://github.com/cloudera/livy提到了以下环境变量:

export SPARK_HOME=/usr/lib/spark
export HADOOP_CONF_DIR=/etc/hadoop/conf

我已经成功构建了Livy,除了上一个任务,它在Spark安装上待决:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] livy-api ........................................... SUCCESS [  9.984 s]
[INFO] livy-client-common ................................. SUCCESS [  6.681 s]
[INFO] livy-test-lib ...................................... SUCCESS [  0.647 s]
[INFO] livy-rsc ........................................... SUCCESS [01:08 min]
[INFO] livy-core_2.10 ..................................... SUCCESS [  7.225 s]
[INFO] livy-repl_2.10 ..................................... SUCCESS [02:42 min]
[INFO] livy-core_2.11 ..................................... SUCCESS [ 56.400 s]
[INFO] livy-repl_2.11 ..................................... SUCCESS [03:06 min]
[INFO] livy-server ........................................ SUCCESS [02:12 min]
[INFO] livy-assembly ...................................... SUCCESS [ 15.959 s]
[INFO] livy-client-http ................................... SUCCESS [ 25.377 s]
[INFO] livy-scala-api_2.10 ................................ SUCCESS [ 40.336 s]
[INFO] livy-scala-api_2.11 ................................ SUCCESS [ 40.991 s]
[INFO] minicluster-dependencies_2.10 ...................... SUCCESS [ 24.400 s]
[INFO] minicluster-dependencies_2.11 ...................... SUCCESS [  5.489 s]
[INFO] livy-integration-test .............................. SUCCESS [ 37.473 s]
[INFO] livy-coverage-report ............................... SUCCESS [  3.062 s]
[INFO] livy-examples ...................................... SUCCESS [  6.841 s]
[INFO] livy-python-api .................................... FAILURE [  8.053 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13:59 min
[INFO] Finished at: 2016-11-29T13:14:10-08:00
[INFO] Final Memory: 76M/2758M
[INFO] ------------------------------------------------------------------------

谢谢.

Answer 1

确保您已设置

export SPARK_HOME=/path/to/spark/home

然后运行mvn -DskipTests package。

即使没有，它也应该可以工作HADOOP_CONF_DIR