SparkContext错误 - 找不到文件/ tmp/spark-events不存在

Question

通过API调用运行Python Spark应用程序 - 提交应用程序 - 响应 - SSH失败进入Worker

我的python应用程序存在于

/root/spark/work/driver-id/wordcount.py

可以在中找到错误

/root/spark/work/driver-id/stderr

显示以下错误 -

Traceback (most recent call last):
  File "/root/wordcount.py", line 34, in <module>
    main()
  File "/root/wordcount.py", line 18, in main
    sc = SparkContext(conf=conf)
  File "/root/spark/python/lib/pyspark.zip/pyspark/context.py", line 115, in __init__
  File "/root/spark/python/lib/pyspark.zip/pyspark/context.py", line 172, in _do_init
  File "/root/spark/python/lib/pyspark.zip/pyspark/context.py", line 235, in _initialize_context
  File "/root/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 1064, in __call__
  File "/root/spark/python/lib/py4j-0.9-src.zip/py4j/protocol.py", line 308, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.io.FileNotFoundException: File file:/tmp/spark-events does not exist.
  at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:402)
  at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:255)
  at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:100)
  at org.apache.spark.SparkContext.<init>(SparkContext.scala:549)
  at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
  at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
  at py4j.Gateway.invoke(Gateway.java:214)
  at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
  at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
  at py4j.GatewayConnection.run(GatewayConnection.java:209)
  at java.lang.Thread.run(Thread.java:745)

它表示 - /tmp/spark-events不存在 - 这是真的但是,在wordcount.py中

from pyspark import SparkContext, SparkConf

... few more lines ...

def main():
    conf = SparkConf().setAppName("MyApp").setMaster("spark://ec2-54-209-108-127.compute-1.amazonaws.com:7077")
    sc = SparkContext(conf=conf)
    sc.stop()

if __name__ == "__main__":
    main()

Answer 1

/tmp/spark-events是Spark存储事件日志的位置.只需在主机中创建此目录即可进行设置.

$mkdir /tmp/spark-events
$ sudo /root/spark-ec2/copy-dir /tmp/spark-events/
RSYNC'ing /tmp/spark-events to slaves...
ec2-54-175-163-32.compute-1.amazonaws.com

Answer 2

在尝试在我的本地计算机上设置我的spark历史记录服务器时,我有相同的'文件文件:/ tmp/spark-events不存在.错误.我已将我的日志目录自定义为非默认路径.要解决这个问题,我需要做两件事.

1. 编辑$ SPARK_HOME/conf/spark-defaults.conf - 添加这两行 spark.history.fs.logDirectory /mycustomdir spark.eventLog.enabled true
2. 创建从/ tmp/spark-events到/ mycustomdir的链接.
   ln -fs /tmp/spark-events /mycustomdir 理想情况下,第1步将完全解决我的问题,但我仍然需要创建链接,所以我怀疑可能还有一个我错过的其他设置.无论如何,一旦我这样做,我就可以运行我的历史服务器并查看我的webui中记录的新作业.

Answer 3

使用spark.eventLog.dir作为客户端/驱动程序

spark.eventLog.dir=/usr/local/spark/history

并使用spark.history.fs.logDirectory作为历史服务器

spark.history.fs.logDirectory=/usr/local/spark/history

如：如何为独立集群非hdfs模式启用spark-history服务器

至少按照 Spark 版本 2.2.1