为什么Spark退出exitCode:16？

Question

我使用Spark 2.0.0和Hadoop 2.7并使用yarn-cluster模式.每次,我都会收到以下错误:

17/01/04 11:18:04 INFO spark.SparkContext: Successfully stopped SparkContext
17/01/04 11:18:04 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 16, (reason: Shutdown hook called before final status was reported.)
17/01/04 11:18:04 INFO util.ShutdownHookManager: Shutdown hook called
17/01/04 11:18:04 INFO util.ShutdownHookManager: Deleting directory /tmp/hadoop-hduser/nm-local-dir/usercache/harry/appcache/application_1475261544699_0833/spark-42e40ac3-279f-4c3f-ab27-9999d20069b8
17/01/04 11:18:04 INFO spark.SparkContext: SparkContext already stopped.

但是,我确实得到了正确的打印输出.相同的代码在Spark 1.4.0-Hadoop 2.4.0中工作正常,我没有看到任何退出代码.

Answer 1

如果应用程序退出错误,则无法清除 此问题.sparkStaging https://issues.apache.org/jira/browse/SPARK-17340在Spark 1.4之后启动(影响版本/ s:1.5.2,1.6.1,2.0.0)

问题是:当运行Spark(纱线,集群模式)和杀死应用程序时,不会清除.sparkStaging.

当这个问题发生在Spark 2.0.X中引发的exitCode 16时

ERROR ApplicationMaster: RECEIVED SIGNAL TERM
INFO ApplicationMaster: Final app status: FAILED, exitCode: 16, (reason: Shutdown hook called before final status was reported.)

是否有可能在您的代码中,有什么东西会杀死应用程序？如果是这样 - 它不应该在Spark 1.4中看到,但应该在Spark 2.0.0中看到

请在代码中搜索"退出"(就像你的代码中有这样的代码一样,错误不会在Spark 1.4中显示,但会在Spark 2.0.0中显示)