当我向集群提交 spark 作业时,它在 shell 中失败并显示以下异常:
> Exception in thread "main" org.apache.spark.SparkException:
> Application application_1497125798633_0065 finished with failed status
> at org.apache.spark.deploy.yarn.Client.run(Client.scala:1244)
> at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1290)
> at org.apache.spark.deploy.yarn.Client.main(Client.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:750)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 17/06/29 10:25:36 INFO ShutdownHookManager: Shutdown hook called
Run Code Online (Sandbox Code Playgroud)
这是它在 Yarn 日志中给出的内容:
> Caused by: java.io.IOException: Failed to connect to /0.0.0.0:35994 at
> org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:232)
> at
> …
Run Code Online (Sandbox Code Playgroud)