无法在HDP 2.5.3上手动启动spark 2.1.0

Question

我在spark 2.0.0上测试了我的火花代码,我遇到了一个错误SPARK-17463,我想使用spark 2.1.0,因为这个版本修复了bug.

但是,我无法为spark 2.1.0提供带有客户端模式的spark-shell

我需要在HDP 2.5.3集群上运行2.1.0.

它引发了一个异常:

    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/02/15 14:28:46 WARN Utils: Service 'SparkUI' could not bind on port 4045. Attempting port 4046.
    java.lang.NoClassDefFoundError: com/sun/jersey/api/client/config/ClientConfig
      at org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
      at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
      at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
      at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
      at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:151)
      at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
      at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
      at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
      at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
      at scala.Option.getOrElse(Option.scala:121)
      at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
      at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95)
      ... 47 elided
    Caused by: java.lang.ClassNotFoundException: com.sun.jersey.api.client.config.ClientConfig
      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      ... 61 more
<console>:14: error: not found: value spark
           import spark.implicits._
                          ^
<console>:14: error: not found: value spark
           import spark.sql
                          ^

Answer 1

在将EMersey-bundle-1.19.1.jar添加到$ SPARK_HOME/jars之后,我在EMR中遇到了与spark-submit相同的问题,问题得到了解决.

你可以从这里下载:http://repo1.maven.org/maven2/com/sun/jersey/jersey-bundle/1.19.1/jersey-bundle-1.19.1.jar

Answer 2

YARN 时间服务与 Spark 提供的库不兼容。请通过设置禁用时间服务spark.hadoop.yarn.timeline-service.enabled=false。

更多详情请访问https://issues.apache.org/jira/browse/SPARK-15343

Answer 3

在 spark-defaults.conf 中添加以下参数并重新启动 Spark 历史服务器。

spark.hadoop.yarn.timeline-service.enabled false