小编Mic*_*pik的帖子

我正在尝试在我的EMR Spark/Hive集群上运行HiveFromSpark示例.

问题

使用yarn-client:

~/spark/bin/spark-submit --master yarn-client --num-executors=19 --class org.apache.spark.examples.sql.hive.HiveFromSpark  ~/spark/lib/spark-examples-1.3.0-hadoop2.4.0.jar

奇迹般有效.但是,使用纱线集群:

~/spark/bin/spark-submit --master yarn-cluster --num-executors=19 --class org.apache.spark.examples.sql.hive.HiveFromSpark ~/spark/lib/spark-examples-1.3.0-hadoop2.4.0.jar

失败了:

Exception in thread "main" org.apache.spark.SparkException: Application finished with failed status
    at org.apache.spark.deploy.yarn.Client.run(Client.scala:622)
    at org.apache.spark.deploy.yarn.Client$.main(Client.scala:647)
    at org.apache.spark.deploy.yarn.Client.main(Client.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

和资源管理器中的错误消息说:

User class threw exception: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

完整日志:

15/04/15 12:02:14 WARN DataNucleus.Persistence: Error creating validator of …