小编Div*_* DD的帖子

这个问题与此类似，但是没有答案。

我正在尝试在YARN模式下的Spark中启用动态分配。我有11个节点群集，其中包含1个主节点和10个工作节点。我在下面的链接中获取指示：

要在YARN中进行设置：http : //spark.apache.org/docs/latest/running-on-yarn.html#configuring-the-external-shuffle-service

需要在spark-defaults.conf中设置配置变量：https : //spark.apache.org/docs/latest/configuration.html#dynamic-allocation https://spark.apache.org/docs/latest/configuration。 html＃shuffle-行为

我还从下面的链接和其他一些资源中获取了参考：https : //jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-dynamic-allocation.html#spark.dynamicAllocation.testing

这是我正在执行的步骤：

1. 在spark-defaults.conf中设置配置变量。我与动态分配和随机播放服务有关的spark-defaults.conf如下：

   spark.dynamicAllocation.enabled=true
   spark.shuffle.service.enabled=true
   spark.shuffle.service.port=7337
   

   Run Code Online (Sandbox Code Playgroud)

2. 在yarn-site.xml中进行更改

   spark.dynamicAllocation.enabled=true
   spark.shuffle.service.enabled=true
   spark.shuffle.service.port=7337
   

   Run Code Online (Sandbox Code Playgroud)

   所有这些步骤都在所有工作程序节点中复制，即spark-defaults.conf具有上述值，而yarn-site.xml具有这些属性。我已确保所有工作节点中都存在/home/hadoop/spark/common/network-yarn/target/scala-2.11/spark-2.2.2-SNAPSHOT-yarn-shuffle.jar。

3. 然后，我在工作节点和主节点中运行$ SPARK_HOME / sbin / start-shuffle-service.sh。在主节点中，我先使用stop-yarn.sh然后再使用start-yarn.sh重新启动YARN。

4. 然后我正在做YARN node -list -all来查看工作程序节点，但是我看不到任何节点

5. 当我禁用财产时

   <property>
       <name>yarn.nodemanager.aux-services</name>
       <value>spark_shuffle</value>
   </property>
   <property>
       <name>yarn.nodemanager.auxservices.spark_shuffle.class</name>
       <value>org.apache.spark.network.yarn.YarnShuffleService</value>
   </property>
   <property>
       <name>yarn.nodemanager.recovery.enabled</name>
       <value>true</value>
   </property>
   <property>
       <name>yarn.application.classpath</name>
       <value> $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/common/*,$HADOOP_MAPRED_HOME/share/hadoop/common/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/hdfs/*,$HADOOP_MAPRED_HOME/share/hadoop/hdfs/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/yarn/*,$HADOOP_MAPRED_HOME/share/hadoop/yarn/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/tools/*,$HADOOP_MAPRED_HOME/share/hadoop/tools/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/client/*,$HADOOP_MAPRED_HOME/share/hadoop/client/lib/*,/home/hadoop/spark/common/network-yarn/target/scala-2.11/spark-2.2.2-SNAPSHOT-yarn-shuffle.jar </value>
   </property>
   

   Run Code Online (Sandbox Code Playgroud)

   我可以正常看到所有工作节点，因此似乎洗牌服务未正确配置。

我正在尝试使用 S3A 分区（或目录，因为我只需要确认提交者是否按预期工作）提交者与 Spark。我正在关注这个链接，它应该很简单，但是我在解决上一个问题时遇到了新问题

用于测试的代码是（内部spark-shell）：

val sourceDF = spark.range(0, 10000)
val datasets = "s3a://bucket-name/test"
sourceDF.write.format("orc").save(datasets + "orc")

spark-defaults.conf 是：

spark.hadoop.fs.s3a.committer.name directory

spark.sql.sources.commitProtocolClass org.apache.spark.internal.io.cloud.PathOutputCommitProtocol

spark.sql.parquet.output.committer.class org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter


Error 1:
scala> sourceDF.write.format("orc").save(datasets + "orc")
java.lang.NoClassDefFoundError: 
org/apache/hadoop/mapreduce/lib/output/PathOutputCommitter
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:230)
at org.apache.spark.internal.io.FileCommitProtocol$.instantiate(FileCommitProtocol.scala:144)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:98)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
at org.apache.spark.sql.execution.datasources.DataSource.writeInFileFormat(DataSource.scala:435)
at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:471)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56) …