NoClassDefFoundError: org/apache/spark/sql/internal/connector/SimpleTableProvider 在 Dataproc 中运行时

Question

我能够在独立模式下运行我的程序。但是当我尝试以集群模式在 Dataproc 中运行时，出现以下错误。PL 有帮助。我的 build.sbt

name := "spark-kafka-streaming"
    
  version := "0.1"
    
  scalaVersion := "2.12.10"
    
  val sparkVersion = "2.4.5"
    
  libraryDependencies += "org.apache.spark" %% "spark-sql" % sparkVersion % "provided"
  libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion % "provided"
  libraryDependencies += "com.google.cloud.bigdataoss" % "gcs-connector" % "hadoop2-2.1.3"
  libraryDependencies += "org.apache.spark" %% "spark-sql-kafka-0-10" % sparkVersion
    
  assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
  assemblyJarName in assembly := s"${name.value}_${scalaBinaryVersion.value}-${sparkVersion}_${version.value}.jar"
    
  assemblyMergeStrategy in assembly := {
      case PathList("org","aopalliance", xs @ _*) => MergeStrategy.last
      case PathList("javax", "inject", xs @ _*) => MergeStrategy.last
      case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
      case PathList("javax", "activation", xs @ _*) => MergeStrategy.last
      case PathList("org", "apache", xs @ _*) => MergeStrategy.last
      case PathList("com", "google", xs @ _*) => MergeStrategy.last
      case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
      case PathList("com", "codahale", xs @ _*) => MergeStrategy.last
      case PathList("com", "yammer", xs @ _*) => MergeStrategy.last
      case "about.html" => MergeStrategy.rename
      case "META-INF/ECLIPSEF.RSA" => MergeStrategy.last
      case "META-INF/mailcap" => MergeStrategy.last
      case "META-INF/mimetypes.default" => MergeStrategy.last
      case "plugin.properties" => MergeStrategy.last
      case "log4j.properties" => MergeStrategy.last
      case y: String if y.contains("UnusedStubClass") => MergeStrategy.first
      case x =>
        val oldStrategy = (assemblyMergeStrategy in assembly).value
        oldStrategy(x)
    }

引起：java.lang.ClassNotFoundException: org.apache.spark.sql.internal.connector.SimpleTableProvider at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java: [418] 第418话

使用的命令： spark-submit --class Main --master yarn --deploy-mode cluster --num-executors 1 --driver-memory 4g --executor-cores 4 --executor-memory 4g --files x.json y.jar

编辑：

集群配置：图像：1.5.4-debian10 spark-submit --version version 2.4.5 Using Scala version 2.12.10, OpenJDK 64-Bit Server VM, 1.8.0_252

Jar- 使用命令 sbt 程序集构建 Uber jar。

gcloud 命令：gcloud dataproc 作业提交 spark --cluster=xyz --region=us-west1 --class=Main --files x.json --jars=spark-kafka-streaming_2.12-3.0.0_0.1.jar

日志：

错误 org.apache.spark.deploy.yarn.Client：应用程序诊断消息：用户类抛出异常：java.lang.NoClassDefFoundError：org/apache/spark/sql/internal/connector/SimpleTableProvider at java.lang.ClassLoader.defineClass1(本机方法）在 java.lang.ClassLoader.defineClass(ClassLoader.java:756) 在 java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) 在 java.net.URLClassLoader.defineClass(URLClassLoader.java:468) 在 java .net.URLClassLoader.access$100(URLClassLoader.java:74) 在 java.net.URLClassLoader$1.run(URLClassLoader.java:369) 在 java.net.URLClassLoader$1.run(URLClassLoader.java:363) 在 java.security .AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at java.lang。ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader. java:370) 在 java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) 在 java.util.ServiceLoader$1.next(ServiceLoader.java:480) 在 scala.collection.convert.Wrappers$JIteratorWrapper.next( Wrappers.scala:44) at scala.collection.Iterator.foreach(Ite​​rator.scala:941) at scala.collection.Iterator.foreach$(Iterator.scala:941) at scala.collection.AbstractIterator.foreach(Ite​​rator.scala: 1429) at scala.collection.IterableLike.foreach(Ite​​rableLike.scala:74) at scala.collection.IterableLike.foreach$(IterableLike.scala:73) at scala.collection.AbstractIterable.foreach(Ite​​rable.scala:56) at scala .collection.TraversableLike。filterImpl(TraversableLike.scala:255) at scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:249) at scala.collection.AbstractTraversable.filterImpl(Traversable.scala:108) at scala.collection.TraversableLike.filter.(TraversableLike.filter) scala:347) at scala.collection.TraversableLike.filter$(TraversableLike.scala:347) at scala.collection.AbstractTraversable.filter(Traversable.scala:108) at org.apache.spark.sql.execution.datasources.DataSource$ .lookupDataSource(DataSource.scala:630) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:194) at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:645) at org。 apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:629) 在 Lineage$.delayedEndpoint$Lineage$1(Lineage.scala:17) 在 Lineage$delayedInit$body.apply(Lineage.scala:3) 在 scala.Function0 .apply$mcV$sp(Function0.scala:39) at scala.Function0.apply$mcV$sp$(Function0.scala:39) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17) at scala.App.$anonfun$main$1$adapted(App.scala:80) at scala.collection.immutable.List.foreach(List.scala:392) at scala.App.main(App.scala:80) at scala .App.main$(App.scala:78) at Lineage$.main(Lineage.scala:3) at Lineage.main(Lineage.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect。 NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy .yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:686) 引起：java.lang.ClassNotFoundException：org.apache。spark.sql.internal.connector.SimpleTableProvider 在 java.net.URLClassLoader.findClass(URLClassLoader.java:382) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:418) 在 java.lang.ClassLoader.loadClass(ClassLoader. java:351) ... 49 更多

根本原因和解决方案： 正如答案中所指出的，这是罐子的问题。我正在使用 IDEA sbt shell 来构建 jar。并且在 shell 启动后不会再次加载 build.sbt 中所做的任何更改。所以，虽然我改变了版本，但它没有被选中，直到我再次重新启动 sbt shell。学得很辛苦。

Answer 1

基于 assemblyJarName in assembly := s"${name.value} ${scalaBinaryVersion.value}-${sparkVersion} ${version.value}.jar" 你的jar名称应该是spark-kafka-streaming_2.12-2.4 .5_0.1.jar。实际的 jar 名称暗示您可能正在使用 Spark 3.0.0 API 并部署在 Spark 2.4.5 上