在使用数据框时获取异常:java.lang.NoSuchMethodError:scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)

Question

我在scala应用程序中使用数据帧并使用spark运行它时收到"java.lang.NoSuchMethodError:scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)"错误.但是,如果我只使用RDD而不使用数据帧,则不会出现相同的pom和设置这样的错误.另外,在浏览其他相同错误的帖子时,提到scala版本必须为2.10,因为spark与2.11 scala不兼容,我使用2.10 scala版本和2.0.0 spark.

Below is the snip from pom:



<properties>
      <spark-assembly>/usr/lib/spark/lib/spark-assembly.jar</spark-assembly>
      <encoding>UTF-8</encoding>
      <hadoop.version>2.7.1</hadoop.version>
      <hbase.version>1.1.1</hbase.version>
      <scala.version>2.10.5</scala.version>
      <scala.tools.version>2.10</scala.tools.version>
      <spark.version>2.0.0</spark.version>
      <phoenix.version>4.7.0-HBase-1.1</phoenix.version>
  </properties>

  <dependencies>
      <dependency>
          <groupId>org.scala-lang</groupId>
          <artifactId>scala-library</artifactId>
          <version>${scala.version}</version>
      </dependency>
      <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-client</artifactId>
          <version>${hadoop.version}</version>
      </dependency>
      <dependency>
          <groupId>org.apache.hbase</groupId>
          <artifactId>hbase-client</artifactId>
          <version>${hbase.version}</version>
      </dependency>
      <dependency>
          <groupId>org.apache.hbase</groupId>
          <artifactId>hbase-server</artifactId>
          <version>${hbase.version}</version>
      </dependency>
      <dependency>
          <groupId>org.apache.spark</groupId>
          <artifactId>spark-core_${scala.tools.version}</artifactId>
          <version>${spark.version}</version>
      </dependency>
      <dependency>
          <groupId>org.apache.spark</groupId>
          <artifactId>spark-sql_${scala.tools.version}</artifactId>
          <version>${spark.version}</version>
      </dependency>
      <dependency>
          <groupId>org.apache.spark</groupId>
          <artifactId>spark-hive_${scala.tools.version}</artifactId>
          <version>${spark.version}</version>
      </dependency>

  </dependencies>

错误:

16/10/19 02:57:26 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror;
java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror;
        at com.abc.xyz.Compare$.main(Compare.scala:64)
        at com.abc.xyz.Compare.main(Compare.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:627)
16/10/19 02:57:26 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror;)
16/10/19 02:57:26 INFO spark.SparkContext: Invoking stop() from shutdown hook

Answer 1

更改scala版本

<scala.version>2.11.8</scala.version>

<scala.tools.version>2.11</scala.tools.version>

并添加

<dependency>
    <groupId>org.scala-lang</groupId>
    <artifactId>scala-reflect</artifactId>
    <version>${scala.version}</version>
  </dependency>