Apache Spark:java.lang.NoSuchMethodError .rddToPairRDDFunctions

Question

sbt package运行得很好,但在spark-submit我收到错误后:

线程"main"中的异常java.lang.NoSuchMethodError:org.apache.spark.SparkContext $ .rddToPairRDDFunctions(Lorg/apache/spark/rdd/RDD; Lscala/reflect/ClassTag; Lscala/reflect/ClassTag; Lscala/math/Ordering ;)Lorg /阿帕奇/火花/ RDD/PairRDDFunctions; 在SmokeStack $ .main(SmokeStack.scala:46)位于sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)的sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.Iv.:60)太阳的SmokeStack.main(SmokeStack.scala)位于org.apache.spark.deploy.SparkSubmit $ .org $ apache $ spark $ deploy $的java.lang.reflect.Method.invoke(Method.java:498)中的.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)在org.apache.spark.spark.deploy.SparkSubmit $ .submit(SparkSubmit.scala:)org.apache.spark.deploy.SparkSubmit $ .doRunMain $ 1(SparkSubmit.scala:185)的SparkSubmit $$ runMain(SparkSubmit.scala:736) 210)org.apache.spark.deploy.SparkSubmit $ .main(SparkSubmit.scala:124)at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

这是违规行:

val sigCounts = rowData.map(row => (row("Signature"), 1)).countByKey()

rowData是RDD Map [String,String]."签名"键存在于地图中的所有项目中.

我怀疑这可能是一个构建问题.下面是我的sbt文件:

name := "Example1"
version := "0.1"
scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.0"
scalacOptions ++= Seq("-feature")

我是Scala的新手所以也许进口不正确？我有:

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import scala.io.Source

Answer 1

java.lang.NoSuchMethodError 通常表明编译的代码版本的版本高于运行时使用的库版本.

使用Spark,这意味着用于编译的Spark版本与部署的版本(在计算机或集群上)不同.

在开发和运行时之间对齐版本应该可以解决此问题.