提交火花时ClassNotFoundException scala.runtime.LambdaDeserialize

Question

我遵循https://spark.apache.org/docs/2.1.0/quick-start.html上的Scala教程

我的scala文件

/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp {
  def main(args: Array[String]) {
    val logFile = "/data/README.md" // Should be some file on your system
    val conf = new SparkConf().setAppName("Simple Application")
    val sc = new SparkContext(conf)
    val logData = sc.textFile(logFile, 2).cache()
    val numAs = logData.filter(line => line.contains("a")).count()
    val numBs = logData.filter(line => line.contains("b")).count()
    println(s"Lines with a: $numAs, Lines with b: $numBs")
    sc.stop()
  }
}

和build.sbt

name := "Simple Project"

version := "1.0"

scalaVersion := "2.12.4"

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "2.2.0" 

我sbt package成功运行了（已经删除了scala源代码和build.sbt之外的所有内容，然后sbt package再次运行）

[info] Loading project definition from /home/cpu11453local/workspace/testspark_scala/project
[info] Loading settings from build.sbt ...
[info] Set current project to Simple Project (in build file:/home/my_name/workspace/testspark_scala/)
[info] Packaging /home/my_name/workspace/testspark_scala/target/scala-2.12/simple-project_2.12-1.0.jar ...
[info] Done packaging.
[success] Total time: 1 s, completed Nov 8, 2017 12:15:24 PM

但是，当我运行spark提交时

$SPARK_HOME/bin/spark-submit --class "SimpleApp" --master local[4] simple-project_2.12-1.0.jar 

我有错误

java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize

在全火花提交输出要点

Answer 1

正如@Alexey所说，将Scala版本更改为2.11可以解决此问题。

build.sbt

name := "Simple Project"

version := "1.0"

scalaVersion := "2.11.11"

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.0" 

请注意，Scala版本必须与Spark匹配。看一下artifactId，spark-core_2.11表示它与scala 2.11兼容（没有向后或向前兼容）