oct*_*ian 5 scala sbt apache-spark
我有以下build.sbt文件:
name := "myProject"
version := "1.0"
scalaVersion := "2.11.8"
javaOptions ++= Seq("-Xms512M", "-Xmx2048M", "-XX:MaxPermSize=2048M", "-XX:+CMSClassUnloadingEnabled")
dependencyOverrides ++= Set(
"com.fasterxml.jackson.core" % "jackson-core" % "2.8.1"
)
// additional libraries
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.0.0" % "provided",
"org.apache.spark" %% "spark-sql" % "2.0.0" % "provided",
"org.apache.spark" %% "spark-hive" % "2.0.0" % "provided",
"com.databricks" %% "spark-csv" % "1.4.0",
"org.scalactic" %% "scalactic" % "2.2.1",
"org.scalatest" %% "scalatest" % "2.2.1" % "test",
"org.scalacheck" %% "scalacheck" % "1.12.4",
"com.holdenkarau" %% "spark-testing-base" % "2.0.0_0.4.4" % "test",
)
Run Code Online (Sandbox Code Playgroud)
但是,当我运行代码时,出现此错误:
An exception or error caused a run to abort.
java.lang.ExceptionInInitializerError
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Jackson version is too old 2.4.4
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:56)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:549)
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
... 58 more
Run Code Online (Sandbox Code Playgroud)
为什么会这样?我已将较新版本的 Jackson 添加到dependencyOverrides(在查看此处Spark Parallelize 之后?(找不到名称为“id”的创建者属性)),因此不应使用旧版本。
jackson-core 和 jackson-databind 版本应该匹配(我相信至少到次要版本)。
所以删除dependencyOverrides并有
libraryDependencies ++= Seq(
...
"com.fasterxml.jackson.core" % "jackson-databind" % "2.8.1"
)
Run Code Online (Sandbox Code Playgroud)
或者在中指定两者dependencyOverrides
dependencyOverrides ++= Set(
"com.fasterxml.jackson.core" % "jackson-core" % "2.8.1"
"com.fasterxml.jackson.core" % "jackson-databind" % "2.8.1"
)
Run Code Online (Sandbox Code Playgroud)
虽然我不确定我是否理解你想要做什么;链接的问题似乎是说你应该使用旧版本(2.4.4)。
| 归档时间: |
|
| 查看次数: |
8480 次 |
| 最近记录: |