Spark Streaming与Kafka 2.0.0依赖项

Question

我正在尝试使用spark-streaming2.0.0来使用kafka 0.8主题,我正在尝试识别我在build.sbt文件中尝试使用这些依赖项所需的依赖项

libraryDependencies += "org.apache.spark" %% "spark-streaming_2.11" % "2.0.0"

当我运行sbt包时,我得到所有这三个罐子的未解决的依赖关系,

但这些罐子确实存在

https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-8_2.11/2.0.0

请帮忙调试这个问题,我是Scala的新手,所以如果我没有做对,请告诉我

Answer 1

问题是您正在指定Scala版本,并且还使用%%哪个尝试来推断您正在使用的Scala版本.

删除一个%:

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.0.0"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.11" % "2.0.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % "2.0.0"

或删除Scala版本:

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0"
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.0.0"
libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka-0-8" % "2.0.0"