SBT无法导入Kafka编码器/解码器类

Question

项目设置:

• 1个生产者 - 序列化对象并将字节发送到Kafka
• 1 spark consumer - 应该在kafka.serializer包中使用DefaultDecoder来消耗字节

问题:

• SBT导入正确的库(kafka-clients + kafka_2.10),但无法在kafka_2.10 jar中找到任何类.
• 好像它是在错误的路径下搜索(org.apache.spark.streaming.kafka而不是org.apache.kafka).

错误讯息:

    object serializer is not a member of package org.apache.spark.streaming.kafka [error] 
import kafka.serializer.DefaultDecoder.

SBT-树

    [info]   +-org.apache.spark:spark-streaming-kafka_2.10:1.6.1
    [info]   | +-org.apache.kafka:kafka_2.10:0.8.2.1 [S] <-- **DefaultDecoder is in here 
but SBT can't find it (org.apache.kafka.serialization.DefaultDecoder)**
    [info]   | | +-org.apache.kafka:kafka-clients:0.8.2.1

built.sbt:

  lazy val commonSettings = Seq(
  organization := "org.RssReaderDemo",
  version := "0.1.0",
  scalaVersion := "2.10.6"
)

resolvers += "Artima Maven Repository" at "http://repo.artima.com/releases"

val spark = "org.apache.spark" % "spark-core_2.10" % "1.6.1"
val sparkStreaming = "org.apache.spark" % "spark-streaming_2.10" % "1.6.1"
val sparkStreamKafka = "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.1"

// Needed to be able to parse the generated avro JSON schema
val jacksonMapperAsl = "org.codehaus.jackson" % "jackson-mapper-asl" % "1.9.13"

val scalactic = "org.scalactic" %% "scalactic" % "2.2.6"
val scalatest = "org.scalatest" %% "scalatest" % "2.2.6" % "test"

val avro = "org.apache.avro" % "avro" % "1.8.0"

lazy val root = (project in file(".")).
  settings(commonSettings: _*).
  settings(
    libraryDependencies += spark,
    libraryDependencies += sparkStreaming,
    libraryDependencies += sparkStreamKafka,
    libraryDependencies += jacksonMapperAsl,
    libraryDependencies += scalactic,
    libraryDependencies += scalatest,
    libraryDependencies += avro
  )

Answer 1

这与SBT无关.你可能有类似的东西

import org.apache.spark.streaming._
import kafka.serializer.DefaultDecoder

由于org.apache.spark.streaming.kafka包存在,此导入解析为org.apache.spark.streaming.kafka.serializer.DefaultDecoder.您可以按如下方式导入正确的类:import _root_.kafka.serializer.DefaultDecoder.见https://wiki.scala-lang.org/display/SYGN/Language+FAQs#LanguageFAQs-HowdoIimport关于斯卡拉进口更多的细节.