使用spark cassandra连接器和SBT编译错误

Question

我正在尝试让DataStax spark cassandra连接器正常工作.我在IntelliJ中创建了一个新的SBT项目,并添加了一个类.该类和我的sbt文件如下所示.创建spark上下文似乎有效,但是,当我取消注释我尝试创建cassandraTable的那一行时,我得到以下编译错误:

错误:scalac:错误的符号引用.CassandraRow.class中的签名是指包org.apache.spark.sql中的术语催化剂,它不可用.它可能在当前类路径中完全丢失,或者类路径上的版本可能与编译CassandraRow.class时使用的版本不兼容.

Sbt对我来说是一种新的东西,我很感激任何帮助,以了解这个错误的含义(当然,如何解决它).

name := "cassySpark1"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.1.0"

libraryDependencies += "com.datastax.spark" % "spark-cassandra-connector" % "1.1.0" withSources() withJavadoc()

libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector-java" % "1.1.0-alpha2" withSources() withJavadoc()

resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

而我的班级:

import org.apache.spark.{SparkConf,SparkContext}

import com.datastax.spark.connector._

object HelloWorld {def main(args:Array [String]):Unit = {System.setProperty("spark.cassandra.query.retry.count","1")

val conf = new SparkConf(true)
  .set("spark.cassandra.connection.host", "cassandra-hostname")
  .set("spark.cassandra.username", "cassandra")
  .set("spark.cassandra.password", "cassandra")

val sc = new SparkContext("local", "testingCassy", conf)

> // val foo = sc.cassandraTable("keyspace name","table name")

val rdd = sc.parallelize(1 to 100)
val sum = rdd.reduce(_+_)

println(sum)   } }

Answer 1

您需要添加spark-sql到依赖项列表

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.1.0"