相关疑难解决方法(0)

我是Scala/Spark堆栈的新手,我正试图弄清楚如何使用SparkSql来测试我的基本技能,以便在TempTables中"映射"RDD,反之亦然.

我有两个不同的.scala文件,具有相同的代码:一个简单的对象(使用def main ...)和一个扩展App的对象.

在简单的对象中,我得到一个错误,因为"No TypeTag available"连接到我的case类Log:

object counter {
  def main(args: Array[String]) {
.
.
.
   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
   import sqlContext.createSchemaRDD
   case class Log(visitatore: String, data: java.util.Date, pagina: String, count: Int)
   val log = triple.map(p => Log(p._1,p._2,p._3,p._4))
   log.registerTempTable("logs")
   val logSessioni= sqlContext.sql("SELECT visitor, data, pagina, count FROM logs")
   logSessioni.foreach(println)
}

行中的错误:log.registerTempTable("logs")表示"没有TypeTag可用于日志".

在另一个文件(对象扩展App)中一切正常:

object counterApp extends App {
.
.
.
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    import sqlContext.createSchemaRDD
    case class Log(visitatore: String, data: java.util.Date, pagina: String, count: …

我在其他SO帖子中读过这个问题,但我仍然不知道我做错了什么.原则上,添加以下两行:

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._

应该做的伎俩,但错误仍然存​​在

这是我的build.sbt:

name := "PickACustomer"

version := "1.0"

scalaVersion := "2.11.7"


libraryDependencies ++= Seq("com.databricks" %% "spark-avro" % "2.0.1",
"org.apache.spark" %% "spark-sql" % "1.6.0",
"org.apache.spark" %% "spark-core" % "1.6.0")

我的scala代码是:

import scala.collection.mutable.Map
import scala.collection.immutable.Vector

import org.apache.spark.rdd.RDD
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.sql._


    object Foo{

    def reshuffle_rdd(rawText: RDD[String]): RDD[Map[String, (Vector[(Double, Double, String)], Map[String, Double])]]  = {...}

    def do_prediction(shuffled:RDD[Map[String, (Vector[(Double, Double, String)], Map[String, Double])]], prediction:(Vector[(Double, Double, String)] => Map[String, Double]) ) : RDD[Map[String, …