使用spark-avro将Avro读入火花

Question

我无法使用spark-avro库读取spark文件.以下是我采取的步骤:

• 从以下网站获取jar:http://mvnrepository.com/artifact/com.databricks/spark-avro_2.10/0.1
• 使用调用spark-shell spark-shell --jars avro/spark-avro_2.10-0.1.jar

• 执行git自述文件中给出的命令:

  import com.databricks.spark.avro._
  import org.apache.spark.sql.SQLContext
  val sqlContext = new SQLContext(sc)
  val episodes = sqlContext.avroFile("episodes.avro") 
  

• 操作sqlContext.avroFile("episodes.avro")失败,并显示以下错误:

  scala> val episodes = sqlContext.avroFile("episodes.avro")
  java.lang.IncompatibleClassChangeError: class com.databricks.spark.avro.AvroRelation has interface org.apache.spark.sql.sources.TableScan as super class
  at java.lang.ClassLoader.defineClass1(Native Method)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
  at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
  at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
  

Answer 1

我的错.自述文件清楚地说:

Versions

Spark changed how it reads / writes data in 1.4, so please use the correct version of this dedicated for your spark version

1.3 -> 1.0.0

1.4+ -> 1.1.0-SNAPSHOT

我用spark:1.3.1和spark-avro : 1.1.0. 当我使用spark-avro时1.0.0,它起作用了.