如何在Zeppelin上运行Spark来分析xml文件

Question

我可以运行Spark shell bin/spark-shell --packages com.databricks:spark-xml_2.11:0.3.0来分析xml文件,例如:

import org.apache.spark.sql.SQLContext

val sqlContext = new SQLContext(sc)
val df = sqlContext.read
    .format("com.databricks.spark.xml")
    .option("rowTag", "book")
    .load("books.xml")

但我怎么能运行Zeppelin来做到这一点.Zeppelin在开始导入时是否需要一些参数com.databricks.spark.xml？现在我得到:

java.lang.RuntimeException:无法加载数据源的类:com.databricks.spark.xml at scala.sys.package $ .error(package.scala:27)at org.apache.spark.sql.sources.ResolvedDataSource $ .lookupDataSource(ddl.scala:220)atg.apache.spark.sql.sources.ResolvedDataSource $ .apply(ddl.scala:233)at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:114) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:104)at $ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC.(:26)at $ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC.(:31)at $ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC.(:33)at $ iwC $$ iwC $$ iwC $$ iwC $$ iwC.(:35)at $ iwC $$ iwC $$ iwC $$ iwC.(:37)at $ iwC $$ iwC $$ iwC.(:39)at $ iwC $$ iwC.(:41)at $ iwC.(:41)at(:45)at.(:49)at.()at.(:7)at.()at $ print()at sun. reflect.NativeMethodAccessorImpl.invoke0(本机方法)在sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)在sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)在java.lang.reflect.Me thod.invoke(Method.java:497)在org.apache.spark.repl.SparkIMain $ ReadEvalPrint.call(SparkIMain.scala:1065)在org.apache.spark.repl.SparkIMain $ Request.loadAndRun(SparkIMain.scala: 1338)在org.apache.spark.repl.SparkIMain.loadAndRunReq $ 1(SparkIMain.scala:840)在org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)在org.apache.spark.repl. SparkIMain.interpret(SparkIMain.scala:819)在org.apache.zeppelin.spark.SparkInterpreter.interpretInput(SparkInterpreter.java:709)在org.apache.zeppelin.spark.SparkInterpreter.interpret(SparkInterpreter.java:674)在组织.apache.zeppelin.spark.SparkInterpreter.interpret(SparkInterpreter.java:667)org.apache.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:57)at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter) .java:93)atg.apache.zeppelin.interpreter.remote.RemoteInterpreterServer $ InterpretJob.jobRun(RemoteInterpreterServer.java:300)at org.apache.zeppelin.scheduler.Job.在java的java.util.concurrent.Executors $ RunnableAdapter.call(Executors.java:511)的org.apache.zeppelin.scheduler.FIFOScheduler $ 1.run(FIFOScheduler.java:134)上运行(Job.java:169). util.concurrent.FutureTask.run(FutureTask.java:266)在java.util.concurrent.ScheduledThreadPoolExecutor中$ $ ScheduledFutureTask.access 201(ScheduledThreadPoolExecutor.java:180)在java.util.concurrent.ScheduledThreadPoolExecutor中$ ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java :293)在java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)在java.util.concurrent.ThreadPoolExecutor中的$ Worker.run(ThreadPoolExecutor.java:617)在java.lang.Thread.run(线程. Java的:745)

Answer 1

在Zeppelin中,您需要在创建SparkContext之前调用这些依赖项.

在单独的单元格中,添加并运行以下内容

%dep
z.reset()
z.addRepo("Spark Packages Repo").url("http://dl.bintray.com/spark-packages/maven")
z.load("com.databricks:spark-xml_2.11:0.3.0")

如果这给你一个类型的错误:"你必须在启动SparkContext之前添加依赖项",只需重新启动解释器或Zeppelin.