我有以下简单SparkR程序,即从中创建SparkR DataFrame和检索/收集数据.
Sys.setenv(HADOOP_CONF_DIR = "/etc/hadoop/conf.cloudera.yarn")
Sys.setenv(SPARK_HOME = "/home/user/Downloads/spark-1.6.1-bin-hadoop2.6")
.libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))
library(SparkR)
sc <- sparkR.init(master="yarn-client",sparkEnvir = list(spark.shuffle.service.enabled=TRUE,spark.dynamicAllocation.enabled=TRUE,spark.dynamicAllocation.initialExecutors="40"))
hiveContext <- sparkRHive.init(sc)
n = 1000
x = data.frame(id = 1:n, val = rnorm(n))
xs <- createDataFrame(hiveContext, x)
xs
head(xs)
collect(xs)
我能够成功创建它并查看信息,但任何与获取数据相关的操作都会抛出错误.
16/07/25 16:33:59 WARN TaskSetManager:阶段17.0中丢失的任务0.3(TID 86,wlos06.nrm.minn.seagate.com):java.net.SocketTimeoutException:接受在java.net.PlainSocketImpl超时. java.net.ServerSocket.implAccept(ServerSocket.java:530)java.net.ServerSocket.accept(ServerSocket.java:498)java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)上的socketAccept(Native Method) org.apache.apark.api.r.RRDD $ .createRWorker(RRDD.scala:432)位于org.apache.spark的org.apache.spark.api.r.BaseRRDD.compute(RRDD.scala:63). rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)atg.apache.spark.rdd.RDD.iterator(RDD.scala:270)org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)位于org.apache.spark.rdd.MapPartitionsRDD.compute的org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)atg.apache.spark.rdd.RDD.iterator(RDD.scala:270) (MapPartitionsRDD.scala:38)位于org.apache.spark的org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306).rdd.RDD.iterator(RDD.scala:270)atg.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) org.apache.spark.rdd.RDD.iterator(RDD.scala:270)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)at org.apache.spark.scheduler.Task.run (Task.scala:89)位于java.util的java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)的org.apache.spark.executor.Executor $ TaskRunner.run(Executor.scala:214). concurrent.ThreadPoolExecutor $ Worker.run(ThreadPoolExecutor.java:615)at java.lang.Thread.run(Thread.java:745)
16/07/25 16:33:59 ERROR TaskSetManager:阶段17.0中的任务0失败了4次; aborting job 16/07/25 16:33:59错误RBackendHandler:org.apache.spark.sql.api.r.SQLUtils上的dfToCols失败invokeJava中的错误(isStatic = TRUE,className,methodName,...):org. apache.spark.SparkException:作业因阶段失败而中止:阶段17.0中的任务0失败4次,最近失败:阶段17.0中丢失任务0.3(TID 86,wlos06.nrm.minn.seagate.com):java.net .SocketTimeoutException:在Java.net.Server.Socket.implAccept(ServerSocket.java:530)的java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)的java.net.PlainSocketImpl.socketAccept(Native …