那些遇到过的问题

Apache Spark：错误执行器 -> 迭代器

mir*_*iro 3 apache-spark

我正在尝试从这里运行示例（Learning Spark book），但出现以下错误：

16/10/07 01:15:26 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.AbstractMethodError: com.oreilly.learningsparkexamples.mini.java.WordCount$1.call(Ljava/lang/Object;)Ljava/util/Iterator;
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:124)
Run Code Online (Sandbox Code Playgroud)

我开始工作： $SPARK_HOME/bin/spark-submit --class com.oreilly.learningsparkexamples.mini.java.WordCount ./target/learning-spark-mini-example-0.0.1.jar ./README.md ./wordcounts

请想法，为什么会发生这种情况？

完整日志：

mini-complete-example$ $SPARK_HOME/bin/spark-submit --class com.oreilly.learningsparkexamples.mini.java.WordCount ./target/learning-spark-mini-example-0.0.1.jar ./README.md ./wordcounts
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/10/07 01:15:23 INFO SparkContext: Running Spark version 2.0.0
16/10/07 01:15:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/10/07 01:15:24 INFO SecurityManager: Changing view acls to: eDS
16/10/07 01:15:24 INFO SecurityManager: Changing modify acls to: eDS
16/10/07 01:15:24 INFO SecurityManager: Changing view acls groups to: 
16/10/07 01:15:24 INFO SecurityManager: Changing modify acls groups to: 
16/10/07 01:15:24 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(eDS); groups with view permissions: Set(); users  with modify permissions: Set(eDS); groups with modify permissions: Set()
16/10/07 01:15:24 INFO Utils: Successfully started service 'sparkDriver' on port 63851.
16/10/07 01:15:24 INFO SparkEnv: Registering MapOutputTracker
16/10/07 01:15:24 INFO SparkEnv: Registering BlockManagerMaster
16/10/07 01:15:24 INFO DiskBlockManager: Created local directory at /private/var/folders/yw/zl5hc321387g3sz2fg3l01980000gq/T/blockmgr-0fb2af5a-8662-4d78-88c8-8e0608f35ff3
16/10/07 01:15:24 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
16/10/07 01:15:24 INFO SparkEnv: Registering OutputCommitCoordinator
16/10/07 01:15:24 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/10/07 01:15:24 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.1.17:4040
16/10/07 01:15:24 INFO SparkContext: Added JAR file:/Users/eDS/dev/learning-spark/mini-complete-example/./target/learning-spark-mini-example-0.0.1.jar at spark://192.168.1.17:63851/jars/learning-spark-mini-example-0.0.1.jar with timestamp 1475795724857
16/10/07 01:15:24 INFO Executor: Starting executor ID driver on host localhost
16/10/07 01:15:24 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 63852.
16/10/07 01:15:24 INFO NettyBlockTransferService: Server created on 192.168.1.17:63852
16/10/07 01:15:24 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.1.17, 63852)
16/10/07 01:15:24 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.17:63852 with 366.3 MB RAM, BlockManagerId(driver, 192.168.1.17, 63852)
16/10/07 01:15:24 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.1.17, 63852)
16/10/07 01:15:25 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 145.5 KB, free 366.2 MB)
16/10/07 01:15:25 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 16.3 KB, free 366.1 MB)
16/10/07 01:15:25 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.1.17:63852 (size: 16.3 KB, free: 366.3 MB)
16/10/07 01:15:25 INFO SparkContext: Created broadcast 0 from textFile at WordCount.java:31
16/10/07 01:15:25 INFO FileInputFormat: Total input paths to process : 1
16/10/07 01:15:25 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
16/10/07 01:15:25 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
16/10/07 01:15:25 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
16/10/07 01:15:25 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
16/10/07 01:15:25 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
16/10/07 01:15:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
16/10/07 01:15:25 INFO SparkContext: Starting job: saveAsTextFile at WordCount.java:46
16/10/07 01:15:25 INFO DAGScheduler: Registering RDD 3 (mapToPair at WordCount.java:39)
16/10/07 01:15:25 INFO DAGScheduler: Got job 0 (saveAsTextFile at WordCount.java:46) with 2 output partitions
16/10/07 01:15:25 INFO DAGScheduler: Final stage: ResultStage 1 (saveAsTextFile at WordCount.java:46)
16/10/07 01:15:25 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
16/10/07 01:15:25 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
16/10/07 01:15:25 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at mapToPair at WordCount.java:39), which has no missing parents
16/10/07 01:15:25 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.9 KB, free 366.1 MB)
16/10/07 01:15:25 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.7 KB, free 366.1 MB)
16/10/07 01:15:25 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.1.17:63852 (size: 2.7 KB, free: 366.3 MB)
16/10/07 01:15:25 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1012
16/10/07 01:15:25 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at mapToPair at WordCount.java:39)
16/10/07 01:15:25 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
16/10/07 01:15:25 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0, PROCESS_LOCAL, 5479 bytes)
16/10/07 01:15:25 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1, PROCESS_LOCAL, 5479 bytes)
16/10/07 01:15:26 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
16/10/07 01:15:26 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
16/10/07 01:15:26 INFO Executor: Fetching spark://192.168.1.17:63851/jars/learning-spark-mini-example-0.0.1.jar with timestamp 1475795724857
16/10/07 01:15:26 INFO TransportClientFactory: Successfully created connection to /192.168.1.17:63851 after 73 ms (0 ms spent in bootstraps)
16/10/07 01:15:26 INFO Utils: Fetching spark://192.168.1.17:63851/jars/learning-spark-mini-example-0.0.1.jar to /private/var/folders/yw/zl5hc321387g3sz2fg3l01980000gq/T/spark-5adda737-293c-483e-bdbe-f8fa7c171211/userFiles-2a2cf77d-5794-4006-a125-5df94550cbf8/fetchFileTemp6160431711224595115.tmp
16/10/07 01:15:26 INFO Executor: Adding file:/private/var/folders/yw/zl5hc321387g3sz2fg3l01980000gq/T/spark-5adda737-293c-483e-bdbe-f8fa7c171211/userFiles-2a2cf77d-5794-4006-a125-5df94550cbf8/learning-spark-mini-example-0.0.1.jar to class loader
16/10/07 01:15:26 INFO HadoopRDD: Input split: file:/Users/eDS/dev/learning-spark/mini-complete-example/README.md:66+66
16/10/07 01:15:26 INFO HadoopRDD: Input split: file:/Users/eDS/dev/learning-spark/mini-complete-example/README.md:0+66
16/10/07 01:15:26 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.AbstractMethodError: com.oreilly.learningsparkexamples.mini.java.WordCount$1.call(Ljava/lang/Object;)Ljava/util/Iterator;
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:124)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:124)
    at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
    at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:192)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
    at org.apache.spark.scheduler.Task.run(Task.scala:85)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
16/10/07 01:15:26 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 1385 bytes result sent to driver
16/10/07 01:15:26 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 307 ms on localhost (1/2)
16/10/07 01:15:26 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-0,5,main]
java.lang.AbstractMethodError: com.oreilly.learningsparkexamples.mini.java.WordCount$1.call(Ljava/lang/Object;)Ljava/util/Iterator;
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:124)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:124)
    at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
    at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:192)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
    at org.apache.spark.scheduler.Task.run(Task.scala:85)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
16/10/07 01:15:26 INFO SparkContext: Invoking stop() from shutdown hook
16/10/07 01:15:26 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.AbstractMethodError: com.oreilly.learningsparkexamples.mini.java.WordCount$1.call(Ljava/lang/Object;)Ljava/util/Iterator;
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:124)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:124)
    at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
    at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:192)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
    at org.apache.spark.scheduler.Task.run(Task.scala:85)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

16/10/07 01:15:26 ERROR TaskSetManager: Task 0 in stage 0.0 failed 1 times; aborting job
16/10/07 01:15:26 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
16/10/07 01:15:26 INFO TaskSchedulerImpl: Cancelling stage 0
16/10/07 01:15:26 INFO SparkUI: Stopped Spark web UI at http://192.168.1.17:4040
16/10/07 01:15:26 INFO DAGScheduler: ShuffleMapStage 0 (mapToPair at WordCount.java:39) failed in 0.364 s
16/10/07 01:15:26 INFO DAGScheduler: Job 0 failed: saveAsTextFile at WordCount.java:46, took 0.448155 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.AbstractMethodError: com.oreilly.learningsparkexamples.mini.java.WordCount$1.call(Ljava/lang/Object;)Ljava/util/Iterator;
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:124)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:124)
    at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
    at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:192)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
    at org.apache.spark.scheduler.Task.run(Task.scala:85)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1450)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1438)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1437)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1437)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
    at scala.Option.foreach(Option.scala:257)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1659)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1618)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1607)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:632)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1871)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1884)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1904)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1219)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply(PairRDDFunctions.scala:1161)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply(PairRDDFunctions.scala:1161)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:358)
    at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:1161)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply$mcV$sp(PairRDDFunctions.scala:1064)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1030)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1030)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:358)
    at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:1030)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply$mcV$sp(PairRDDFunctions.scala:956)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:956)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:956)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:358)
    at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:955)
    at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply$mcV$sp(RDD.scala:1440)
    at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1419)
    at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1419)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:358)
    at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1419)
    at org.apache.spark.api.java.JavaRDDLike$class.saveAsTextFile(JavaRDDLike.scala:549)
    at org.apache.spark.api.java.AbstractJavaRDDLike.saveAsTextFile(JavaRDDLike.scala:45)
    at com.oreilly.learningsparkexamples.mini.java.WordCount.main(WordCount.java:46)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.AbstractMethodError: com.oreilly.learningsparkexamples.mini.java.WordCount$1.call(Ljava/lang/Object;)Ljava/util/Iterator;
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:124)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:124)
    at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
    at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:192)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
    at org.apache.spark.scheduler.Task.run(Task.scala:85)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
16/10/07 01:15:26 ERROR LiveLis

小智 5

我得到了同样的错误：

错误执行器：阶段 0.0 中任务 0.0 中的异常（TID 0）

和

java.lang.AbstractMethodError: ...


首先，检查以下路径中的火花版本：

%SPARK_HOME%/jars/spark-core_x.xx-yyyjar

并修改pom.xml文件如下所示：

<dependency> <!-- Spark dependency -->
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_x.xx</artifactId>
    <version>y.y.y</version>
    <scope>provided</scope>
</dependency>
Run Code Online (Sandbox Code Playgroud)

希望对像我这样的状态有所帮助。

归档时间：

查看次数：

3212 次

最近记录：

8 年，7 月 前
相关归档
Apache Spark Web UI中"Stage Skipped"的含义是什么？ 79
通过检查值是否在列表中以及其他条件来过滤Spark DataFrame 22
如何在 pyspark 中查找数据帧的大小（以 MB 为单位）？ 18
如何使用 Array[Int] 将 csv 文件加载到 Spark DataFrame 中 5
从 Scala 中的自定义日期返回 x 天的日期 4
如何有条件地从列中删除前两个字符 4
使用 AWSglue 合并多个 parquet 文件并在 s3 中创建更大的 parquet 文件 4
内存不足错误构建spark时出错 2
在火花结构化流中动态更改 hdfs 写入路径 1
PySpark - 从数据框中删除第一行 1
难疑归档
jQuery是否存在"存在"功能？ 2669
HTML中id属性的有效值是什么？ 1945
什么是控制倒置？ 1704
C#在foreach中重用变量是否有原因？ 1631
如何在Python中追加文件？ 1446
使用*args和**kwargs 1367
我应该如何道德地接近用户密码存储以便以后的明文检索？ 1346
如何从其他线程更新GUI？ 1331
在jQuery中删除事件处理程序的最佳方法？ 1038
如何进行HTTP POST Web请求 1033