小编Pie*_*ito的帖子

Spark kryo_serializers 和 Broadcast<Map<Object, Iterable<GowallaDataLocation>>> java.io.IOException: java.lang.UnsupportedOperationException

当我尝试访问 Broadcast 变量时,我收到此异常:

17/03/26 03:04:23 警告 TaskSetManager:在 3.0 阶段丢失任务 0.0(TID 10,192.168.56.5,执行程序 1):java.io.IOException:org.apache.spark.util 处的 java.lang.UnsupportedOperationException .Utils$.tryOrIOException(Utils.scala:1276) 在 org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:206) 在 org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:206) 66) 在 org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:66) 在 org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:96) 在 GowallaTask$2.call(GowallaTask.java: 214) 在 org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1.apply(JavaRDDLike.scala:351) 在 org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1.apply( JavaRDDLlike.scala:351) 在 scala.collection.Iterator$class.foreach(Ite​​rator.scala:893) 在 org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28) 在 org.apache.spark.rdd.RDD$$anonfun$ foreach$1$$anonfun$apply$28.apply(RDD.scala:917) 在 org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:917) 在 org .apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944) 在 org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944) 在 org.apache.spark .scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala) :282) 在 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 在 java.util.concurrent。ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 在 java.lang.Thread.run(Thread.java:745)

引起:在 com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:162) 在 com.esotericsoftware.kryo.serializers 的 java.util.AbstractMap.put(AbstractMap.java:209) 的 java.lang.UnsupportedOperationException .MapSerializer.read(MapSerializer.java:39) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790) at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:244) at org. apache.spark.broadcast.TorrentBroadcast$$anonfun$10.apply(TorrentBroadcast.scala:286) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1303) at org.apache.spark.broadcast.TorrentBroadcast …

java kryo apache-spark

3
推荐指数
1
解决办法
2022
查看次数

标签 统计

apache-spark ×1

java ×1

kryo ×1