我得到了这个奇怪的例外.我在Hadoop 2.6.4上使用Spark 1.6.0并在YARN集群上提交Spark作业.
16/07/23 20:05:21 WARN hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block BP-532134798-128.110.152.143-1469321545728:blk_1073741865_1041
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2203)
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:867)
16/07/23 20:49:09 ERROR server.TransportRequestHandler: Error sending result RpcResponse{requestId=4719626006875125240, body=NioManagedBuffer{buf=java.nio.HeapByteBuffer[pos=0 lim=81 cap=81]}} to ms0440.utah.cloudlab.us/128.110.152.175:58944; closing connection
java.nio.channels.ClosedChannelException
在Hadoop 2.6.0运行时,我得到这个错误,并认为该例外可能是有点像一个错误的这个,但即使改变这对Hadoop的2.6.4后,我得到了同样的错误.没有任何内存问题,我的群集适用于HDFS和内存.我经历了这个和这个,但没有运气.
注意:1.我使用Apache Hadoop和Spark而不是任何CDH/HDP.2.我能够在HDFS中复制数据,甚至能够在该集群上执行另一个作业.
我正在尝试在集群中运行我的 JAR yarn-cluster，但一段时间后出现异常。INFO失败之前的最后一个是Uploading resource。我已经检查了所有安全组，成功hsdf ls但仍然收到错误。
./bin/spark-submit --class MyMainClass --master 纱线簇 /tmp/myjar-1.0.jar myjarparameter
16/01/21 16:13:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/01/21 16:13:52 INFO client.RMProxy: Connecting to ResourceManager at yarn.myserver.com/publicip:publicport
16/01/21 16:13:53 INFO yarn.Client: Requesting a new application from cluster with 10 NodeManagers
16/01/21 16:13:53 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (13312 MB per …