小编use*_*571的帖子

在独立集群上运行 spark shell 时出现太大的帧错误

我在针对本地运行的 Spark 独立集群启动 spark shell 时遇到问题。有任何想法吗?我在 spark 3.1.0-SNAPSHOT 上运行它。

在本地模式下启动 shell 或常规应用程序工作正常,但都失败并显示以下命令。

$ ./bin/spark-shell --master spark://localhost:8080
...
20/04/05 00:34:47 WARN StandaloneAppClient$ClientEndpoint: Could not connect to localhost:8080: java.lang.IllegalArgumentException: Too large frame: 5211883372140375593
20/04/05 00:34:47 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master localhost:8080
org.apache.spark.SparkException: Exception thrown in awaitResult: 
        at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:303)
        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
        at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
        at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)
        at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anon$1.run(StandaloneAppClient.scala:106)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Too large frame: 5211883372140375593
        at org.sparkproject.guava.base.Preconditions.checkArgument(Preconditions.java:119)
        at org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:148)
        at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:98)
        at …
Run Code Online (Sandbox Code Playgroud)

apache-spark

5
推荐指数
1
解决办法
1436
查看次数

标签 统计

apache-spark ×1