Py4JJavaError：调用 o1670.collectToPython 时发生错误

Question

我正在尝试将 Spark RDD 转换为 Pandas DataFrame。

\n\n

我使用 csv 文件作为示例。该文件有 10 以下是前 3 行：

\n\n

“可堆叠储物架的埃尔登底座，铂金”，穆罕默德·麦金泰尔，3，-213.25,38.94,35，努勒维特，存储和组织，0.8

\n\n

“1.7 立方英尺紧凑型“立方体”办公冰箱”，Barry French，293,457.81,208.16,68.02，努勒维特，电器，0.58

\n\n

“Cardinal Slant-D\xef\xbf\xbd 环形活页夹，重型乙烯基”，Barry French，293,46.71,8.69,2.99，努勒维特，活页夹和活页夹配件，0.39

\n\n

我的代码在这里：

\n\n

import pandas as pd\nimport pyspark\nfrom pyspark.sql import SparkSession\n\nspark = SparkSession.builder.appName("HelloWorld").getOrCreate()\nsc = spark.sparkContext\n\n\nfrom pyspark.sql.types import StructType\nfrom pyspark.sql.types import StructField\nfrom pyspark.sql.types import StringType\nfrom pyspark.sql.context import SQLContext\n\nschema = StructType([StructField(str(i), StringType(), True) for i in range(10)])\n\ntext = sc.textFile(\'data_53000kb.csv\')\ntext = text.map(lambda x: [c.strip() for c in x.split(\',\')])\ndf = spark.createDataFrame(text, schema)\ndf.toPandas()\n

\n\n

此时我收到以下错误：

\n\n

Py4JJavaError: An error occurred while calling o1670.collectToPython.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 40.0 failed 1 times, most recent failure: Lost task 0.0 in stage 40.0 (TID 72, localhost, executor driver): java.net.SocketException: Connection reset by peer: socket write error\n    at java.net.SocketOutputStream.socketWrite0(Native Method)\n    at java.net.SocketOutputStream.socketWrite(Unknown Source)\n    at java.net.SocketOutputStream.write(Unknown Source)\n    at java.io.BufferedOutputStream.flushBuffer(Unknown Source)\n    at java.io.BufferedOutputStream.write(Unknown Source)\n    at java.io.DataOutputStream.write(Unknown Source)\n    at java.io.FilterOutputStream.write(Unknown Source)\n    at org.apache.spark.api.python.PythonRDD$.writeUTF(PythonRDD.scala:394)\n    at org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:214)\n    at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:224)\n    at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:224)\n    at scala.collection.Iterator$class.foreach(Iterator.scala:891)\n    at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)\n    at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)\n    at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:561)\n    at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:346)\n    at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)\n    at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:195)\n\nDriver stacktrace:\n    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1891)\n    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1879)\n    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1878)\n    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1878)\n    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:927)\n    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:927)\n    at scala.Option.foreach(Option.scala:257)\n    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:927)\n    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2112)\n    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2061)\n    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2050)\n    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)\n    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:738)\n    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)\n    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)\n    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)\n    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)\n    at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:990)\n    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\n    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)\n    at org.apache.spark.rdd.RDD.withScope(RDD.scala:385)\n    at org.apache.spark.rdd.RDD.collect(RDD.scala:989)\n    at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:299)\n    at org.apache.spark.sql.Dataset$$anonfun$collectToPython$1.apply(Dataset.scala:3263)\n    at org.apache.spark.sql.Dataset$$anonfun$collectToPython$1.apply(Dataset.scala:3260)\n    at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3370)\n    at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)\n    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)\n    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)\n    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3369)\n    at org.apache.spark.sql.Dataset.collectToPython(Dataset.scala:3260)\n    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)\n    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n    at java.lang.reflect.Method.invoke(Unknown Source)\n    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n    at py4j.Gateway.invoke(Gateway.java:282)\n    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n    at py4j.commands.CallCommand.execute(CallCommand.java:79)\n    at py4j.GatewayConnection.run(GatewayConnection.java:238)\n    at java.lang.Thread.run(Unknown Source)\nCaused by: java.net.SocketException: Connection reset by peer: socket write error\n    at java.net.SocketOutputStream.socketWrite0(Native Method)\n    at java.net.SocketOutputStream.socketWrite(Unknown Source)\n    at java.net.SocketOutputStream.write(Unknown Source)\n    at java.io.BufferedOutputStream.flushBuffer(Unknown Source)\n    at java.io.BufferedOutputStream.write(Unknown Source)\n    at java.io.DataOutputStream.write(Unknown Source)\n    at java.io.FilterOutputStream.write(Unknown Source)\n    at org.apache.spark.api.python.PythonRDD$.writeUTF(PythonRDD.scala:394)\n    at org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:214)\n    at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:224)\n    at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:224)\n    at scala.collection.Iterator$class.foreach(Iterator.scala:891)\n    at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)\n    at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)\n    at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:561)\n    at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:346)\n    at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)\n    at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:195)\n

\n\n

我现在能做什么？

\n

Answer 1

df.toPandas() 将所有数据收集到驱动程序节点，因此这是非常昂贵的操作。还有一个名为 maxResultSize 的 Spark 属性

Spark.driver.maxResultSize（默认1G）-->每个Spark操作（例如收集）的所有分区的序列化结果总大小的限制（以字节为单位）。至少应为 1M，或 0 表示无限制。如果总大小超过此限制，作业将被中止。限制过高可能会导致驱动程序内存不足错误（取决于spark.driver.memory和JVM中对象的内存开销）。设置适当的限制可以保护驱动程序免受内存不足错误的影响。

如果估计的数据大小大于 maxResultSize 给定作业将被中止。这里的目标是保护您的应用程序免受驱动程序丢失的影响，仅此而已。

您可能需要增加 maxResultSize