小编Hao*_*Hao的帖子

在 pyspark 中的 DataFrame 上使用 toPandas() 时出现神秘的“pyarrow.lib.ArrowInvalid:浮点值被截断”错误

我在不是很大的 DataFrame 上使用 toPandas() ,但出现以下异常:

18/10/31 19:13:19 ERROR Executor: Exception in task 127.2 in stage 13.0 (TID 2264)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
    File "/home/hadoop/spark2.3.1/python/lib/pyspark.zip/pyspark/worker.py", line 230, in main
      process()
    File "/home/hadoop/spark2.3.1/python/lib/pyspark.zip/pyspark/worker.py", line 225, in process
      serializer.dump_stream(func(split_index, iterator), outfile)
    File "/home/hadoop/spark2.3.1/python/lib/pyspark.zip/pyspark/serializers.py", line 261, in dump_stream
      batch = _create_batch(series, self._timezone)
    File "/home/hadoop/spark2.3.1/python/lib/pyspark.zip/pyspark/serializers.py", line 239, in _create_batch
      arrs = [create_array(s, t) for s, t in series]
    File "/home/hadoop/spark2.3.1/python/lib/pyspark.zip/pyspark/serializers.py", line 239, in <listcomp>
      arrs = [create_array(s, t) for s, t in series] …
Run Code Online (Sandbox Code Playgroud)

apache-spark apache-spark-sql pyspark apache-arrow pyarrow

5
推荐指数
2
解决办法
9683
查看次数