小编rav*_*a k的帖子

皮斯帕克。Spark.SparkException:作业因阶段失败而中止:阶段 15.0 中的任务 0 失败 1 次,java.net.SocketException:连接重置

我是 pyspark 的新手,我正在尝试使用 pyspark 在 Prophet 中运行多个时间序列(作为分布式计算,因为我有 100 个时间序列需要预测),但我有如下错误。

import time 
start_time = time.time()
sdf = spark.createDataFrame(data)
print('%0.2f min: Lags' % ((time.time() - start_time) / 60))
sdf.createOrReplaceTempView('Quantity')
spark.sql("select Reseller_City, Business_Unit, count(*) from Quantity group by Reseller_City, Business_Unit order by Reseller_City, Business_Unit").show()
query = 'SELECT Reseller_City, Business_Unit, conditions, black_week, promos, Sales_Date as ds, sum(Rslr_Sales_Quantity) as y FROM Quantity GROUP BY Reseller_City, Business_Unit, conditions, black_week, promos, ds ORDER BY Reseller_City, Business_Unit, ds'
spark.sql(query).show()
sdf.rdd.getNumPartitions()
store_part = (spark.sql(query).repartition(spark.sparkContext.defaultParallelism['Reseller_City','Business_Unit'])).cache()

store_part.explain()

from pyspark.sql.types import * …
Run Code Online (Sandbox Code Playgroud)

python apache-spark apache-spark-sql pyspark

4
推荐指数
1
解决办法
5万
查看次数

标签 统计

apache-spark ×1

apache-spark-sql ×1

pyspark ×1

python ×1