Pyspark - 工人和驱动程序中的不同 Python 版本

Question

我正在 python3 环境中使用 pyspark。我有一个数据框，我正在尝试将一列密集 vectos 拆分为多个列值。我的 df 是这样的：

df_vector = kmeansModel_2.transform(finalData).select(['scalaredFeatures', 
                                                       'prediction'])
df_vector.show()

+--------------------+----------+
|    scalaredFeatures|prediction|
+--------------------+----------+
|[0.56785108466505...|         0|
|[1.41962771166263...|         0|
|[2.20042295307707...|         0|
|[0.14196277116626...|         0|
|[1.41962771166263...|         0|
+-------------------------------+

好吧，为了完成我的任务，我使用了以下代码：

def extract(row):
    return (row.prediction, ) + tuple(row.scalaredFeatures.toArray().tolist())

df = df_vector.rdd.map(extract)toDF(["prediction"])

不幸的是，我收到一个错误：

Py4JJavaError: An error occurred while calling 
z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: 
Task 0 in stage 52.0 failed 1 times, most recent failure: Lost task 
0.0 in stage 52.0 (TID 434, localhost, executor driver): 
org.apache.spark.api.python.PythonException: Traceback (most recent 
call last):
 File "pyspark/worker.py", line 123, in main
("%d.%d" % sys.version_info[:2], version))
Exception: Python in worker has different version 2.7 than that in 
driver 3.6, PySpark cannot run with different minor versions.Please 
check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON 
are correctly set.

有没有人可以帮助我完成这项任务？谢谢！

Answer 1

如果您使用 PyCharm，您可以添加 PYSPARK_PYTHON 和 PYSPARK_DRIVER_PYTHON 来运行/调试配置。