使用.toArray()作为Spark矢量后应该是什么类型？

Question

我想将我的矢量转移到数组,所以我使用

get_array = udf(lambda x: x.toArray(),ArrayType(DoubleType()))
result3 = result2.withColumn('list',get_array('features'))
result3.show()

列features是矢量dtype.但是Spark告诉我

 net.razorvine.pickle.PickleException: expected zero arguments for construction of ClassDict (for numpy.core.multiarray._reconstruct)

我知道原因必须是我在UDF中使用的类型,所以我尝试了get_array = udf(lambda x: x.toArray(),ArrayType(FloatType())),这也无法工作.我知道它在传输后是numpy.narray,但我怎样才能正确显示？

以下是我获取数据帧结果的代码:

df4 = indexed.groupBy('uuid').pivot('name').sum('fre')
df4 = df4.fillna(0)
from pyspark.ml.feature import VectorAssembler 
assembler = VectorAssembler(
    inputCols=df4.columns[1:],
    outputCol="features")
dataset = assembler.transform(df4)
bk = BisectingKMeans(k=8,  seed=2, featuresCol="features")
result2 = bk.fit(dataset).transform(dataset)

这是索引的样子:

+------------------+------------+---------+-------------+------------+----------+--------+----+
|              uuid|    category|     code|   servertime|         cat|       fre|catIndex|name|
+------------------+------------+---------+-------------+------------+----------+--------+----+
|   351667085527886|         398|     null|1503084585000|         398|0.37951264|     2.0|  a2|
|   352279079643619|         403|     null|1503105476000|         403| 0.3938634|     3.0|  a3|
|   352279071621894|         398|     null|1503085396000|         398|0.38005984|     2.0|  a2|
|   357653074851887|         398|     null|1503085552000|         398| 0.3801652|     2.0|  a2|
|   354287077780760|         407|     null|1503085603000|         407|0.38019964|     5.0|  a5|
|0_8f394ebf3f67597c|         403|     null|1503084183000|         403|0.37924168|     3.0|  a3|
|   353528084062994|         403|     null|1503084234000|         403|0.37927604|     3.0|  a3|
|   356626072993852|   100000504|100000504|1503104781000|   100000504| 0.3933774|     0.0|  a0|
|   351667081062615|   100000448|      398|1503083901000|         398|0.37905172|     2.0|  a2|
|   354330089551058|1.00000444E8|     null|1503084004000|1.00000444E8|0.37912107|    34.0| a34|
+------------------+------------+---------+-------------+------------+----------+--------+----+

在result2,我有一些类型的列double,然后我使用VectorAssembler汇编这些双列到一个向量features,这是我想要传输到数组的列.

Answer 1

NumPy的类型不支持的返回值的UserDefinedFunctions.您必须将输出转换为标准Python list:

udf(lambda x: x.toArray().tolist(), ArrayType(DoubleType()))