在 PySpark 中将 StringType 列转换为 ArrayType

Question

我有一个包含“EVENT_ID”列的数据框，其数据类型为字符串。我正在运行 FPGrowth 算法，但抛出以下错误

Py4JJavaError: An error occurred while calling o1711.fit. 
:java.lang.IllegalArgumentException: requirement failed: 
The input column must be array, but got string.

列 EVENT_ID 有值

E_34503_Probe
E_35203_In
E_31901_Cbc

我正在使用下面的代码将字符串列转换为数组类型

df2 = df.withColumn("EVENT_ID", df["EVENT_ID"].cast(types.ArrayType(types.StringType())))

但我收到以下错误

Py4JJavaError: An error occurred while calling o1874.withColumn.
: org.apache.spark.sql.AnalysisException: cannot resolve '`EVENT_ID`' due to data type mismatch: cannot cast string to array<string>;;

如何将此列转换为数组类型或使用字符串类型运行 FPGrowth 算法？

Answer 1

原答案

请尝试以下操作。

In  [0]: from pyspark.sql.types import StringType
         from pyspark.sql.functions import col, regexp_replace, split

In  [1]: df = spark.createDataFrame(["E_34503_Probe", "E_35203_In", "E_31901_Cbc"], StringType()).toDF("EVENT_ID")
         df.show()
Out [1]: +-------------+
         |     EVENT_ID|
         +-------------+
         |E_34503_Probe|
         |   E_35203_In|
         |  E_31901_Cbc|
         +-------------+

In  [2]: df_new = df.withColumn("EVENT_ID", split(regexp_replace(col("EVENT_ID"), r"(^\[)|(\]$)|(')", ""), ", "))
         df_new.printSchema()
Out [2]: root
          |-- EVENT_ID: array (nullable = true)
          |    |-- element: string (containsNull = true)

我希望它会有所帮助。

编辑答案

正如@pault 在他的评论中很好地指出的那样，更简单的解决方案如下：

In  [0]: from pyspark.sql.types import StringType
         from pyspark.sql.functions import array

In  [1]: df = spark.createDataFrame(["E_34503_Probe", "E_35203_In", "E_31901_Cbc"], StringType()).toDF("EVENT_ID")
         df.show()
Out [1]: +-------------+
         |     EVENT_ID|
         +-------------+
         |E_34503_Probe|
         |   E_35203_In|
         |  E_31901_Cbc|
         +-------------+

In  [2]: df_new = df.withColumn("EVENT_ID", array(df["EVENT_ID"]))
         df_new.printSchema()
Out [2]: root
           |-- EVENT_ID: array (nullable = false)
           |    |-- element: string (containsNull = true)