我在spark Dataframe中有一个"StructType"列,它有一个数组和一个字符串作为子字段.我想修改数组并返回相同类型的新列.我可以用UDF处理它吗？或者有哪些替代方案？
import org.apache.spark.sql.types._
import org.apache.spark.sql.Row
val sub_schema = StructType(StructField("col1",ArrayType(IntegerType,false),true) :: StructField("col2",StringType,true)::Nil)
val schema = StructType(StructField("subtable", sub_schema,true) :: Nil)
val data = Seq(Row(Row(Array(1,2),"eb")), Row(Row(Array(3,2,1), "dsf")) )
val rd = sc.parallelize(data)
val df = spark.createDataFrame(rd, schema)
df.printSchema
root
|-- subtable: struct (nullable = true)
| |-- col1: array (nullable = true)
| | |-- element: integer (containsNull = false)
| |-- col2: string (nullable = true)
看来我需要一个类型为Row的UDF
val u = udf((x:Row) => x)
>> Schema for type org.apache.spark.sql.Row is not supported …