小编A k*_*ram的帖子

我如何只过滤非空数组

import  org.apache.spark.sql.types.ArrayType

  val arrayFields = secondDF.schema.filter(st => st.dataType.isInstanceOf[ArrayType])
  val names = arrayFields.map(_.name)

或者是这个代码

val DF1=DF.select(col("key"),explode(col("objectiveAttachment")).as("collection")).select(col("collection.*"),col("key"))

|-- objectiveAttachment: array (nullable = true) 
 | |-- element: string (containsNull = true) 

我收到这个错误

 org.apache.spark.sql.AnalysisException: Can only star expand struct data types. Attribute: ArrayBuffer(collection);

任何帮助表示赞赏。

如何将数组展平为包含列 [a,b,c,d,e] 的数据帧

root
 |-- arry: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- a string (nullable = true)
 |    |    |-- b: long (nullable = true)
 |    |    |-- c: string (nullable = true)
 |    |    |-- d: string (nullable = true)
 |    |    |-- e: long (nullable = true)

任何帮助表示赞赏。