仅过滤非空数组数据帧火花

Question

我如何只过滤非空数组

import  org.apache.spark.sql.types.ArrayType

  val arrayFields = secondDF.schema.filter(st => st.dataType.isInstanceOf[ArrayType])
  val names = arrayFields.map(_.name)

或者是这个代码

val DF1=DF.select(col("key"),explode(col("objectiveAttachment")).as("collection")).select(col("collection.*"),col("key"))

|-- objectiveAttachment: array (nullable = true) 
 | |-- element: string (containsNull = true) 

我收到这个错误

 org.apache.spark.sql.AnalysisException: Can only star expand struct data types. Attribute: ArrayBuffer(collection);

任何帮助表示赞赏。

Answer 1

使用函数大小

import org.apache.spark.sql.functions._

secondDF.filter(size($"objectiveAttachment") > 0)