Spark将列组合为嵌套数组

Question

如何将spark中的列组合为嵌套数组？

val inputSmall = Seq(
    ("A", 0.3, "B", 0.25),
    ("A", 0.3, "g", 0.4),
    ("d", 0.0, "f", 0.1),
    ("d", 0.0, "d", 0.7),
    ("A", 0.3, "d", 0.7),
    ("d", 0.0, "g", 0.4),
    ("c", 0.2, "B", 0.25)).toDF("column1", "transformedCol1", "column2", "transformedCol2")

类似的东西

+-------+---------------+---------------+------- +
|column1|transformedCol1|transformedCol2|combined|
+-------+---------------+---------------+------ -+
|      A|            0.3|            0.3[0.3, 0.3]|
+-------+---------------+---------------+-------+

Answer 1

如果要将多个列组合到ArrayType的新列中,可以使用以下array函数:

import org.apache.spark.sql.functions._
val result = inputSmall.withColumn("combined", array($"transformedCol1", $"transformedCol2"))
result.show()

+-------+---------------+-------+---------------+-----------+
|column1|transformedCol1|column2|transformedCol2|   combined|
+-------+---------------+-------+---------------+-----------+
|      A|            0.3|      B|           0.25|[0.3, 0.25]|
|      A|            0.3|      g|            0.4| [0.3, 0.4]|
|      d|            0.0|      f|            0.1| [0.0, 0.1]|
|      d|            0.0|      d|            0.7| [0.0, 0.7]|
|      A|            0.3|      d|            0.7| [0.3, 0.7]|
|      d|            0.0|      g|            0.4| [0.0, 0.4]|
|      c|            0.2|      B|           0.25|[0.2, 0.25]|
+-------+---------------+-------+---------------+-----------+