基于列值迭代Spark Dataframe

Question

我在Spark中有一个带有以下数据的数据框

{ID:"1",CNT:"2", Age:"21", Class:"3"}   
{ID:"2",CNT:"3", Age:"24", Class:"5"}

我想基于CNT值迭代数据框并生成如下输出:

{ID:"1",CNT:"1", Age:"21", Class:"3"}  
{ID:"1",CNT:"2", Age:"21", Class:"3"}  
{ID:"2",CNT:"1", Age:"24", Class:"5"}  
{ID:"2",CNT:"2", Age:"24", Class:"5"}  
{ID:"2",CNT:"3", Age:"24", Class:"5"}

有人可以知道如何实现这一目标.

Answer 1

您可以将数据框转换为rdd,用于flatMap展开它,然后将其转换回数据框:

val df = Seq((1,2,21,3),(2,3,24,5)).toDF("ID", "CNT", "Age", "Class")

case class Person(ID: Int, CNT: Int, Age: Int, Class: Int)

df.as[Person].rdd.flatMap(p => (1 to p.CNT).map(Person(p.ID, _, p.Age, p.Class))).toDF.show
+---+---+---+-----+
| ID|CNT|Age|Class|
+---+---+---+-----+
|  1|  1| 21|    3|
|  1|  2| 21|    3|
|  2|  1| 24|    5|
|  2|  2| 24|    5|
|  2|  3| 24|    5|
+---+---+---+-----+