scala - Spark:如何在循环中结合所有数据帧

Question

有没有办法让数据帧联合数据帧在循环中？

这是一个示例代码

var fruits = List(
    "apple"
    ,"orange"
    ,"melon"

) 


for (x <- fruits){

           var df = Seq(("aaa","bbb",x)).toDF("aCol","bCol","name")

}

我想做一些像

aCol | bCol | fruitsName
aaa,bbb,apple
aaa,bbb,orange
aaa,bbb,melon

再次感谢

Answer 1

Steffen Schmitz的答案是我认为最简洁的答案.如果您正在寻找更多自定义(字段类型等),下面是一个更详细的答案:

import org.apache.spark.sql.types.{StructType, StructField, StringType}
import org.apache.spark.sql.Row

//initialize DF
val schema = StructType(
  StructField("aCol", StringType, true) ::
  StructField("bCol", StringType, true) ::
  StructField("name", StringType, true) :: Nil)
var initialDF = spark.createDataFrame(sc.emptyRDD[Row], schema)

//list to iterate through
var fruits = List(
    "apple"
    ,"orange"
    ,"melon"
)

for (x <- fruits) {
  //union returns a new dataset
  initialDF = initialDF.union(Seq(("aaa", "bbb", x)).toDF)
}

//initialDF.show()

引用:

• 如何使用指定的架构创建空DataFrame？
• https://spark.apache.org/docs/2.0.1/api/java/org/apache/spark/sql/Dataset.html
• https://docs.databricks.com/spark/latest/faq/append-a-row-to-rdd-or-dataframe.html

Answer 2

你可以创建一个DataFrames 序列然后使用reduce:

val results = fruits.
  map(fruit => Seq(("aaa", "bbb", fruit)).toDF("aCol","bCol","name")).
  reduce(_.union(_))

results.show()

Answer 3

如果您有不同/多个数据框，则可以使用下面的代码，这是有效的。

val newDFs = Seq(DF1,DF2,DF3)
newDFs.reduce(_ union _)

Answer 4

在for循环中:

val fruits = List("apple", "orange", "melon")

( for(f <- fruits) yield ("aaa", "bbb", f) ).toDF("aCol", "bCol", "name")