如何在Scala中合并三个DataFrame

Question

如何在Spark-Scala中合并3个DataFrame？我完全没有任何想法我怎么能做到这一点.在stackOverFlow上我找不到类似的例子.

我有3个类似的DataFrame.Column的名称和它们的编号相同.差异只是行的值.

DataFrame1:

+----+------+----+---+
|type| Model|Name|ID |
+----+------+----+---+
|  1 |wdasd |xyzd|111|
|  1 |wd    |zdfd|112|
|  1 |bdp   |2gfs|113|
+----+------+----+---+

DataFrame2:

+----+------+----+---+
|type| Model|Name|ID |
+----+------+----+---+
|  2 |wdasd |xyzd|221|
|  2 |wd    |zdfd|222|
|  2 |bdp   |2gfs|223|
+----+------+----+---+

DataFrame3:

+----+------+----+---+
|type| Model|Name|ID |
+----+------+----+---+
|  3 |AAAA  |N_AM|331|
|  3 |BBBB  |NA_M|332|
|  3 |CCCC  |MA_N|333|
+----+------+----+---+

我想要这种类型的DataFrame

MergeDataFrame:

+----+------+----+---+
|type| Model|Name|ID |
+----+------+----+---+
|  1 |wdasd |xyzd|111|
|  1 |wd    |zdfd|112|
|  1 |bdp   |2gfs|113|
|  2 |wdasd |xyzd|221|
|  2 |wd    |zdfd|222|
|  2 |bdp   |2gfs|223|
|  3 |AAAA  |N_AM|331|
|  3 |BBBB  |NA_M|332|
|  3 |CCCC  |MA_N|333|
+----+------+----+---+

Answer 1

Spark提供了一个union和unionAll.看起来他们正在弃用该unionAll函数,所以我将使用如下union函数:

dataFrame1.union(dataFrame2).union(dataFrame3)

请注意,为了合并数据帧,数据帧必须具有完全相同的完全相同的列名.

请参阅此处的spark文档