在火花scala中展平元素

Question

我在RDD中有这样的数据:

RDD[((Int, Int, Int), ((Int, Int), Int))]

如:

(((9,679,16),((2,274),1)), ((250,976,13),((2,218),1)))

我希望输出为:

((9,679,16,2,274,1),(250,976,13,2,218,1))

加入2 rdds后:

val joinSale = salesTwo.join(saleFinal)

我得到了结果集.我尝试了以下代码.

joinSale.flatMap(x => x).take(100).foreach(println)

我试过map/flatMap但是做不到.任何想法如何实现这样的场景？提前致谢 ..

Answer 1

您可以使用scala中的模式匹配来完成此操作.只需将您的元组修改逻辑包装在类似于下面的地图中:

val mappedJoinSale = joinSale.map { case ((a, b, c), ((d, e), f)) => (a, b, c, d, e, f) }

使用您的示例,我们有:

scala> val example = sc.parallelize(Array(((9,679,16),((2,274),1)), ((250,976,13),((2,218),1))))
example: org.apache.spark.rdd.RDD[((Int, Int, Int), ((Int, Int), Int))] = ParallelCollectionRDD[0] at parallelize at <console>:12

scala> val mapped = example.map { case ((a, b, c), ((d, e), f)) => (a, b, c, d, e, f) }
mapped: org.apache.spark.rdd.RDD[(Int, Int, Int, Int, Int, Int)] = MappedRDD[1] at map at <console>:14

scala> mapped.take(2).foreach(println)
...
(9,679,16,2,274,1)
(250,976,13,2,218,1)