火花中是否有"解释RDD"

Question

特别是,如果我说

rdd3 = rdd1.join(rdd2)

然后当我打电话时rdd3.collect,根据Partitioner使用情况,要么在节点分区之间移动数据,要么在每个分区上本地完成连接(或者,就我所知,完全不同的东西).这取决于RDD论文所谓的"狭义"和"广泛"依赖关系,但谁知道优化器在实践中有多好.

无论如何,我可以从跟踪输出中收集实际发生的事情,但是打电话会很好rdd3.explain.

这样的事情存在吗？

Answer 1

我认为toDebugString会安抚你的好奇心.

scala> val data = sc.parallelize(List((1,2)))
data: org.apache.spark.rdd.RDD[(Int, Int)] = ParallelCollectionRDD[8] at parallelize at <console>:21

scala> val joinedData = data join data
joinedData: org.apache.spark.rdd.RDD[(Int, (Int, Int))] = MapPartitionsRDD[11] at join at <console>:23

scala> joinedData.toDebugString
res4: String =
(8) MapPartitionsRDD[11] at join at <console>:23 []
 |  MapPartitionsRDD[10] at join at <console>:23 []
 |  CoGroupedRDD[9] at join at <console>:23 []
 +-(8) ParallelCollectionRDD[8] at parallelize at <console>:21 []
 +-(8) ParallelCollectionRDD[8] at parallelize at <console>:21 []

每个缩进都是一个阶段,所以这应该分为两个阶段.

此外,优化器相当不错,但我建议使用DataFrames如果您使用1.3+作为优化器,在许多情况下甚至更好:)