spark中的cache()会改变RDD的状态还是创建一个新的状态？

Question

这个问题是对前一个问题的跟进.如果我在Spark中缓存两次相同的RDD会发生什么.

在调用cache()RDD时,RDD的状态是否发生了变化(返回的RDD只是this为了易于使用),还是创建了一个新的RDD,包装了现有的RDD？

以下代码中会发生什么:

// Init
JavaRDD<String> a = ... // some initialise and calculation functions.
JavaRDD<String> b = a.cache();
JavaRDD<String> c = b.cache();

// Case 1, will 'a' be calculated twice in this case 
// because it's before the cache layer:
a.saveAsTextFile(somePath);
a.saveAsTextFile(somePath);

// Case 2, will the data of the calculation of 'a' 
// be cached in the memory twice in this case
// (once as 'b' and once as 'c'):
c.saveAsTextFile(somePath);

Answer 1

在RDD上调用cache()时,RDD的状态是否已更改(并且返回的RDD只是为了易于使用)或者创建了一个新的RDD,包装了现有的RDD

RDD返回相同:

/**
 * Mark this RDD for persisting using the specified level.
 *
 * @param newLevel the target storage level
 * @param allowOverride whether to override any existing level with the new one
 */
  private def persist(newLevel: StorageLevel, allowOverride: Boolean): this.type = {
  // TODO: Handle changes of StorageLevel
  if (storageLevel != StorageLevel.NONE && newLevel != storageLevel && !allowOverride) {
    throw new UnsupportedOperationException(
      "Cannot change storage level of an RDD after it was already assigned a level")
}
  // If this is the first time this RDD is marked for persisting, register it
  // with the SparkContext for cleanups and accounting. Do this only once.
  if (storageLevel == StorageLevel.NONE) {
    sc.cleaner.foreach(_.registerRDDForCleanup(this))
    sc.persistRDD(this)
  }
  storageLevel = newLevel
  this
}

缓存不会对所述RDD造成任何副作用.如果它已标记为持久性,则不会发生任何事情.如果不是这样,唯一的副作用就是将其注册到SparkContext副作用不在其RDD本身的情况,而在于背景.

编辑:

看JavaRDD.cache,似乎底层调用将导致另一个调用JavaRDD:

/** Persist this RDD with the default storage level (`MEMORY_ONLY`). */
def cache(): JavaRDD[T] = wrapRDD(rdd.cache())

其中wrapRDD要求JavaRDD.fromRDD:

object JavaRDD {

  implicit def fromRDD[T: ClassTag](rdd: RDD[T]): JavaRDD[T] = new JavaRDD[T](rdd)
  implicit def toRDD[T](rdd: JavaRDD[T]): RDD[T] = rdd.rdd
}

这将导致新的分配JavaRDD.也就是说,内部实例RDD[T]将保持不变.