Spark:按组排序记录？

Question

我有一组记录,我需要:

1)按'日期','城市'和'亲切'分组

2)按奖项对每组进行排序

在我的代码中:

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext

object Sort {

  case class Record(name:String, day: String, kind: String, city: String, prize:Int)

  val recs = Array (
      Record("n1", "d1", "k1", "c1", 10),
      Record("n1", "d1", "k1", "c1", 9),
      Record("n1", "d1", "k1", "c1", 8),
      Record("n2", "d2", "k2", "c2", 1),
      Record("n2", "d2", "k2", "c2", 2),
      Record("n2", "d2", "k2", "c2", 3)
      )

  def main(args: Array[String]): Unit = {
    val conf = new SparkConf()
      .setAppName("Test")
      .set("spark.executor.memory", "2g")
    val sc = new SparkContext(conf)
    val rs = sc.parallelize(recs)
    val rsGrp = rs.groupBy(r => (r.day, r.kind, r.city)).map(_._2)
    val x = rsGrp.map{r => 
      val lst = r.toList
      lst.map{e => (e.prize, e)}
      }
    x.sortByKey()
  }

}

当我尝试对组进行排序时,我收到错误:

value sortByKey is not a member of org.apache.spark.rdd.RDD[List[(Int, 
 Sort.Record)]]

怎么了？怎么排序？

Answer 1

您需要定义一个Key,然后mapValues对它们进行排序.

import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.rdd.RDD
import org.apache.spark.SparkContext._

  object Sort {

    case class Record(name:String, day: String, kind: String, city: String, prize:Int)

    // Define your data

    def main(args: Array[String]): Unit = {
      val conf = new SparkConf()
        .setAppName("Test")
        .setMaster("local")
        .set("spark.executor.memory", "2g")
      val sc = new SparkContext(conf)
      val rs = sc.parallelize(recs)

      // Generate pair RDD neccesary to call groupByKey and group it
      val key: RDD[((String, String, String), Iterable[Record])] = rs.keyBy(r => (r.day, r.city, r.kind)).groupByKey

      // Once grouped you need to sort values of each Key
      val values: RDD[((String, String, String), List[Record])] = key.mapValues(iter => iter.toList.sortBy(_.prize))

      // Print result
      values.collect.foreach(println)
    }
}

Answer 2

groupByKey很贵,它有两个含义:

1. 大多数数据平均在剩余的N-1个分区中进行混洗.
2. 相同密钥的所有记录都会在单个执行程序的内存中加载,从而可能导致内存错误.

根据您的使用情况,您有更好的选择:

1. 如果您不关心排序,请使用reduceByKey或aggregateByKey.
2. 如果您想在不进行任何转换的情况下进行分组和排序,请使用repartitionAndSortWithinPartitions(Spark 1.3.0+ http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd .OrderedRDDFunctions)但要非常小心你指定的分区并测试它,因为你现在依赖可能改变不同环境中行为的副作用.另请参阅此存储库中的示例:https://github.com/sryza/aas/blob/master/ch08-geotime/src/main/scala/com/cloudera/datascience/geotime/RunGeoTime.scala.
3. 如果要应用转换或不可简化的聚合(折叠或扫描)应用于已排序记录的可迭代,请查看此库:spark-sorted https://github.com/tresata/spark-sorted.它为配对的rdds提供了3个API:mapStreamByKey,foldLeftByKey和scanLeftByKey.