我正在尝试使用spark执行K最近邻搜索.
我有一个RDD [Seq [Double]],我正在计划用实际的行和邻居列表返回RDD [(Seq [Double],Seq [Seq [Double]])]
val out = data.map(row => {
val neighbours = data.top(num = 3)(new Ordering[Seq[Double]] {
override def compare(a:Seq[Double],b:Seq[Double]) = {
euclideanDistance(a,row).compare(euclideanDistance(b,row))*(-1)
}
})
(row,neighbours.toSeq)
})
并且它在spark提交时给出以下错误
15/04/29 21:15:39 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 2, 192.168.1.7): org.apache.spark.SparkException: RDD transformations and actions can only be invoked by the driver, not inside of other transformations; for example, rdd1.map(x => rdd2.values.count() * x) is invalid because the values transformation and count action cannot …