小编S. *_* SP的帖子

Spark任务不可序列化

我已经尝试了所有在StackOverflow上发现的该问题的解决方案,但是尽管如此,我还是无法解决。我有一个“ MainObj”对象,该对象实例化了“ Recommendation”对象。当我调用“ recommendationProducts”方法时,总是出现错误。这是该方法的代码:

def recommendationProducts(item: Int): Unit = {

val aMatrix = new DoubleMatrix(Array(1.0, 2.0, 3.0))

def cosineSimilarity(vec1: DoubleMatrix, vec2: DoubleMatrix): Double = {
  vec1.dot(vec2) / (vec1.norm2() * vec2.norm2())
}

val itemFactor = model.productFeatures.lookup(item).head
val itemVector = new DoubleMatrix(itemFactor)

//Here is where I get the error:
val sims = model.productFeatures.map { case (id, factor) =>
  val factorVector = new DoubleMatrix(factor)
  val sim = cosineSimilarity(factorVector, itemVector)
  (id, sim)
}

val sortedSims = sims.top(10)(Ordering.by[(Int, Double), Double] {
  case (id, similarity) => …
Run Code Online (Sandbox Code Playgroud)

serialization scala exception serializable apache-spark

2
推荐指数
1
解决办法
2150
查看次数