Scala并行收集运行时令人费解

Question

编辑:我的样本量太小.当我对8个CPU的实际数据进行运行时,我看到速度提高了7.2倍.在我的代码中添加4个字符并不太简陋;)

我目前正在尝试"销售"管理使用Scala的好处,特别是在扩展CPU时.为此,我创建了一个简单的测试应用程序,它可以执行一系列矢量数学运算,并且发现在我的四核机器上运行时并不是特别好.有趣的是,我发现运行时是第一次通过集合时最差,并且随后的调用会变得更好.并行集合中是否存在导致此问题的一些懒惰事物,或者我只是做错了？应该注意的是,我来自C++/C#世界,所以我完全可能以某种方式搞砸了我的配置.无论如何,这是我的设置:

InteliJ Scala插件

Scala 2.9.1.final

Windows 7 64位,四核处理器(无超线程)

import util.Random

  // simple Vector3D class that has final x,y,z components a length, and a '-' function
  class Vector3D(val x:Double,  val y:Double, val z:Double)
  {
    def length = math.sqrt(x*x+y*y+z*z)
    def -(rhs : Vector3D ) = new Vector3D(x - rhs.x, y - rhs.y, z - rhs.z)
  }

object MainClass {

  def main(args : Array[String]) =
  {
    println("Available CPU's: " + Runtime.getRuntime.availableProcessors())
    println("Parallelism Degree set to: " + collection.parallel.ForkJoinTasks.defaultForkJoinPool.getParallelism);
    // my position
    val myPos = new Vector3D(0,0,0);

    val r = new Random(0);

    // define a function nextRand that gets us a random between 0 and 100
    def nextRand = r.nextDouble() * 100;

    // make 10 million random targets
    val targets = (0 until 10000000).map(_ => new Vector3D(nextRand, nextRand, nextRand)).toArray
    // take the .par hit before we start profiling
    val parTargets = targets.par

    println("Created " + targets.length + " vectors")

    // define a range function
    val rangeFunc : (Vector3D => Double) = (targetPos) => (targetPos - myPos).length

    // we'll select ones that are <50
    val within50 : (Vector3D => Boolean) = (targetPos) => rangeFunc(targetPos) < 50

    // time it sequentially
    val startTime_sequential = System.currentTimeMillis()
    val numTargetsInRange_sequential = targets.filter(within50)
    val endTime_sequential = System.currentTimeMillis()
    println("Sequential (ms): " + (endTime_sequential - startTime_sequential))

    // do the parallel version 10 times
    for(i <- 1 to 10)
    {

      val startTime_par = System.currentTimeMillis()
      val numTargetsInRange_parallel = parTargets.filter(within50)
      val endTime_par = System.currentTimeMillis()

      val ms = endTime_par - startTime_par;
      println("Iteration[" + i + "] Executed in " + ms + " ms")
    }
  }
}

该程序的输出是:

Available CPU's: 4
Parallelism Degree set to: 4
Created 10000000 vectors
Sequential (ms): 216
Iteration[1] Executed in 227 ms
Iteration[2] Executed in 253 ms
Iteration[3] Executed in 76 ms
Iteration[4] Executed in 78 ms
Iteration[5] Executed in 77 ms
Iteration[6] Executed in 80 ms
Iteration[7] Executed in 78 ms
Iteration[8] Executed in 78 ms
Iteration[9] Executed in 79 ms
Iteration[10] Executed in 82 ms

那么这里发生了什么？前两次我们做过滤器,它慢了,然后事情加快了？我知道本质上会有并行性的启动成本,我只是想弄清楚在我的应用程序中表达并行性是有意义的,特别是我希望能够向Management显示一个运行3-4次的程序在四核盒子上更快.这不是一个好问题吗？

想法？

Answer 1

你有微基准疾病.您最有可能对JIT编译阶段进行基准测试.你需要先预先运行JIT来预热你的JIT.

可能最好的想法是使用像http://code.google.com/p/caliper/这样的微基准测试框架来处理所有这些问题.

编辑:有一个很好的SBT模板用于Caliper基准测试Scala项目,参考此博客文章

Answer 2

事情确实加快了,但这与并行与顺序无关,你不是在比较苹果与苹果.JVM有一个JIT(及时)编译器,只有在代码被使用了一定次数后才会编译一些字节代码.因此,您在第一次迭代中看到的是对于尚未JIT编辑的代码执行速度较慢以及正在进行的JIT编译本身的时间.删除.par所有顺序就是我在我的机器上看到的(迭代次数减少10倍,因为我使用的是旧机器):

Sequential (ms): 312
Iteration[1] Executed in 117 ms
Iteration[2] Executed in 112 ms
Iteration[3] Executed in 112 ms
Iteration[4] Executed in 112 ms
Iteration[5] Executed in 114 ms
Iteration[6] Executed in 113 ms
Iteration[7] Executed in 113 ms
Iteration[8] Executed in 117 ms
Iteration[9] Executed in 113 ms
Iteration[10] Executed in 111 ms

但这都是顺序的!您可以通过使用JVM -XX:+PrintCompilation(设置JAVA_OPTS或使用-J-XX:+PrintCompilationscala选项)来查看JVM在JIT方面的作用.在第一次迭代中,您将看到大量JVM打印语句显示正在进行JIT编辑,然后它会在以后稳定下来.

因此,要比较苹果和苹果,首先运行没有par,然后添加par并运行相同的程序.在我的双核上,当.par我使用时:

Sequential (ms): 329
Iteration[1] Executed in 197 ms
Iteration[2] Executed in 60 ms
Iteration[3] Executed in 57 ms
Iteration[4] Executed in 58 ms
Iteration[5] Executed in 59 ms
Iteration[6] Executed in 73 ms
Iteration[7] Executed in 56 ms
Iteration[8] Executed in 60 ms
Iteration[9] Executed in 58 ms
Iteration[10] Executed in 57 ms

一旦稳定,或多或少加速2倍.

相关的说明,你要小心的另一件事是装箱和解拳,特别是如果你只是比较Java.像过滤器这样的scala库高阶函数正在对原始类型进行装箱和取消装箱,这通常是那些将代码从Java转换为Scala的人最初失望的原因.

虽然它不适用于这种情况,因为for它超出了时间,但是使用for代替的代价也有一些成本while,但2.9.1编译器在使用-optimizescalac标志时应该做得不错.