SBT测试任务如何管理类路径以及如何从SBT测试正确启动Java进程

Question

在我使用的一个Scala测试中ProcessBuilder,我在单独的JVM中启动了3个Apache Spark流应用程序.(两个或多个Spark流应用程序不能在同一个JVM中共存.)一个Spark应用程序处理数据并摄入Apache Kafka,其他人阅读.此外,测试涉及写入NoSQL数据库.

使用时ProcessBuilder,Spark应用程序的类路径使用以下命令设置:

val classPath = System.getProperty("java.class.path")

在IntelliJ中运行测试按预期工作,但在CI系统上,测试由SBT的测试任务调用.该java.class.path在后一种情况下,将完全的sbt.jar,所以孩子JVM退出时NoClassFoundException,再次,符合市场预期.:-)

我正在寻找一种方法来使用与测试实际使用的相同类路径从SBT测试中"跨越"JVM.例如,如果在项目中调用测试core,core则应将项目的类路径提供给Spark应用程序启动的子JVM.不幸的是,我不知道如何在SBT任务中检索正确的类路径 - 然后可以将其提供给子JVM.

Answer 1

Tests.Setup可用于访问 SBT 中的类路径：

testOptions in Test += Tests.Setup { classLoader =>
   // give Spark classpath via classLoader
}

例如，在我的机器上Tests.Setup(classLoader => println(classLoader))给出

> test
ClasspathFilter(
  parent = URLClassLoader with NativeCopyLoader with RawResources(
  urls = List(/home/mario/sandbox/sbt/so-classpath/target/scala-2.12/test-classes, /home/mario/sandbox/sbt/so-classpath/target/scala-2.12/classes, /home/mario/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.4.jar, /home/mario/.ivy2/cache/org.scalatest/scalatest_2.12/bundles/scalatest_2.12-3.0.5.jar, /home/mario/.ivy2/cache/org.scalactic/scalactic_2.12/bundles/scalactic_2.12-3.0.5.jar, /home/mario/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.12.4.jar, /home/mario/.ivy2/cache/org.scala-lang.modules/scala-xml_2.12/bundles/scala-xml_2.12-1.0.6.jar),
  parent = DualLoader(a = java.net.URLClassLoader@3fcb37f1, b = java.net.URLClassLoader@271053e1),
  resourceMap = Set(app.class.path, boot.class.path),
  nativeTemp = /tmp/sbt_741bc913/sbt_c770779a
)
  root = sun.misc.Launcher$AppClassLoader@33909752
  cp = Set(/home/mario/.ivy2/cache/jline/jline/jars/jline-2.14.5.jar, /home/mario/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.12.4.jar, /home/mario/.ivy2/cache/org.scala-lang/scala-compiler/jars/scala-compiler-2.12.4.jar, /home/mario/.ivy2/cache/org.scala-lang.modules/scala-xml_2.12/bundles/scala-xml_2.12-1.0.6.jar, /home/mario/sandbox/sbt/so-classpath/target/scala-2.12/classes, /home/mario/.ivy2/cache/org.scalatest/scalatest_2.12/bundles/scalatest_2.12-3.0.5.jar, /home/mario/.sbt/boot/scala-2.10.7/org.scala-sbt/sbt/0.13.17/test-interface-1.0.jar, /home/mario/.ivy2/cache/org.scalactic/scalactic_2.12/bundles/scalactic_2.12-3.0.5.jar, /home/mario/sandbox/sbt/so-classpath/target/scala-2.12/test-classes, /home/mario/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.4.jar)
)

我们看到的

.../target/scala-2.12/test-classes
.../target/scala-2.12/classes

存在。

另一方面，要从测试本身中检索类路径：

val classLoader = this.getClass.getClassLoader
// give Spark classpath via classLoader

例如，在我的机器上，println(classLoader)给出以下测试

class CubeCalculatorTest extends FunSuite {
  test("CubeCalculator.cube") {
    val classLoader = this.getClass.getClassLoader
    println(classLoader)
    assert(CubeCalculator.cube(3) === 27)
  }
}

印刷

URLClassLoader with NativeCopyLoader with RawResources(
  urls = List(/home/mario/sandbox/sbt/so-classpath/target/scala-2.12/test-classes, /home/mario/sandbox/sbt/so-classpath/target/scala-2.12/classes, /home/mario/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.4.jar, /home/mario/.ivy2/cache/org.scalatest/scalatest_2.12/bundles/scalatest_2.12-3.0.5.jar, /home/mario/.ivy2/cache/org.scalactic/scalactic_2.12/bundles/scalactic_2.12-3.0.5.jar, /home/mario/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.12.4.jar, /home/mario/.ivy2/cache/org.scala-lang.modules/scala-xml_2.12/bundles/scala-xml_2.12-1.0.6.jar),
  parent = DualLoader(a = java.net.URLClassLoader@6307eb76, b = java.net.URLClassLoader@271053e1),
  resourceMap = Set(app.class.path, boot.class.path),
  nativeTemp = /tmp/sbt_fa64d1a1/sbt_66bd50e2
)

我们又可以在哪里看到

.../target/scala-2.12/test-classes
.../target/scala-2.12/classes

存在。

要实际将类路径传递到ProcessBuilder测试中：

import java.net.URLClassLoader
import sys.process._

class CubeCalculatorTest extends FunSuite {
  test("CubeCalculator.cube") {
    val classLoader = this.getClass.getClassLoader
    val classpath = classLoader.asInstanceOf[URLClassLoader].getURLs.map(_.getFile).mkString(":")
    s"java -classpath $classpath MyExternalApp".!
    ...
  }
}