我正在尝试运行小火花应用程序,并得到以下异常:
Exception in thread "main" java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:262)
at org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:217)
at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
at scala.Option.getOrElse(Option.scala:120)
Run Code Online (Sandbox Code Playgroud)
相关的gradle依赖项部分:
compile('org.apache.spark:spark-core_2.10:1.3.1')
compile('org.apache.hadoop:hadoop-mapreduce-client-core:2.6.2') {force = true}
compile('org.apache.hadoop:hadoop-mapreduce-client-app:2.6.2') {force = true}
compile('org.apache.hadoop:hadoop-mapreduce-client-shuffle:2.6.2') {force = true}
compile('com.google.guava:guava:19.0') { force = true }
Run Code Online (Sandbox Code Playgroud) 运行时:
public static void main(String... args) throws InterruptedException {
while (true) {
System.out.print(".");
Thread.sleep(200);
}
}
Run Code Online (Sandbox Code Playgroud)
从junit运行相同的代码时:
@Test
public void test() throws Exception {
while (true) {
System.out.print(".");
Thread.sleep(200);
}
}
Run Code Online (Sandbox Code Playgroud)
有不同的行为:
对于main() - 输出按预期显示,因为进程运行("." - >".." - >"...")
但是,对于JUnit,当运行相同的代码时,进程运行时不显示输出 - 仅在退出测试时刷新.
为什么会这样?如果我需要在测试运行期间在控制台中显示状态,有没有办法解决它?
我需要打印到同一行,所以使用println不适合.
在Intellij以前的版本中,在我的build.gradle中,以下内容足以让intellij在项目树中显示在使用gradle插件栏中的"refresh"时标记为测试源的integrationTest目录:
sourceSets {
integrationTest {
java {
compileClasspath += main.output + test.output
runtimeClasspath += main.output + test.output
srcDir 'src/integrationTest/java'
}
}
}
configurations {
integrationTestCompile.extendsFrom testCompile
integrationTestRuntime.extendsFrom testRuntime
}
task integrationTest(type: Test) {
testClassesDir = project.sourceSets.integrationTest.output.classesDir
classpath = project.sourceSets.integrationTest.runtimeClasspath
it.shouldRunAfter test
}
Run Code Online (Sandbox Code Playgroud)
它应该在这个截图中看起来像.但是在Intellij 16中,它不再起作用了.添加以下块并运行构思任务也无济于事:
idea {
module {
testSourceDirs += file('src/integrationTest/java')
scopes.TEST.plus += configurations.integrationTestCompile
scopes.TEST.plus += configurations.integrationTestRuntime
}
}
Run Code Online (Sandbox Code Playgroud)
可能是什么问题呢?
apache-spark ×1
gradle ×1
guava ×1
hadoop ×1
java ×1
junit ×1
junit4 ×1
mapreduce ×1
system.out ×1
testing ×1