那些遇到过的问题

来自 Janino 和 Commons-Compiler 的 Spark java.lang.NoSuchMethodError

Guy*_*afe 3 java gradle apache-spark

我正在构建一个使用 Spark 进行基于随机森林分类的​​应用程序。尝试运行此程序时，我从该行收到异常： StringIndexerModel labelIndexer = new StringIndexer().setInputCol("label").setOutputCol("indexedLabel").fit(data);

看起来代码以某种方式达到了 Janino 2.7.8 版，尽管我知道我需要 3.0.7。我不知道如何正确设置依赖项以强制构建使用正确的版本。它似乎总是尝试使用2.7.8。

我有可能以某种方式清理缓存吗？

这是来自gradle dependencies：

+--- org.codehaus.janino:janino:3.0.7 -> 2.7.8
|    +--- org.codehaus.janino:commons-compiler:3.0.7
Run Code Online (Sandbox Code Playgroud)

定义依赖项的 Gradle 部分：

dependencies {
  compile('org.apache.hadoop:hadoop-mapreduce-client-core:2.7.2') { force = true }
  compile('org.apache.hadoop:hadoop-common:2.7.2') { force = true }
  // https://mvnrepository.com/artifact/org.codehaus.janino/janino
  compile (group: 'org.codehaus.janino', name: 'janino', version: '3.0.7') {
    force = true
    exclude group: 'org.codehaus.janino', module: 'commons-compiler'
  }
  // https://mvnrepository.com/artifact/org.codehaus.janino/commons-compiler
  compile (group: 'org.codehaus.janino', name: 'commons-compiler', version: '3.0.7') {
    force = true
    exclude group: 'org.codehaus.janino', module: 'janino'
  }
  // https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.11
  compile (group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.2.0') {
    exclude group: 'org.codehaus.janino', module: 'janino'
    exclude group: 'org.codehaus.janino', module: 'commons-compiler'
  }
  // https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11
  compile (group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.2.0') {
    exclude group: 'org.codehaus.janino', module: 'janino'
    exclude group: 'org.codehaus.janino', module: 'commons-compiler'
  }
  // https://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.11
  compile (group: 'org.apache.spark', name: 'spark-mllib_2.11', version: '2.2.0') {
    exclude group: 'org.codehaus.janino', module: 'janino'
    exclude group: 'org.codehaus.janino', module: 'commons-compiler'
  }
  // https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind
  runtime group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.6.5'
  // https://mvnrepository.com/artifact/com.fasterxml.jackson.module/jackson-module-scala_2.11
  runtime group: 'com.fasterxml.jackson.module', name: 'jackson-module-scala_2.11', version: '2.6.5'
  compile group: 'com.google.code.gson', name: 'gson', version: '2.8.1'
  compile group: 'org.apache.logging.log4j', name: 'log4j-api', version: '2.4.1'
  compile group: 'org.apache.logging.log4j', name: 'log4j-core', version: '2.4.1'
  testCompile 'org.testng:testng:6.9.4'
  testCompile 'org.mockito:mockito-core:1.10.19'
}
Run Code Online (Sandbox Code Playgroud)

例外：

Exception in thread "main" java.lang.NoSuchMethodError: org.codehaus.commons.compiler.Location.<init>(Ljava/lang/String;SS)V
    at org.codehaus.janino.Scanner.location(Scanner.java:261)
    at org.codehaus.janino.Parser.location(Parser.java:2742)
    at org.codehaus.janino.Parser.parseImportDeclarationBody(Parser.java:209)
    at org.codehaus.janino.ClassBodyEvaluator.makeCompilationUnit(ClassBodyEvaluator.java:255)
    at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:222)
    at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:192)
    at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:80)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:960)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:1027)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:1024)
    at org.spark_project.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
    at org.spark_project.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
    at org.spark_project.guava.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
    at org.spark_project.guava.cache.LocalCache$Segment.get(LocalCache.java:2257)
    at org.spark_project.guava.cache.LocalCache.get(LocalCache.java:4000)
    at org.spark_project.guava.cache.LocalCache.getOrLoad(LocalCache.java:4004)
    at org.spark_project.guava.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.compile(CodeGenerator.scala:906)
    at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:375)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116)
    at org.apache.spark.sql.execution.DeserializeToObjectExec.doExecute(objects.scala:95)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116)
    at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
    at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
    at org.apache.spark.sql.Dataset.rdd$lzycompute(Dataset.scala:2581)
    at org.apache.spark.sql.Dataset.rdd(Dataset.scala:2578)
    at org.apache.spark.ml.feature.StringIndexer.fit(StringIndexer.scala:111)
Run Code Online (Sandbox Code Playgroud)

Hau*_*Hau 9

与 Zobayer Hasan 的回答类似，我也需要更新org.codehaus.janino:commons-compiler到 3.0.8 版。

就我而言，我只使用org.apache.spark:spark-sql_2.11，但我发现它依赖于org.codehaus.janino:janino版本3.0.8和org.codehaus.janino:commons-compiler版本3.0.0。我能够通过将 3.0.8 版添加commons-compiler到 Maven来解决我的问题，dependencyManagement而无需使用exclusions：

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.codehaus.janino</groupId>
            <artifactId>commons-compiler</artifactId>
            <version>3.0.8</version>
        </dependency>
    </dependencies>
</dependencyManagement>
Run Code Online (Sandbox Code Playgroud)

  • 这也帮助我解决了我的问题。当我们将应用程序从 Spring Boot 1.5.X 升级到 Spring Boot 2.1.X 后，此错误开始发生 (2认同)

Zob*_*san 7

可能你已经解决了问题，我今天也遇到了同样的错误。但是，我不明白您为什么要进行这些排除，而且它们对我来说似乎不正确：

  // https://mvnrepository.com/artifact/org.codehaus.janino/janino
  compile (group: 'org.codehaus.janino', name: 'janino', version: '3.0.7') {
    force = true
    exclude group: 'org.codehaus.janino', module: 'commons-compiler'
  }
  // https://mvnrepository.com/artifact/org.codehaus.janino/commons-compiler
  compile (group: 'org.codehaus.janino', name: 'commons-compiler', version: '3.0.7') {
    force = true
    exclude group: 'org.codehaus.janino', module: 'janino'
  }
Run Code Online (Sandbox Code Playgroud)

我们只需要排除org.codehaus.janino:commons-compiler从org.apache.spark:spark-mllib_2.11（其它火花的依赖已经存在的传递依赖于mllib，就没有必要将它们添加或排除公地编译从他们单独），然后包括org.codehaus.janino:commons-compiler:3.0.7回。

这是一个工作项目的依赖块。我的项目是用 Maven 构建的，但我相信任何人都可以将其转换为 Gradle 等效项。

<!--Spark Libraries-->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>${spark.version}</version>
    <scope>provided</scope>
    <!--Dropping Logger Dependencies-->
    <exclusions>
        <exclusion>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
        <exclusion>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-mllib_2.11</artifactId>
    <version>${spark.version}</version>
    <scope>provided</scope>
    <!--Dropping commons-compiler-->
    <exclusions>
        <exclusion>
            <groupId>org.codehaus.janino</groupId>
            <artifactId>commons-compiler</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>org.codehaus.janino</groupId>
    <artifactId>commons-compiler</artifactId>
    <version>3.0.8</version>
</dependency>
Run Code Online (Sandbox Code Playgroud)

注意：commons-compiler 2.7.8 在使用 spring boot 发行版和 Elasticsearch 2.4 时对我来说也很好用。在我们升级到 spring boot 里程碑版本 2.0.0.M7 和 Elasticsearch 5.6 之后，只需要升级到 3.0.8。

归档时间：

查看次数：

6385 次

最近记录：

7 年，4 月 前
相关归档
HashMap和字典ADT之间的区别 60
调试Spark 1.6.0中的"检测到托管内存泄漏" 32
Android studio/Gradle javadoc任务 13
无法解密安全内容条目：java.io.IOException：getSecretKey 失败：密码不是 ASCII 12
使用spark-sql GROUP BY解决性能和内存问题 11
java.lang.NoClassDefFoundError:org/apache/spark/Logging 9
Spark mllib预测奇怪的数字或NaN 4
kotlin-gradle-plugin 如何使用 'kotlin.code.style' 属性？ 4
Spark在内存中执行TB文件 3
gradle - configure() 方法做什么 3
难疑归档
jQuery是否存在"存在"功能？ 2669
如何计算C#中某人的年龄？ 1754
如何基于通配符匹配以递归方式查找当前和子文件夹中的所有文件？ 1695
Java是否支持默认参数值？ 1569
Android中的gravity和layout_gravity有什么区别？ 1286
Android Studio中的Gradle是什么？ 1257
JavaScript中是否有常量？ 1118
在Python中将两个列表转换为字典 1101
如何将参数传递给批处理文件？ 1100
自我的目的是什么？ 1061