Але*_*лов 7 java yaml docker spring-boot apache-spark
同学们好。我开发了一个基于 sparkLauncher 的应用程序，它运行一个可执行的 jar，其中有 5 个操作。每个操作取决于特定的变量。我有一个主要的 hadoop 集群 spark2.3.0-hadoop2.6.5。它运作良好。我的工作代码的一部分：
private void runSparkJob(String pathToJar, final LocalDate startDate, final LocalDate endDate) {
if (executionInProgress.get()) {
LOGGER.warn("Execution already in progress");
return;
}
Process sparkProcess = null;
try {
LOGGER.info("Create SparkLauncher. SparkHome: [{}]. JarPath: [{}].", sparkHome, vmJarPath);
executionInProgress.set(true);
sparkProcess = new SparkLauncher()
.setAppName(activeOperationProfile)
.setSparkHome(sparkHome) //sparkHome folder on main cluster
.setAppResource(pathToJar) // jar with 5 operation
.setConf(SparkLauncher.DRIVER_EXTRA_JAVA_OPTIONS,
String.format("-Drunner.operation-profile=%1$s -Doperation.startDate=%2$s -Doperation.endDate=%3$s", activeOperationProfile, startDate,endDate))
.setConf(SparkLauncher.DRIVER_MEMORY, "12G")
.redirectToLog(LOGGER.getName())
.setMaster("yarn")
.launch();
sparkProcess.waitFor();
int exitCode = sparkProcess.exitValue();
if (exitCode != 0) {
throw new RuntimeException("Illegal exit code. Expected: [0]. Actual: [" + exitCode + "]");
}
} catch (IOException | InterruptedException e) {
LOGGER.error("Error occurred while running SparkApplication.", e);
throw new RuntimeException(e);
} finally {
if (sparkProcess != null && sparkProcess.isAlive()) {
LOGGER.warn("Process still alive. Try to kill");
sparkProcess.destroy();
}
executionInProgress.set(false);
}
}
我已经启动了一个 docker 容器，下载的 spark 2.3.0-hadoop6 在哪里。测试人员需要这个容器。我将 master 更改为 .setMaster("local")，将带有 sparkHome、jarsWithOperations 路径的新配置文件和没有阴影的打包 jar（尝试使用阴影，但对我不起作用）。当我尝试运行我的 sparkLaunch 应用程序时，我现在有一个例外：
2018-08-06 14:47:53,150 信息 [nmmbrSparkBaseOperationsRunner.runSparkJob] 105：创建 SparkLauncher。SparkHome：[/opt/bigtv/spark/spark-2.3.0-bin-hadoop2.6]。JarPath：[/opt/bigtv/bin/multirating-bigdata-operations-MASTER-SNAPSHOT.jar]。2018-08-06 14:47:54,905 INFO [oaspark.launcher.OutputRedirector.redirect] 63: 2018-08-06 14:47:54 WARN NativeCodeLoader:62 - 无法为您的平台加载本机 Hadoop 库...在适用的情况下使用内置 Java 类 2018-08-06 14:47:57,042 INFO [oaspark.launcher.OutputRedirector.redirect] 63：2018-08-06 14:47:57 错误 SpringApplication:842 - 应用程序运行失败 2018-08 -06 14:47:57,043 信息 [oaspark.launcher.OutputRedirector.redirect] 63： java.lang.NoSuchMethodError: org.yaml.snakeyaml.Yaml.(Lorg/yaml/snakeyaml/constructor/BaseConstructor;Lorg/yaml/snakeyaml/representer/Representer;Lorg/yaml/snakeyaml/DumperOptions;Lorg/yaml/snakeyaml ;Lorg/yaml/snakeyaml/resolver/Resolver;)V 2018-08-06 14:47:57,043 信息 [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.env.OriginTrackedYamlLoader.createYaml(OriginTrackedYamlLoader.java:70) 2018-04:406:14:406 57,043 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.beans.factory.config.YamlProcessor.process(YamlProcessor.java:139) 2018-08-06 14:47:57,044 INFO [oaspark. OutputRedirector.redirect] 63：在 org.springframework.boot.env.OriginTrackedYamlLoader.load(OriginTrackedYamlLoader.java:75) 2018-08-06 14:47:57,044 INFO [oaspark.launcher.OutputRedirector.redirect] 在 org.3: springframework.boot.env.YamlPropertySourceLoader.load(YamlPropertySourceLoader.java:50) 2018-08-06 14:47:57,044 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.loadDocuments(ConfigFileApplicationListener.java:547) 2018-08-06 14:47:57,044 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context .config.ConfigFileApplicationListener$Loader.load(ConfigFileApplicationListener.java:517) 2018-08-06 14:47:57,045 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener$ Loader.loadForFileExtension(ConfigFileApplicationListener.java:496) 2018-08-06 14:47:57,045 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.load(ConfigFileApplicationListener) .java:464) 2018-08-06 14:47:57,045 信息 [oaspark.launcher.OutputRedirector.redirect] 63 ：在 org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.lambda$null$6(ConfigFileApplicationListener.java:446) 2018-08-06 14:47:57,046 INFO [oaspark.launcher.OutputRedirector.重定向] 63：在 java.lang.Iterable.forEach(Iterable.java:75) 2018-08-06 14:47:57,046 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context。 config.ConfigFileApplicationListener$Loader.lambda$load$7(ConfigFileApplicationListener.java:445) 2018-08-06 14:47:57,046 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 java.lang.Iterable.forEach(Iterable. java:75) 2018-08-06 14:47:57,046 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener$Loader。load(ConfigFileApplicationListener.java:442) 2018-08-06 14:47:57,046 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.load(ConfigFileApplicationListener.java :330) 2018-08-06 14:47:57,047 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener.addPropertySources(ConfigFileApplicationListener.java:212) 2018-0 06 14:47:57,047 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener.postProcessEnvironment(ConfigFileApplicationListener.java:195) 2018-08-06 14:4047:57 [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config。ConfigFileApplicationListener.onApplicationEnvironmentPreparedEvent(ConfigFileApplicationListener.java:182) 2018-08-06 14:47:57,047 INFO [oaspark.launcher.OutputRedirector.redirect] 63 ：在 org.springframework.boot.context.config.ConfigFileApplicationListenerApplication.on :168) 2018-08-06 14:47:57,048 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:170-1042220-172) :47:57,048 INFO [oaspark.launcher.OutputRedirector.redirect] 63: 在 org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) 2018-08-06 14:447:launcher, INoaspark.context.event.SimpleApplicationEventMulticaster.invokeListener .OutputRedirector。重定向] 63 ：在 org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) 2018-08-06 14:47:57,048 INFO [oaspark.launcher.OutputRedirector.redirect] org.spring.framework. context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127) 2018-08-06 14:47:57,049 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.event.EventPreedPublishingRun (EventPublishingRunListener.java:74) 2018-08-06 14:47:57,049 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:54) 2018-8 06 14:47:57,049 信息 [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:358) 2018-08-06 14:47:57,049 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org. springframework.boot.SpringApplication.run(SpringApplication.java:317) 2018-08-06 14:47:57,050 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.SpringApplication.run(SpringApplication.java) :1255) 2018-08-06 14:47:57,050 信息 [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.SpringApplication.run(SpringApplication.java:1243) 2018-08-06 14:47 :57,050 信息 [oaspark.launcher.OutputRedirector.redirect] 63: 在 net.mediascope.multirating.bigdata.operations.OperationRunner.main(OperationRunner.java:21) 2018-08-06 14:47:57,050 INFO [oaspark.launcher.OutputRedirector.redirect] 63 : at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2018-08-06 14:47:57,INFOAspark launcher.OutputRedirector.redirect] 63：at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 2018-08-06 14:47:57,051 INFO [oaspark.launcher.OutputRedirector.redirect] 63 .at sun. DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2018-08-06 14:47:57,051 INFO [oaspark.launcher.OutputRedirector.redirect] 63 : 在 java.lang.reflect.Method.invoke(Method.2005198) -08-06 14:47:57,051 信息 [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) 2018-08-06 14:47:57 ,051 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.loader.Launcher.launch(Launcher.java:87) 2018-08-06 14:47:57,052 INFO [oaspark.launcher.OutputRedirector.重定向] 63：在 org.springframework.boot.loader.Launcher.launch(Launcher.java:50) 2018-08-06 14:47:57,052 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.springframework。 boot.loader.JarLauncher.main(JarLauncher.java:51) 2018-08-06 14:47:57,052 INFO [oaspark.launcher.OutputRedirector.redirect] 63 : at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)-2018 08-06 14:47:57,052 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 2018-08-06 14:47:57,053 INFOlauncher.redirect .OutputRedirector.redirect] 63：在 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2018-08-06 14:47:57,053 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 ref.java.lang.lang. Method.invoke(Method.java:498) 2018-08-06 14:47:57,053 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52 ) 2018-08-06 14:47:57,053 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit. scala:879) 2018-08-06 14:47:57,054 信息 [oaspark.launcher.OutputRedirector.redirect] 63：在 org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197) 2018 -06 14:47:57,054 信息 [oaspark.launcher.OutputRedirector.redirect] 63：在 org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227) 2018-08-06 14:47:57,054 INFO [oaspark.launcher.OutputRedirector.redirect] 63在 org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136) 2018-08-06 14:47:57,054 INFO [oaspark.launcher.OutputRedirector.redirect] 63：在 org.apache.spark.deploy .SparkSubmit.main(SparkSubmit.scala) 2018-08-06 14:47:57,058 INFO [oaspark.launcher.OutputRedirector.redirect] 63 : 2018-08-06 14:47:57 INFO ShutdownHookdown1hookManager:52 -08-06 14:47:57,060 INFO [oaspark.launcher.OutputRedirector.redirect] 63: 2018-08-06 14:47:57 INFO ShutdownHookManager:54 - 删除目录/tmp/spark-55b54924-e629 -ed34d7f35a8b 2018-08-06 14:47:57,151 信息 [osbalConditionEvaluationReportLoggingListener.logAutoConfigurationReport] 101 ：
启动 ApplicationContext 时出错。要显示条件报告，请在启用“调试”的情况下重新运行您的应用程序。
在我的项目中，我从 spring 5.0 开始使用蛇 yaml 1.19，没有其他依赖项。我不明白问题是什么，也许当我把它放入 docker 容器手册时，除了火花之外还需要安装其他东西。
来自带有操作的模块的 Pom：
<dependencies>
<dependency>
<groupId>net.mediascope</groupId>
<artifactId>multirating-bigdata-core</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>
<!-- Data Base -->
<dependency>
<groupId>org.jdbi</groupId>
<artifactId>jdbi</artifactId>
<version>2.71</version>
</dependency>
<dependency>
<groupId>com.microsoft.sqlserver</groupId>
<artifactId>sqljdbc42</artifactId>
<version>4.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.janino</groupId>
<artifactId>commons-compiler</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
</dependency>
<dependency>
<groupId>net.sourceforge.jtds</groupId>
<artifactId>jtds</artifactId>
<version>1.3.1</version>
</dependency>
</dependencies>
<profiles>
<profile>
<id>local</id>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<profiles>
<profile>${profile.active}</profile>
</profiles>
<executable>true</executable>
</configuration>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>hadoop</id>
<build>
<!--?????????? ??? ????????? Spring-Boot ?????????? ??? ?????? ????? Spark-->
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.handlers</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.schemas</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.provides</resource>
</transformer>
<transformer
implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
<resource>META-INF/spring.factories</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>${start-class}</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>I
| 归档时间： |
|
| 查看次数： |
7453 次 |
| 最近记录： |