我试图在Flink中使用Scala XML库来解析XML,但我无法使其工作.请注意,我需要在相同的处理函数中对我的代码使用序列化和非序列化(字符串)版本.
我尝试过不同的解决方案,它们总是在IntelliJ中工作,但是当我在Flink集群上运行时却没有.他们总是回归不同java.lang.LinkageError: com/sun/org/apache/xerces/internal/jaxp/SAXParserImpl$JAXPSAXParser; 我尝试了很多东西,但我仍然得到类似于这个的错误.
这是我的Flink Job的样子:
object StreamingJob {
import org.apache.flink.streaming.api.scala._
val l = List(
"""<ciao>ciao</ciao>""",
)
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
// set up kafka section excluded
env.setParallelism(10)
val stream = env.fromCollection(l)
stream
.uid("process")
.map(new Processor)
.print
env.execute("Flink-TEST")
}
}
这是我的处理功能的一个例子:
import javax.xml.parsers.{SAXParser, SAXParserFactory}
import org.apache.flink.api.common.functions.MapFunction
import scala.xml.{Elem, XML}
import scala.xml.factory.XMLLoader
class Processor extends MapFunction[String, String] {
override def map(translatedMessage: String): String = {
val xml = Processor.xmlLoader.loadString(translatedMessage)
xml.toString
}
} …尝试运行可执行jar文件时,我正面临"循环占位符引用"异常.这是详细的例外.
org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'postProcessProperties' defined in class path resource [applicationContext.xml]: Circular placeholder reference 'processor.core.poolsize' in property definitions
[echo] at org.springframework.beans.factory.config.PropertyPlaceholderConfigurer.processProperties(PropertyPlaceholderConfigurer.java:287)
[echo] at org.springframework.beans.factory.config.PropertyResourceConfigurer.postProcessBeanFactory(PropertyResourceConfigurer.java:75)
[echo] at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:663)
[echo] at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:638)
[echo] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:407)
[echo] at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:139)
[echo] at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:83)
[echo] at com.autodesk.postprocess.engine.PostProcessEngine.start(PostProcessEngine.java:39)
[echo] at com.autodesk.postprocess.engine.PostProcessEngine.main(PostProcessEngine.java:29)
这是一个spring应用程序,它使用外部属性文件在启动时读取值.这是春天的定义.到目前为止,这种方法运作良好.
<bean id="propertyConfig"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_NEVER" />
<property name="ignoreResourceNotFound" value="true" />
<property name="locations">
<list>
<value>classpath:/postprocess.properties</value>
</list>
</property>
<property name="properties">
<props>
<prop key="processor.core.poolsize">${processor.core.poolsize}</prop>
<prop key="processor.max.poolsize">${processor.max.poolsize}</prop>
</props>
</property>
</bean>
<bean id="postProcessProperties"
class="org.springframework.beans.factory.config.PropertiesFactoryBean">
<property …我在Apache Storm中运行Geotools,并且在geotools依赖管理方面有一段时间.当我在本地运行风暴集群(Windows 7笔记本电脑)时,每件事都有效,但当我部署到集群时,我在日志中遇到此异常.
编辑:这是我使用的Java调用此功能
GridCoverage2D image =
new GeoTiffReader(f).read(new GeneralParameterValue[]{policy, gridsize, useJaiRead});
/**
* reproject to WGS84
*/
CoordinateReferenceSystem targetCRS = CRS.decode("EPSG:4326");
GridCoverage2D reprojectedImage = (GridCoverage2D) Operations.DEFAULT.resample(image, targetCRS);
这是我的环境
geotools 11.1
java 7
POM is below
running on windows 7 when in local mode (works perfectly here)
this problem happens on Ubuntu 12.04.5 LTS (GNU/Linux 3.2.0-63-virtual x86_64)
here's the exception from the log
[ERROR] Exception in Bolt org.geotools.data.DataSourceException: GEOTIFF Module Error Report
No code "EPSG:32637" from authority "EPSG" found for …我们需要使用Maven构建一个jar,它包含所有依赖项,但是所有这些依赖项都被重命名(重定位).
假设我们自己的包都以com.mycompagny.projectx.*".我们希望项目依赖项将其包重命名为以" embedded" 开头,而不是我们自己的类.
以maven-shade-plugin为例,我无法做到这一点:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<configuration>
<shadedArtifactAttached>true</shadedArtifactAttached>
<createDependencyReducedPom>true</createDependencyReducedPom>
<artifactSet>
<includes>
<include>*.*</include>
</includes>
</artifactSet>
<relocations>
<relocation>
<pattern>*</pattern>
<shadedPattern>embedded.</shadedPattern>
<excludes>
<exclude>com.mycompagny.projectx.*</exclude>
</excludes>
</relocation>
</relocations>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
这<pattern>*</pattern>是无效的.另外,如果我使用<pattern></pattern>(空字符串),那么所有内容都会重新定位到"嵌入式"包,甚至是资源("META-INF"目录)!当然,我们希望资源保留在jar的根部.
我想我们可以创建多个<relocation>要素,一个是依赖的每个包,但是这将是大量的工作:<relocation>com</relocation>,<relocation>net</relocation>,<relocation>javax</relocation>等.
如何轻松地重新定位超级jar中的所有依赖项,而不涉及我们自己的类,资源和"META-INF"目录？
我有一个包含两个依赖项A和B的Maven项目.每一种过渡地依赖于Ç,但它们依赖于不同版本的Ç.假设A取决于C版本1,B取决于C版本2.
不幸的是,一个不是字节码兼容版本2,也不是乙与第1版(碰巧,一个是源兼容与2版本,但我不认为这将有助于我们在这里.)
这意味着我在项目中需要两个版本的传递依赖项,我需要A使用版本1,B需要使用版本2.
有办法做到这一点吗？
我曾假设我需要使用shade插件来重新定位A的包名及其所有依赖项,但这似乎不可能.如果我遮蔽A,它的依赖关系不会被遮挡,它仍然会选择版本2,并且无法运行.
对于我的带有嵌入式tomcat的Spring启动应用程序,由于某些限制,我需要废除 spring-boot-maven-plugin并需要使用maven-shade-plugin.使用maven package命令,我可以成功创建jar文件.但是,我的所有控制器都停止工作,并给我404错误.此外,我的资源文件夹中的静态内容不再提供.总是得到404错误.
我的pom.xml
<dependencyManagement>
<dependencies>
<dependency>
<!-- Import dependency management from Spring Boot -->
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>1.2.7.RELEASE</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<java.version>1.7</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>abc.MyMainClass</mainClass>
</transformer>
</transformers>
</configuration>
</execution> …我正在使用maven shade插件为我的项目生成一个合并jar.jar是按预期生成的,当我尝试使用jar并运行它时,我得到了一个
java.lang.SecurityException:Manifest主要属性错误的签名文件摘要无效.
我搜索了上面的错误消息,许多人建议从META-INF目录中排除清单签名.因此,我已经包含了从目录中排除这些文件的步骤[我看到两个文件的名称JARSIGN_.RSA和JARSIGN_.SF],但由于一些奇怪的原因,maven shade插件无法从META-INF目录中排除这些文件.谁能解释一下我可能做错了什么？我的pom.xml在下面,我用来生成jar的命令是:
mvn clean package shade:shade
的pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.abc.xyz</groupId>
<artifactId>myjar</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<url>http://maven.apache.org</url>
<properties>
<!-- A few custom properties -->
</properties>
<dependencies>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.3.1</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>18.0</version>
</dependency>
<!-- Other The dependencies are here -->
</dependencies>
<repositories>
<!-- Repository Information -->
</repositories>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
<!-- Maven Shade Plugin -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.2</version>
<executions> …我在我的项目中使用了 maven shade 插件来重新定位一个包下的所有依赖 jar 类，例如 org.shade.*
当我尝试在其他应用程序中使用该阴影 jar 作为 maven 依赖项时，它会拉取依赖项 jar。
我的期望是当 uber/shaded jar 作为 maven 依赖项包含时，它不应该拉任何其他依赖类 jar，因为这些类已经在 shaded jar 中重新打包。
当我在intellij Idea中运行main方法时，项目中存在log4j2，它可以正确打印日志。
当我使用maven-shade-plugin包项目来jar文件，并将jar作为独立应用程序运行时，它显示错误：
java -cp package.jar com.xxx.TestMain
控制台输出
ERROR StatusLogger Unrecognized format specifier [d]
ERROR StatusLogger Unrecognized conversion specifier [d] starting at position 16 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [thread]
ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [level]
ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [logger]
ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position …Spring 3.1.1.RELEASE与Apache Maven 3.0.3和Maven Shade插件1.6.
使用mvn shade插件将工件打包在uber-jar中,包括它的依赖项:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>1.6</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.MyApplication</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
似乎打包很好,但在执行抱怨Spring NamespaceHandler问题:
Caused by: org.springframework.beans.factory.parsing.BeanDefinitionParsingException:
Configuration problem: Unable to locate Spring NamespaceHandler for XML schema namespace [http://www.springframework.org/schema/util]
Offending resource: class path resource [spring/test-context.xml]
这适用于util和p命名空间,但期望它不限于这些:
xmlns:util="http://www.springframework.org/schema/util"
xmlns:p="http://www.springframework.org/schema/p"
如果我手写重写属性或列表(util),问题就会消失.