Spark 和 Maven：SecurityException：清单的签名文件摘要无效

Question

在提交复制并粘贴到下面的 Spark 应用程序时，我遇到了以下错误的问题。我首先对这个问题进行了一些谷歌搜索，并确定这可能是一个签名 jar 的问题，该签名 jar 是作为我的 jar 构建过程的依赖项加载的。因此，我更改了 POM 以过滤并将其从依赖项中删除。然而，即使在我这样做之后，jar 似乎仍然报告无效的签名文件摘要。

我正在通过 IntelliJ IDE 通过 Build > Build Artifacts 构建我的 jar。
我通过文件 > 项目结构 > 工件 > 提取到目标 Jar 创建了我的工件。

有没有人在 Maven 和 intellij IDE 中编译了 Spark 应用程序，并为我提供了如何正确执行此操作的步骤？

java.lang.SecurityException: Invalid signature file digest for Manifest main att                                                                                                                     ributes
    at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVeri                                                                                                                     fier.java:286)
    at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier                                                                                                                     .java:239)
    at java.util.jar.JarVerifier.processEntry(JarVerifier.java:307)
    at java.util.jar.JarVerifier.update(JarVerifier.java:218)
    at java.util.jar.JarFile.initializeVerifier(JarFile.java:345)
    at java.util.jar.JarFile.getInputStream(JarFile.java:412)
    at sun.misc.JarIndex.getJarIndex(JarIndex.java:137)
    at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:674)
    at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:666)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:665)
    at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:638)
    at sun.misc.URLClassPath$3.run(URLClassPath.java:366)
    at sun.misc.URLClassPath$3.run(URLClassPath.java:356)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.misc.URLClassPath.getLoader(URLClassPath.java:355)
    at sun.misc.URLClassPath.getLoader(URLClassPath.java:332)
    at sun.misc.URLClassPath.getResource(URLClassPath.java:198)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:358)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:351)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:77)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

聚甲醛

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.analysis</groupId>
<artifactId>sentiment</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<name>SENTIMENTTWITTER</name>

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <version>2.4</version>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.3</version>
            <configuration>
                <createDependencyReducedPom>false</createDependencyReducedPom>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
            </configuration>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <transformers>
                            <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                <resource>resource.conf</resource>
                            </transformer>
                            <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>com.verizon.npi.sentiment.MainApp</mainClass>
                            </transformer>
                        </transformers>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.12.4</version>
            <configuration>
                <includes>
                    <include>**/*.class</include>
                </includes>
            </configuration>
        </plugin>
    </plugins>
</build>


<dependencies>
        <dependency>
            <groupId>com.sparkjava</groupId>
            <artifactId>spark-core</artifactId>
            <version>1.1.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka_2.10</artifactId>
            <version>1.2.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>1.2.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.10</artifactId>
            <version>1.2.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.10</artifactId>
            <version>1.2.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.10</artifactId>
            <version>1.2.1</version>
        </dependency>


    </dependencies>

Answer 1

解决了我的问题。问题在于我构建应用程序的方式。您应该通过执行 Maven 目标命令“mvn package”来构建它，而不是通过 intellij 工件构建过程来构建它。这将使用 POM.xml 中指定的过滤条件构建您的 mvn 包。

我的 POM 中的过滤器仍然适用，如果其他人遇到签名 jar 被作为其 jar 的依赖项推送的问题，我在 POM 中的过滤器将删除它们。