Spark Kafka Streaming Issue

Question

我正在使用maven

我添加了以下依赖项

   <dependency> <!-- Spark dependency -->
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_2.10</artifactId>
      <version>1.1.0</version>
    </dependency>   <dependency> <!-- Spark dependency -->
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming-kafka_2.10</artifactId>
      <version>1.1.0</version>
    </dependency>

我还在代码中添加了jar

SparkConf sparkConf = new SparkConf().setAppName("KafkaSparkTest");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
sc.addJar("/home/test/.m2/repository/org/apache/spark/spark-streaming-kafka_2.10/1.0.2/spark-streaming-kafka_2.10-1.0.2.jar");
JavaStreamingContext jssc = new JavaStreamingContext(sc, new Duration(5000)); 

它可以很好地解决任何错误,当我通过spark-submit运行时,我收到以下错误,非常感谢任何帮助.谢谢你的时间.

bin/spark-submit --class "KafkaSparkStreaming" --master local[4] try/simple-project/target/simple-project-1.0.jar

线程"main"中的异常java.lang.NoClassDefFoundError:org/apache/spark/streaming/kafka/KafkaUtils位于KafkaSparkStreamingTest(KafkaSparkStreaming.java:40),位于sun.reflect的KafkaSparkStreaming.main(KafkaSparkStreaming.java:23). sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)中的NativeMethodAccessorImpl.invoke0(Native Method)位于sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method. java:606)org.apache.spark.deploy.SparkSubmit $ .launch(SparkSubmit.scala:303)atg.apache.spark.deploy.SparkSubmit $ .main(SparkSubmit.scala:55)at org.apache.spark .deploy.SparkSubmit.main(SparkSubmit.scala)引起:java.lang.ClassNotFoundException:java.net.URLClassLoader $ 1.run(URLClassLoader.java:366)中的org.apache.spark.streaming.kafka.KafkaUtils

Answer 1

我遇到了同样的问题,我通过构建带依赖关系的jar来解决它.

1. 删除代码中的"sc.addJar()".

2. 将下面的代码添加到pom.xml

   <build>
       <sourceDirectory>src/main/java</sourceDirectory>
       <testSourceDirectory>src/test/java</testSourceDirectory>
       <plugins>
         <!--
                      Bind the maven-assembly-plugin to the package phase
           this will create a jar file without the storm dependencies
           suitable for deployment to a cluster.
          -->
         <plugin>
           <artifactId>maven-assembly-plugin</artifactId>
           <configuration>
             <descriptorRefs>
               <descriptorRef>jar-with-dependencies</descriptorRef>
             </descriptorRefs>
             <archive>
               <manifest>
                 <mainClass></mainClass>
               </manifest>
             </archive>
           </configuration>
           <executions>
             <execution>
               <id>make-assembly</id>
               <phase>package</phase>
               <goals>
                 <goal>single</goal>
               </goals>
             </execution>
           </executions>
         </plugin>
       </plugins>
   </build>    
   

3. mvn包

4. 提交"example-jar-with-dependencies.jar"