标签: sparkcore

您好，我尝试在本地笔记本电脑上运行 Spark。我在 intelijidea 中创建了一个 mvn 项目，在我的主类中我有一行如下所示，当我尝试运行一个项目时，我收到如下错误

 val spark = SparkSession.builder().master("local").getOrCreate()

21/11/02 18:02:35 INFO BlockManagerMasterEndpoint：BlockManagerMasterEndpoint up 线程“main”中出现异常 java.lang.IllegalAccessError：类 org.apache.spark.storage.StorageUtils$ （在未命名模块 @0x34e9fd99 中）无法访问类 sun。 nio.ch.DirectBuffer（在模块 java.base 中），因为模块 java.base 不会将 sun.nio.ch 导出到 org.apache.spark.storage.StorageUtils$.(StorageUtils.scala:213) 处的未命名模块 @0x34e9fd99 org.apache.spark.storage.BlockManagerMasterEndpoint.(BlockManagerMasterEndpoint.scala:110) 在 org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348) 在 org.apache.spark.SparkEnv$.registerOrLookupEndpoint $1(SparkEnv.scala:287) 在 org.apache.spark.SparkEnv$.create(SparkEnv.scala:336) 在 org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191) 在 org.apache.spark .SparkContext.createSparkEnv(SparkContext.scala:27​​7) 在 org.apache.spark.SparkContext.(SparkContext.scala:460) 在 org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690) 在 org.apache.在 org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession. scala:943) 在 Main$.main(Main.scala:8) 在 Main.main(Main.scala)

我对 pom 的依赖

 <dependencies>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.13</artifactId>
    <version>3.2.0</version>
</dependency>

<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.13</artifactId>
    <version>3.2.0</version>

</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive --> …

我有下面的列表对象：

List<Tuple2<Tuple3<String, Long, Object>, Tuple2<Long, Object[]>>> rddQueue =
    Arrays.asList(tupleList);

如何将其转换为 JavaDStream Spark ？预先感谢！

我正在开发一个使用spark-core lib的插件。当我将其作为Java应用程序运行时还可以，但是当我在Jenkins中运行插件时，它会显示一条错误，指出

    java.lang.ExceptionInInitializerError
    at org.apache.spark.package$.<init>(package.scala:91)
    at org.apache.spark.package$.<clinit>(package.scala)
    at org.apache.spark.SparkContext$$anonfun$3.apply(SparkContext.scala:185)
    at org.apache.spark.SparkContext$$anonfun$3.apply(SparkContext.scala:185)
    at org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
    at org.apache.spark.SparkContext.logInfo(SparkContext.scala:74)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:185)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2275)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:831)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:823)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:823)
    at com.plugin.goettingen_plugin.HelloWorldBuilder.perform(HelloWorldBuilder.java:88)
    at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:75)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:785)
    at hudson.model.Build$BuildExecution.build(Build.java:205)
    at hudson.model.Build$BuildExecution.doRun(Build.java:162)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
    at hudson.model.Run.execute(Run.java:1741)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:408)
Caused by: org.apache.spark.SparkException: Error while locating file spark-version-info.properties
    at org.apache.spark.package$SparkBuildInfo$.liftedTree1$1(package.scala:75)
    at org.apache.spark.package$SparkBuildInfo$.<init>(package.scala:61)
    at org.apache.spark.package$SparkBuildInfo$.<clinit>(package.scala)
    ... 23 more
Caused by: java.lang.NullPointerException
    at java.util.Properties$LineReader.readLine(Properties.java:434)
    at java.util.Properties.load0(Properties.java:353)
    at java.util.Properties.load(Properties.java:341)
    at org.apache.spark.package$SparkBuildInfo$.liftedTree1$1(package.scala:64)
    ... …

我是scala的新手,并且在INTELLJ中获取以下代码的错误,任何人都可以帮助解决它

     import org.apache.spark.{SparkContext, SparkConf}
     object wordcount {
     def main(args: Array[String])
      {
      val conf = new SparkConf()
      .setMaster("local[*]")
      .setAppName("TestSpark")
      .set("spark.executor.memory","2g")

       val sc = new SparkContext(conf)
       val a  = sc.parallelize(Seq("This is the firstline", "This is the  
       second line", "This is the third line"))              
       val count = a.flatMap(x => x.split(" "))
       val counts = count.map(word => (word,1)).reduceByKey((x,y) => x+y)
        counts.foreach(println)

       }

       }

我收到以下错误:

      Exception in thread "main" java.lang.NoSuchMethodError:    
     scala.Predef$.refArrayOps([Ljava/lang/Object;)
      Lscala/collection/mutable/ArrayOps;
      at org.apache.spark.util.Utils$.getCallSite(Utils.scala:1342)
     at org.apache.spark.SparkContext.<init>(SparkContext.scala:81)
     at wordcount$.main(wordcount.scala:12)
     at wordcount.main(wordcount.scala)
     Using Spark's default log4j profile: …

有人可以提及这些因素的区别吗

• 节点/机器数
• 记忆
• 核心数
• 设定
• 部署方式
• 每种模式的优点
• 什么时候应该使用
• 可能的例子

另外，如果我在单个笔记本电脑上本地运行spark，那么该模式是本地模式还是独立模式？