错误：发生 JNI 错误，请检查您的安装，然后使用 SBT 在 IntelliJ IDEA for Scala-Spark 程序中重试

Question

import org.apache.spark.sql.SparkSession

object RDDBroadcast extends App {

  val spark = SparkSession.builder()
    .appName("SparkByExamples.com")
    .master("local")
    .getOrCreate()

  val states = Map(("NY","New York"),("CA","California"),("FL","Florida"))
  val countries = Map(("USA","United States of America"),("IN","India"))

  val broadcastStates = spark.sparkContext.broadcast(states)
  val broadcastCountries = spark.sparkContext.broadcast(countries)

  val data = Seq(("James","Smith","USA","CA"),
    ("Michael","Rose","USA","NY"),
    ("Robert","Williams","USA","CA"),
    ("Maria","Jones","USA","FL")
  )

  val rdd = spark.sparkContext.parallelize(data)

  val rdd2 = rdd.map(f=>{
    val country = f._3
    val state = f._4
    val fullCountry = broadcastCountries.value(country)
    val fullState = broadcastStates.value(state)
    (f._1,f._2,fullCountry,fullState)
  })

  println(rdd2.collect().mkString("\n"))

}

上面是获取国家和州名称的 Spark-scala 代码。在 InteliJ IDEA 中编译代码时，出现如下错误：

*Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:650)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:632)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 7 more*

我尝试了多种解决方案，例如检查 java 和 javac 版本，但版本是同步的。 单击此处查看图像。

我还检查了项目文件->项目结构...->模块中的java版本，并将其与运行->编辑配置进行比较，后者也匹配。

Build.sbt代码：点击此处查看sbt代码

我正在使用 VMware Workstation 16 Player，其上安装了 Linux 操作系统中的 inteliJ。java版本是1.8.0_301

单击此处查看 Linux 中的 InteliJ IDE 镜像

Answer 1

我遇到了同样的问题，通过设置运行/调试配置 - >修改选项 - > Java - >将提供范围的依赖项添加到类路径为是来解决它