运行Spark hello world代码时sbt出错？

Question

运行spark hello world程序时出现以下错误.

[info] Updating {file:/C:/Users/user1/IdeaProjects/sqlServer/}sqlserver...
[info] Resolving org.apache.spark#spark-core_2.12;2.1.1 ...
[warn]  module not found: org.apache.spark#spark-core_2.12;2.1.1
[warn] ==== local: tried
[warn]   C:\Users\user1\.ivy2\local\org.apache.spark\spark-core_2.12\2.1.1\ivys\ivy.xml
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.12/2.1.1/spark-core_2.12-2.1.1.pom
[warn] ==== local-preloaded-ivy: tried
[warn]   C:\Users\user1\.sbt\preloaded\org.apache.spark\spark-core_2.12\2.1.1\ivys\ivy.xml
[warn] ==== local-preloaded: tried
[warn]   file:/C:/Users/user1/.sbt/preloaded/org/apache/spark/spark-core_2.12/2.1.1/spark-core_2.12-2.1.1.pom
[info] Resolving jline#jline;2.14.3 ...
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.apache.spark#spark-core_2.12;2.1.1: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn]  Note: Unresolved dependencies path:
[warn]          org.apache.spark:spark-core_2.12:2.1.1 (C:\Users\user1\IdeaProjects\sqlServer\build.sbt#L7-8)
[warn]            +- mpa:mpa_2.12:1.0
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.12;2.1.1: not found
[error] Total time: 1 s, completed May 9, 2017 11:05:44 AM

这是build.sbt,

name := "Mpa"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.1"

我的Spark webcome消息.

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.1
      /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_111)
Type in expressions to have them evaluated.
Type :help for more information.

更新:

我改变了built.sbt对

name := "Mpa"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core_2.11" % "2.1.0"

但还是得到了

[info] Updating {file:/C:/Users/user1/IdeaProjects/sqlServer/}sqlserver...
[info] Resolving org.apache.spark#spark-core_2.11_2.11;2.1.0 ...
[warn]  module not found: org.apache.spark#spark-core_2.11_2.11;2.1.0
[warn] ==== local: tried
[warn]   C:\Users\user1\.ivy2\local\org.apache.spark\spark-core_2.11_2.11\2.1.0\ivys\ivy.xml
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.11_2.11/2.1.0/spark-core_2.11_2.11-2.1.0.pom
[warn] ==== local-preloaded-ivy: tried
[warn]   C:\Users\user1\.sbt\preloaded\org.apache.spark\spark-core_2.11_2.11\2.1.0\ivys\ivy.xml
[warn] ==== local-preloaded: tried
[warn]   file:/C:/Users/user1/.sbt/preloaded/org/apache/spark/spark-core_2.11_2.11/2.1.0/spark-core_2.11_2.11-2.1.0.pom
[info] Resolving jline#jline;2.12.1 ...
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.apache.spark#spark-core_2.11_2.11;2.1.0: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn]  Note: Unresolved dependencies path:
[warn]          org.apache.spark:spark-core_2.11_2.11:2.1.0 (C:\Users\user1\IdeaProjects\sqlServer\build.sbt#L7-8)
[warn]            +- mpa:mpa_2.11:1.0
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.11_2.11;2.1.0: not found
[error] Total time: 1 s, completed May 9, 2017 1:01:01 PM

Answer 1

您在built.sbt文件中有错误,您必须更改%%为%:

name := "Mpa"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" % "spark-core" % "2.1.1"

%% 要求Sbt将当前的scala版本添加到工件中

你可以spark-core_2.11用%得到的问题解决了.

// https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0"

希望这可以帮助!

Answer 2

我遇到了同样的错误。

build.sbt

name := "Simple Project"  
version := "1.0"  
scalaVersion := "2.12.3"  
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.0"  

只需将scalaVersion更改为2.11.8或更低。而且有效。