在IntelliJ Idea中运行Apache Spark示例应用程序

Question

我试图在Netbeans中运行SparkPi.scala示例程序.不幸的是,我对Spark很新,并且无法成功执行它.

我的偏好是只在Netbeans中工作并从那里执行.我知道火花也允许从火花控制台执行 - 但我不喜欢采取这种方法.

这是我的build.sbt文件内容:

name := "SBTScalaSparkPi"

version := "1.0"

scalaVersion := "2.10.6"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"

这是我的plugins.sbt文件内容:

logLevel := Level.Warn

这是我试图执行的程序:

import scala.math.random

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf

/** Computes an approximation to pi */
object SparkPi {
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("Spark Pi")
    val spark = new SparkContext(conf)
    val slices = if (args.length > 0) args(0).toInt else 2
    val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow
    val count = spark.parallelize(1 until n, slices).map { i =>
        val x = random * 2 - 1
        val y = random * 2 - 1
        if (x*x + y*y < 1) 1 else 0
      }.reduce(_ + _)
    println("Pi is roughly " + 4.0 * count / n)
    spark.stop()
  }
}

JDK版本:1.8.

我在尝试执行代码时得到的错误如下:

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/03/25 07:50:25 INFO SparkContext: Running Spark version 1.6.1
16/03/25 07:50:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/03/25 07:50:26 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:401)
at SparkPi.main(SparkPi.scala)
16/03/25 07:50:26 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.<init>
at SparkPi$.main(SparkPi.scala:28)
at SparkPi.main(SparkPi.scala)
Process finished with exit code 1

在此先感谢您的帮助.

Answer 1

必须在配置中设置主URL

您必须spark.master在SparkConf中设置一个.您必须设置两个必需参数 - 您已设置的主控和AppName.有关更多详细信息,请参阅文档中的初始化Spark部分.

你应该使用哪位大师？有关所有选项,请参阅主URL部分.最简单的测试选项是local在本地计算机上运行整个Spark系统(驱动程序,主服务器,工作程序),无需额外配置.

要通过Scala API设置主服务器:

val conf = new SparkConf().setAppName("Spark Pi").setMaster("local")
val spark = new SparkContext(conf)