火花大师无反应

Question

我正在尝试在 Mac 上以独立模式运行一个简单的 Spark 应用程序。

我设法运行./sbin/start-master.sh来启动主服务器和工作人员。

./bin/spark-shell --master spark://MacBook-Pro.local:7077也有效，我可以在 Master WebUI 的运行应用程序列表中看到它

现在我正在尝试编写一个简单的 Spark 应用程序。

import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.SparkContext._

object SimpleApp {
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("Simple Application")
                          .setMaster("spark://MacBook-Pro.local:7077")
    val sc = new SparkContext(conf)

    val logFile = "README.md"
    val logData = sc.textFile(logFile, 2).cache()
    val numAs = logData.filter(line => line.contains("a")).count()
    val numBs = logData.filter(line => line.contains("b")).count()
    println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
  }
}

运行这个简单的应用程序会给出错误消息，表明 Master 没有响应

15/02/15 09:47:47 INFO AppClient$ClientActor: Connecting to master spark://MacBook-Pro.local:7077...
15/02/15 09:47:48 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkMaster@MacBook-Pro.local:7077] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/02/15 09:48:07 INFO AppClient$ClientActor: Connecting to master spark://MacBook-Pro.local:7077...
15/02/15 09:48:07 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkMaster@MacBook-Pro.local:7077] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/02/15 09:48:27 INFO AppClient$ClientActor: Connecting to master spark://MacBook-Pro.local:7077...
15/02/15 09:48:27 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkMaster@MacBook-Pro.local:7077] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/02/15 09:48:47 ERROR SparkDeploySchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
15/02/15 09:48:47 WARN SparkDeploySchedulerBackend: Application ID is not initialized yet.
15/02/15 09:48:47 ERROR TaskSchedulerImpl: Exiting due to error from cluster scheduler: All masters are unresponsive! Giving up.

知道有什么问题吗？谢谢

Answer 1

您可以在调用时设置主站spark-submit，或者（正如您在此处所做的那样）通过 显式设置它SparkConf。尝试按照Spark 配置文档中的示例进行操作，并按如下方式设置 master：

\n\n

val conf = new SparkConf().setMaster("local[2]")

\n\n

来自同一页面（解释后面括号中的数字local）：“请注意，我们使用 local[2] 运行，这意味着两个线程 - 表示 \xe2\x80\x9cminimal\xe2\x80\x9d 并行性，这可以帮助检测错误仅当我们在分布式上下文中运行时才存在。”

\n