相关疑难解决方法(0)

我已经在线学习了一些教程,但他们不能Spark 1.5.1在OS X El Capitan上工作(10.11)

基本上我已经运行了这个命令下载 apache-spark

brew update
brew install scala
brew install apache-spark

更新了.bash_profile

# For a ipython notebook and pyspark integration
if which pyspark > /dev/null; then
  export SPARK_HOME="/usr/local/Cellar/apache-spark/1.5.1/libexec/"
  export PYSPARK_SUBMIT_ARGS="--master local[2]"
fi

跑

ipython profile create pyspark

创建了~/.ipython/profile_pyspark/startup/00-pyspark-setup.py以这种方式配置的启动文件

# Configure the necessary Spark environment
import os
import sys

# Spark home
spark_home = os.environ.get("SPARK_HOME")

# If Spark V1.4.x is detected, then add ' pyspark-shell' to
# the end of the 'PYSPARK_SUBMIT_ARGS' environment …

我正在尝试提交 spark-submit，但由于奇怪的消息而失败。

 Error: Could not find or load main class org.apache.spark.launcher.Main
 /opt/spark/bin/spark-class: line 96: CMD: bad array subscript

这是我第一次看到这种错误。我试图检查 spark-class 文件的代码，但无法破译导致问题的原因。

    # Turn off posix mode since it does not allow process substitution
set +o posix
CMD=()
DELIM=$'\n'
CMD_START_FLAG="false"
while IFS= read -d "$DELIM" -r ARG; do
  if [ "$CMD_START_FLAG" == "true" ]; then
    CMD+=("$ARG")
  else
    if [ "$ARG" == $'\0' ]; then
      # After NULL character is consumed, change the delimiter and consume command string.
      DELIM=''
      CMD_START_FLAG="true"
    elif [ …