Spark上下文'sc'未定义

Question

我是Spark的新手,我正在尝试通过参考以下网站来安装PySpark.

http://ramhiser.com/2015/02/01/configuring-ipython-notebook-support-for-pyspark/

我试图安装两个预构建的包,也通过SBT构建Spark包.

当我尝试在IPython Notebook中运行python代码时,我得到以下错误.

    NameError                                 Traceback (most recent call last)
   <ipython-input-1-f7aa330f6984> in <module>()
      1 # Check that Spark is working
----> 2 largeRange = sc.parallelize(xrange(100000))
      3 reduceTest = largeRange.reduce(lambda a, b: a + b)
      4 filterReduceTest = largeRange.filter(lambda x: x % 7 == 0).sum()
      5 

      NameError: name 'sc' is not defined

在命令窗口中,我可以看到以下错误.

<strong>Failed to find Spark assembly JAR.</strong>
<strong>You need to build Spark before running this program.</strong>

请注意,当我执行spark-shell命令时,我得到了一个scala提示符

更新:

在朋友的帮助下,我能够通过更正.ipython/profile_pyspark/startup/00-pyspark-setup.py文件的内容来解决与Spark程序集JAR相关的问题.

我现在只有Spark Context变量的问题.更改标题以适当反映我当前的问题.

Answer 1

在路径中使用pyspark后需要执行以下操作:

from pyspark import SparkContext
sc =SparkContext()

Answer 2

一种解决方案是添加pyspark-shell到shell环境变量PYSPARK_SUBMIT_ARGS:

export PYSPARK_SUBMIT_ARGS="--master local[2] pyspark-shell"

python/pyspark/java_gateway.py中有一个更改,pyspark-shell如果用户设置了PYSPARK_SUBMIT_ARGS变量,则需要PYSPARK_SUBMIT_ARGS包含.

Answer 3

你必须创建SparkContext的实例,如下所示:

进口:

from pyspark import SparkContext

然后:

sc =SparkContext.getOrCreate()

NB:sc =SparkContext.getOrCreate()效果不错sc =SparkContext().

Answer 4

只是一点点改进.在python脚本文件的顶部添加以下内容.

#! /bin/python
from pyspark import SparkContext, SparkConf
sc =SparkContext()

# your code starts here