我正试图在macbook air上运行pyspark.当我尝试启动它时,我收到错误:
Exception: Java gateway process exited before sending the driver its port number
Run Code Online (Sandbox Code Playgroud)
当sc =启动时调用SparkContext().我试过运行以下命令:
./bin/pyspark
./bin/spark-shell
export PYSPARK_SUBMIT_ARGS="--master local[2] pyspark-shell"
Run Code Online (Sandbox Code Playgroud)
无济于事.我也看过这里:
Spark + Python - 在向驱动程序发送端口号之前退出Java网关进程?
但问题从未得到解答.请帮忙!谢谢.
我是Spark的新手,我正在尝试通过参考以下网站来安装PySpark.
http://ramhiser.com/2015/02/01/configuring-ipython-notebook-support-for-pyspark/
我试图安装两个预构建的包,也通过SBT构建Spark包.
当我尝试在IPython Notebook中运行python代码时,我得到以下错误.
NameError Traceback (most recent call last)
<ipython-input-1-f7aa330f6984> in <module>()
1 # Check that Spark is working
----> 2 largeRange = sc.parallelize(xrange(100000))
3 reduceTest = largeRange.reduce(lambda a, b: a + b)
4 filterReduceTest = largeRange.filter(lambda x: x % 7 == 0).sum()
5
NameError: name 'sc' is not defined
Run Code Online (Sandbox Code Playgroud)
在命令窗口中,我可以看到以下错误.
<strong>Failed to find Spark assembly JAR.</strong>
<strong>You need to build Spark before running this program.</strong>
Run Code Online (Sandbox Code Playgroud)
请注意,当我执行spark-shell命令时,我得到了一个scala提示符
更新:
在朋友的帮助下,我能够通过更正.ipython/profile_pyspark/startup/00-pyspark-setup.py文件的内容来解决与Spark程序集JAR相关的问题.
我现在只有Spark Context变量的问题.更改标题以适当反映我当前的问题.
因此,我尝试使用以下命令在 Python 2.7 中创建 Spark 会话:
#Initialize SparkSession and SparkContext
from pyspark.sql import SparkSession
from pyspark import SparkContext
#Create a Spark Session
SpSession = SparkSession \
.builder \
.master("local[2]") \
.appName("V2 Maestros") \
.config("spark.executor.memory", "1g") \
.config("spark.cores.max","2") \
.config("spark.sql.warehouse.dir", "file:///c:/temp/spark-warehouse")\
.getOrCreate()
#Get the Spark Context from Spark Session
SpContext = SpSession.sparkContext
Run Code Online (Sandbox Code Playgroud)
我收到以下指向python\lib\pyspark.zip\pyspark\java_gateway.py
路径的错误
Exception: Java gateway process exited before sending the driver its port number
Run Code Online (Sandbox Code Playgroud)
试图查看 java_gateway.py 文件,内容如下:
import atexit
import os
import sys
import select
import signal
import …
Run Code Online (Sandbox Code Playgroud)