小编Gri*_*sov的帖子

从本地 jupyter 笔记本连接到 Spark 集群

我尝试从本地计算机上的笔记本连接到远程 Spark Master。

当我尝试创建 SparkContext 时

sc = pyspark.SparkContext(master = "spark://remote-spark-master-hostname:7077", 
                          appName="jupyter notebook_test"),
Run Code Online (Sandbox Code Playgroud)

我收到以下异常:

/opt/.venv/lib/python3.7/site-packages/pyspark/context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
    134         try:
    135             self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
--> 136                           conf, jsc, profiler_cls)
    137         except:
    138             # If an error occurs, clean up in order to allow future SparkContext creation:

/opt/.venv/lib/python3.7/site-packages/pyspark/context.py in _do_init(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, jsc, profiler_cls)
    196 
    197         # Create the Java SparkContext …
Run Code Online (Sandbox Code Playgroud)

py4j apache-spark pyspark jupyter-notebook

7
推荐指数
1
解决办法
6589
查看次数

标签 统计

apache-spark ×1

jupyter-notebook ×1

py4j ×1

pyspark ×1