相关疑难解决方法(0)

我花了几天时间试图让Spark与我的Jupyter笔记本和Anaconda一起工作.这是我的.bash_profile的样子:

PATH="/my/path/to/anaconda3/bin:$PATH"

export JAVA_HOME="/my/path/to/jdk"
export PYTHON_PATH="/my/path/to/anaconda3/bin/python"
export PYSPARK_PYTHON="/my/path/to/anaconda3/bin/python"

export PATH=$PATH:/my/path/to/spark-2.1.0-bin-hadoop2.7/bin
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook" pyspark
export SPARK_HOME=/my/path/to/spark-2.1.0-bin-hadoop2.7
alias pyspark="pyspark --conf spark.local.dir=/home/puifais --num-executors 30 --driver-memory 128g --executor-memory 6g --packages com.databricks:spark-csv_2.11:1.5.0"

当我输入时/my/path/to/spark-2.1.0-bin-hadoop2.7/bin/spark-shell,我可以在命令行shell中启动Spark.输出sc不是空的.它似乎工作正常.

当我输入时pyspark,它会启动我的Jupyter笔记本电脑.当我创建一个新的Python3笔记本时,会出现此错误:

[IPKernelApp] WARNING | Unknown error in handling PYTHONSTARTUP file /my/path/to/spark-2.1.0-bin-hadoop2.7/python/pyspark/shell.py: 

而sc在我的笔记本Jupyter是空的.

有谁可以帮助解决这种情况？

只是想澄清一下:错误结束后冒号后面没有任何内容.我也尝试使用这篇文章创建我自己的启动文件,我在这里引用,所以你不必去看那里:

我创建了一个简短的初始化脚本init_spark.py,如下所示:

from pyspark import SparkConf, SparkContext
conf = SparkConf().setMaster("yarn-client")
sc = SparkContext(conf = conf)

Run Code Online (Sandbox Code Playgroud)

并将其放在〜/ .ipython/profile_default/startup /目录中

当我这样做时,错误变为:

[IPKernelApp] WARNING | Unknown error in handling PYTHONSTARTUP …

我正在尝试在虚拟环境中使用Jupyter Notebook.我创建了一个新的virtualenv virtualenv ker12 + activate +安装了特定版本的keras或任何其他库.

同样如在IPython笔记本中使用virtualenv中提到的那样:

pip install ipykernel

和

python -m ipykernel install --user --name=my-virtualenv-name

当我运行笔记本并写出 ! which jupyter输出是正确的

/用户/ MYNAME/virtualenv中/ ker12 /斌/蟒蛇

但是当我尝试导入库时,例如import keras存在错误.

ImportError:没有名为keras的模块

但是当我写pip freeze | grep Keras 输出时从另一边是:

Keras == 1.2.0

更新1:

这个问题与Keras无关,它与任何其他库(例如pandas)一起出现

如果我打印os.path输出如下:

来自/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc的<'module'posixpath'

从"命令行python"开始,os.path看起来是正确的

来自'/Users/my_name/virtualenv/ker12/lib/python2.7/posixpath.pyc'的<'module'posixpath'>

更新2:

如果我sys.path从终端打印并且jupyter输出也不同:

从终端

/Users/myname/virtualenv/ker12/lib/python27.zip /Users/myname/virtualenv/ker12/lib/python2.7 /Users/myname/virtualenv/ker12/lib/python2.7/plat-darwin/Users/myname /virtualenv/ker12/lib/python2.7/plat-mac /Users/myname/virtualenv/ker12/lib/python2.7/plat-mac/lib-scriptpackages/Users/myname/virtualenv/ker12/lib/python2.7/lib-tk /Users/myname/virtualenv/ker12/lib/python2.7/lib-old /Users/myname/virtualenv/ker12/lib/python2.7/lib-dynload /usr/local/Cellar/python/2.7 .13/Frameworks/Python.framework/Versions/2.7/lib/python2.7 /usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin /usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk /usr/local/Cellar/python/2.7.13/Frameworks/Python.framework /Versions/2.7/lib/python2.7/plat-mac /usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages /Users/myname/virtualenv/ker12/lib/python2.7/site-packages …