我在iPython笔记本(python v.3.6)中使用PySpark(v.2.1.0)而不是我的Mac(Sierra 10.12.3 Beta)中的virtualenv.
1.我在终端拍摄推出了iPython笔记本 -
PYSPARK_PYTHON=python3 PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook" /Applications/spark-2.1.0-bin-hadoop2.7/bin/pyspark
Run Code Online (Sandbox Code Playgroud)
2.将我的文件加载到Spark Context并确保其加载 -
>>>lines = sc.textFile("/Users/PanchusMac/Dropbox/Learn_py/Virtual_Env/pyspark/README.md")
>>>for i in lines.collect():
print(i)
Run Code Online (Sandbox Code Playgroud)
它工作正常并将结果打印在我的控制台上,如下所示:
# Apache Spark
Spark is a fast and general cluster computing system for Big Data. It provides
high-level APIs in Scala, Java, Python, and R, and an optimized engine that
supports general computation graphs for data analysis. It also supports a
rich set of higher-level tools including Spark SQL for SQL and DataFrames,
MLlib for machine learning, GraphX …Run Code Online (Sandbox Code Playgroud) 我目前在我的 Mac Sierra 上使用 java 1.8。
java -version
java version "1.8.0_51"
Java(TM) SE Runtime Environment (build 1.8.0_51-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)
Run Code Online (Sandbox Code Playgroud)
现在我想运行“jshell”来直接从终端执行简单的java命令。但我最终得到了以下错误消息 -
Unable to locate an executable at "/Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/bin/jshell" (-1)
Run Code Online (Sandbox Code Playgroud)
我需要采取什么解决方法来避免出现此错误消息?
apache-spark ×1
java ×1
java-9 ×1
jshell ×1
macos ×1
pyspark ×1
python ×1
terminal ×1
virtualenv ×1