相关疑难解决方法(0)

我正在推出一个pyspark计划:

$ export SPARK_HOME=
$ export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.9-src.zip
$ python

和py代码:

from pyspark import SparkContext, SparkConf

SparkConf().setAppName("Example").setMaster("local[2]")
sc = SparkContext(conf=conf)

如何添加jar依赖项,例如Databricks csv jar？使用命令行,我可以像这样添加包:

$ pyspark/spark-submit --packages com.databricks:spark-csv_2.10:1.3.0 

但我没有使用任何这些.该程序是一个更大的工作流程的一部分,没有使用spark-submit我应该能够运行我的./foo.py程序,它应该工作.

• 我知道你可以设置extraClassPath的spark属性,但你必须将JAR文件复制到每个节点？
• 尝试了conf.set("spark.jars","jar1,jar2"),它们对py4j CNF异常也不起作用

我在Jupyter笔记本上使用pyspark.以下是Spark设置的方式:

import findspark
findspark.init(spark_home='/home/edamame/spark/spark-2.0.0-bin-spark-2.0.0-bin-hadoop2.6-hive', python_path='python2.7')

    import pyspark
    from pyspark.sql import *

    sc = pyspark.sql.SparkSession.builder.master("yarn-client").config("spark.executor.memory", "2g").config('spark.driver.memory', '1g').config('spark.driver.cores', '4').enableHiveSupport().getOrCreate()

    sqlContext = SQLContext(sc)

然后,当我这样做:

spark_df = sqlContext.createDataFrame(df_in)

哪里df_in是熊猫数据帧.然后我得到以下错误:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-9-1db231ce21c9> in <module>()
----> 1 spark_df = sqlContext.createDataFrame(df_in)


/home/edamame/spark/spark-2.0.0-bin-spark-2.0.0-bin-hadoop2.6-hive/python/pyspark/sql/context.pyc in createDataFrame(self, data, schema, samplingRatio)
    297         Py4JJavaError: ...
    298         """
--> 299         return self.sparkSession.createDataFrame(data, schema, samplingRatio)
    300 
    301     @since(1.3)

/home/edamame/spark/spark-2.0.0-bin-spark-2.0.0-bin-hadoop2.6-hive/python/pyspark/sql/session.pyc in createDataFrame(self, data, schema, samplingRatio)
    520             rdd, schema = self._createFromRDD(data.map(prepare), schema, samplingRatio)
    521         else:
--> 522             rdd, …