相关疑难解决方法(0)

我试图了解如何在YARN群集/客户端上运行spark.我脑子里有以下问题.

1. 是否有必要在纱线群中的所有节点上安装火花？我认为它应该是因为集群中的工作节点执行任务并且应该能够解码驱动程序发送到集群的spark应用程序中的代码(spark API)？

2. 它在文档中说"确保HADOOP_CONF_DIR或YARN_CONF_DIR指向包含Hadoop集群的(客户端)配置文件的目录".为什么客户端节点在将作业发送到集群时必须安装Hadoop？

如果我启动pyspark然后运行此命令:

import my_script; spark = my_script.Sparker(sc); spark.collapse('./data/')

一切都很好.但是,如果我尝试通过命令行和spark-submit执行相同的操作,则会收到错误消息:

Command: /usr/local/spark/bin/spark-submit my_script.py collapse ./data/
  File "/usr/local/spark/python/pyspark/rdd.py", line 352, in func
    return f(iterator)
  File "/usr/local/spark/python/pyspark/rdd.py", line 1576, in combineLocally
    merger.mergeValues(iterator)
  File "/usr/local/spark/python/pyspark/shuffle.py", line 245, in mergeValues
    for k, v in iterator:
  File "/.../my_script.py", line 173, in _json_args_to_arr
    js = cls._json(line)
RuntimeError: uninitialized staticmethod object

my_script:

...
if __name__ == "__main__":
    args = sys.argv[1:]
    if args[0] == 'collapse':
        directory = args[1]
        from pyspark import SparkContext
        sc = SparkContext(appName="Collapse")
        spark = Sparker(sc)
        spark.collapse(directory)
        sc.stop() …