PySpark的addPyFile方法使SparkContext无

Question

我一直试图这样做.在PySpark shell中,我将SparkContext作为sc.但是当我使用addPyFile方法时,它会生成SparkContext None:

>>> sc2 = sc.addPyFile("/home/ec2-user/redis.zip")
>>> sc2 is None
True

怎么了？

Answer 1

下面是pyspark(v1.1.1)addPyFile的源代码.(官方pyspark文档中1.4.1的源链接因我写这篇文章而被破坏)

它返回None,因为没有return声明.另请参见: 在python中,如果函数没有return语句,它返回什么？

所以,如果你做sc2 = sc.addPyFile("mymodule.py") 的当然sc2是None,因为.addPyFile()不会返回任何东西!

取而代之的是,只需调用sc.addPyFile("mymodule.py")并继续使用sc作为SparkContext

def addPyFile(self, path): 
635          """ 
636          Add a .py or .zip dependency for all tasks to be executed on this 
637          SparkContext in the future.  The C{path} passed can be either a local 
638          file, a file in HDFS (or other Hadoop-supported filesystems), or an 
639          HTTP, HTTPS or FTP URI. 
640          """ 
641          self.addFile(path) 
642          (dirname, filename) = os.path.split(path)  # dirname may be directory or HDFS/S3 prefix 
643   
644          if filename.endswith('.zip') or filename.endswith('.ZIP') or filename.endswith('.egg'): 
645              self._python_includes.append(filename) 
646              # for tests in local mode 
647              sys.path.append(os.path.join(SparkFiles.getRootDirectory(), filename))