Python在不阻塞父级的情况下加入进程

Question

我正在编写一个程序来查看包含下载URL的新文件的特定目录.一旦检测到新文件,它将创建一个新进程以在父进程继续观看目录时进行实际下载.我正在使用Process界面multiprocessing.我遇到的问题是,除非我调用process.join()子进程仍在运行,但process.join()是一个阻塞函数,它无法创建子进程来处理实际的下载.

我的问题是,有没有办法以非阻塞的方式加入子进程,这将允许父进程继续做它的事情？

部分代码:

def main(argv):
  # parse command line args
  ...
  # set up variables
  ...
  watch_dir(watch_dir, download_dir)


def watch_dir(wDir, dDir):
  # Grab the current watch directory listing
  before = dict([(f, None) for f in os.listdir (wDir)])

  # Loop FOREVER
  while 1:
    # sleep for 10 secs
    time.sleep(10)

    # Grab the current dir listing
    after = dict([(f, None) for f in os.listdir (wDir)])

    # Get the list of new files
    added = [f for f in after if not f in before]
    # Get the list of deleted files
    removed = [f for f in before if not f in after]

    if added:
      # We have new files, do your stuff
      print "Added: ", ", ".join(added)

      # Call the new process for downloading
      p = Process(target=child, args=(added, wDir, dDir))
      p.start()
      p.join()

    if removed:
      # tell the user the file was deleted
      print "Removed: ", ", ".join(removed)

    # Set before to the current
    before = after

def child(filename, wDir, dDir):
  # Open filename and extract the url
  ...
  # Download the file and to the dDir directory
  ...
  # Delete filename from the watch directory
  ...
  # exit cleanly
  os._exit(0)

父母等待孩子完成执行,然后继续p.join()(据我所知)正确.但这违背了创造孩子的整个目的.如果我离开,p.join()那么孩子仍然活跃,ps ax | greppython给我'python <defunct>'.

我希望这个孩子能够完成它的工作并且在没有阻止父母的情况下离开.有办法吗？

Answer 1

您可以设置一个单独的线程来进行连接.让它监听您推送子进程句柄的队列:

class Joiner(Thread):
    def __init__(self, q):
        self.__q = q
    def run(self):
        while True:
            child = self.__q.get()
            if child == None:
                return
            child.join()

然后,代替p.join(),做joinq.put(p)和做一个joinq.put(None)信号线程停止.确保使用FIFO队列.

Answer 2

在你的while循环中,调用

multiprocessing.active_children()

返回当前进程的所有活孩子的列表.调用它会产生"加入"已经完成的任何进程的副作用.