同时在不同的目录(os.chdir)中工作(并行线程)

Question

我想并行同步我的所有vcs目录.我要去目录并运行特殊的命令行脚本来同步git或mercurial存储库.这是一个缓慢的过程,所以我想尝试使它平行.

但是我的并行线程争夺"当前目录"有困难所以我需要一些技巧在同一时间在不同的目录中工作.

当前解决方案

def syncrepos(repos):
  for r in repos.split("\n"):
    if r:
      print("------ repository: ", r)
      thrd = ThreadingSync(r)
      thrd.setDaemon(True)
      thrd.start()

ThreadingSync的位置

class ThreadingSync(threading.Thread):
  def __init__(self, repo):
    threading.Thread.__init__(self)
    self.repo = repo
  def run(self):
    r = self.repo.split("-t")
    path = (r[0]).strip()
    if len(r) < 2:
      vcs = VCS.git
    else:
      vcs = {
    'git'       : VCS.git,
    'git git'   : VCS.git_git,
    'git hg'    : VCS.git_mercurial,
    'git svn'   : VCS.git_subversion,
    'git vv'    : VCS.git_veracity,
    'hg hg'     : VCS.hg_hg}[(r[1]).strip()]
    os.chdir(path)
    if vcs == VCS.git:
      checkGitModifications()
      gitSync()
    ... etc

并且gitSync是

def gitSync(): 
  pretty(cmd("git pull origin master"))
  pretty(cmd("git fetch upstream master"))
  pretty(cmd("git pull --rebase upstream master"))
  pretty(cmd("git push -f origin master"))

当然这不完美,但它确实我的工作,我想加快它.

如何为每个存储库/目录生成一个子进程(Thrad安全实现os.chdir)？

Answer 1

创建一个工作池来运行子例程:

http://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers

在你的情况下可能是这样的:

from multiprocessing import Pool
import os

def gitSync(repo):
    print "I am", repo, "and my cwd is:", os.getcwd()
    os.chdir(repo)
    print "I am", repo, "and my cwd is:", os.getcwd()

if __name__ == '__main__':
    dir = os.getcwd()
    repos = [item for item in os.listdir(dir) if os.path.isdir(os.path.join(dir, item))]
    print repos
    pool = Pool(maxtasksperchild=1)
    pool.map(gitSync, repos)
    pool.close()
    pool.join()

请注意,池可以使调试有点困难,因为父级通常不会透露多于一个孩子的死亡,所以让它首先使用单线程.

编辑:那很有意思欣赏 - 注意池的新论点maxtasksperchild=1.该过程不在rebooted调用之间,因此当您在一次调用中更改目录时,当该过程被重用时,您仍然在该目录中.在这里,我只是通过告诉池在每次调用后杀死进程来解决它.

john:captcrunch john$ python foo.py 
['.git', '.idea', 'fixtures', 'lib', 'obj', 'raw', 'tests']
I am .git and my cwd is: /Users/john/code/linz/src/captcrunch
I am .git and my cwd is: /Users/john/code/linz/src/captcrunch/.git
I am .idea and my cwd is: /Users/john/code/linz/src/captcrunch
I am .idea and my cwd is: /Users/john/code/linz/src/captcrunch/.idea
I am fixtures and my cwd is: /Users/john/code/linz/src/captcrunch
I am fixtures and my cwd is: /Users/john/code/linz/src/captcrunch/fixtures
I am lib and my cwd is: /Users/john/code/linz/src/captcrunch
I am lib and my cwd is: /Users/john/code/linz/src/captcrunch/lib
I am obj and my cwd is: /Users/john/code/linz/src/captcrunch
I am obj and my cwd is: /Users/john/code/linz/src/captcrunch/obj
I am raw and my cwd is: /Users/john/code/linz/src/captcrunch
I am raw and my cwd is: /Users/john/code/linz/src/captcrunch/raw
I am tests and my cwd is: /Users/john/code/linz/src/captcrunch
I am tests and my cwd is: /Users/john/code/linz/src/captcrunch/tests