Python中的多线程缩略图生成

Question

我想递归一个图像目录并为每个图像生成缩略图.我的机器上有12个可用的核心.什么是利用它们的好方法？我没有太多编写多线程应用程序的经验,所以任何简单的示例代码都是值得赞赏的.提前致谢.

Answer 1

抽象

使用进程而不是线程,因为由于GIL,Python对CPU密集型线程效率低下.多处理的两种可能解决方案是:

该multiprocessing模块

如果您使用内部缩略图制作工具(例如,PIL),则首选此选项.只需编写缩略图制作功能,并行启动12.当其中一个进程完成后,在其插槽中运行另一个进程.

改编自Python文档,这里的脚本应该使用12个核心:

from multiprocessing import Process
import os

def info(title):  # For learning purpose, remove when you got the PID\PPID idea
    print title
    print 'module:', __name__
    print 'parent process:', os.getppid(), 
    print 'process id:', os.getpid()

def f(name):      # Working function
    info('function f')
    print 'hello', name

if __name__ == '__main__':
    info('main line')
    processes=[Process(target=f, args=('bob-%d' % i,)) for i  in range(12)]
    [p.start() for p in processes]
    [p.join()  for p in processes]

附录:使用 multiprocess.pool()

根据soulman的评论,您可以使用提供的进程拉取.

我已经改编了一些代码multiprocessing manual.请注意,您可能应该使用multiprocessing.cpu_count()而不是4自动确定CPU的数量.

from multiprocessing import Pool
import datetime

def f(x):  # You thumbnail maker function, probably using some module like PIL
    print '%-4d: Started at %s' % (x, datetime.datetime.now())
    return x*x

if __name__ == '__main__':
    pool = Pool(processes=4)              # start 4 worker processes
    print pool.map(f, range(25))          # prints "[0, 1, 4,..., 81]"

这给出了(注意打印输出没有严格的订购!):

0   : Started at 2011-04-28 17:25:58.992560
1   : Started at 2011-04-28 17:25:58.992749
4   : Started at 2011-04-28 17:25:58.992829
5   : Started at 2011-04-28 17:25:58.992848
2   : Started at 2011-04-28 17:25:58.992741
3   : Started at 2011-04-28 17:25:58.992877
6   : Started at 2011-04-28 17:25:58.992884
7   : Started at 2011-04-28 17:25:58.992902
10  : Started at 2011-04-28 17:25:58.992998
11  : Started at 2011-04-28 17:25:58.993019
12  : Started at 2011-04-28 17:25:58.993056
13  : Started at 2011-04-28 17:25:58.993074
14  : Started at 2011-04-28 17:25:58.993109
15  : Started at 2011-04-28 17:25:58.993127
8   : Started at 2011-04-28 17:25:58.993025
9   : Started at 2011-04-28 17:25:58.993158
16  : Started at 2011-04-28 17:25:58.993161
17  : Started at 2011-04-28 17:25:58.993179
18  : Started at 2011-04-28 17:25:58.993230
20  : Started at 2011-04-28 17:25:58.993233
19  : Started at 2011-04-28 17:25:58.993249
21  : Started at 2011-04-28 17:25:58.993252
22  : Started at 2011-04-28 17:25:58.993288
24  : Started at 2011-04-28 17:25:58.993297
23  : Started at 2011-04-28 17:25:58.993307
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 
 289, 324, 361, 400, 441, 484, 529, 576]

该subprocess模块

该subprocess模块对于运行外部进程非常有用,因此如果您打算使用像imagemagicks 这样的外部缩略图制作工具,则该模块更受青睐convert.代码示例:

import subprocess as sp

processes=[sp.Popen('your-command-here', shell=True, 
                    stdout=sp.PIPE, stderr=sp.PIPE) for i in range(12)]

现在,迭代进程.如果任何进程已完成(使用subprocess.poll()),请将其删除并将新进程添加到列表中.