python多处理与线程对于Windows和Linux上的cpu绑定工作

Question

所以我敲了一些测试代码,看看多处理模块如何在线程上调整cpu绑定工作.在Linux上,我得到了我期望的性能提升:

linux (dual quad core xeon):
serialrun took 1192.319 ms
parallelrun took 346.727 ms
threadedrun took 2108.172 ms

我的双核macbook pro显示了相同的行为:

osx (dual core macbook pro)
serialrun took 2026.995 ms
parallelrun took 1288.723 ms
threadedrun took 5314.822 ms

然后我去了Windows机器上试了一下,得到了一些非常不同的结果.

windows (i7 920):
serialrun took 1043.000 ms
parallelrun took 3237.000 ms
threadedrun took 2343.000 ms

为什么哦,为什么,多处理方法在Windows上这么慢？

这是测试代码:

#!/usr/bin/env python

import multiprocessing
import threading
import time

def print_timing(func):
    def wrapper(*arg):
        t1 = time.time()
        res = func(*arg)
        t2 = time.time()
        print '%s took %0.3f ms' % (func.func_name, (t2-t1)*1000.0)
        return res
    return wrapper


def counter():
    for i in xrange(1000000):
        pass

@print_timing
def serialrun(x):
    for i in xrange(x):
        counter()

@print_timing
def parallelrun(x):
    proclist = []
    for i in xrange(x):
        p = multiprocessing.Process(target=counter)
        proclist.append(p)
        p.start()

    for i in proclist:
        i.join()

@print_timing
def threadedrun(x):
    threadlist = []
    for i in xrange(x):
        t = threading.Thread(target=counter)
        threadlist.append(t)
        t.start()

    for i in threadlist:
        i.join()

def main():
    serialrun(50)
    parallelrun(50)
    threadedrun(50)

if __name__ == '__main__':
    main()

Answer 1

多处理的python文档归咎于缺少os.fork()来解决Windows中的问题.它可能适用于此.

看看导入psyco时会发生什么.首先,easy_install它:

C:\Users\hughdbrown>\Python26\scripts\easy_install.exe psyco
Searching for psyco
Best match: psyco 1.6
Adding psyco 1.6 to easy-install.pth file

Using c:\python26\lib\site-packages
Processing dependencies for psyco
Finished processing dependencies for psyco

将它添加到python脚本的顶部:

import psyco
psyco.full()

我得到这些结果没有:

serialrun took 1191.000 ms
parallelrun took 3738.000 ms
threadedrun took 2728.000 ms

我得到这些结果:

serialrun took 43.000 ms
parallelrun took 3650.000 ms
threadedrun took 265.000 ms

平行仍然很慢,但其他人烧橡胶.

编辑:同样,尝试使用多处理池.(这是我第一次尝试这个,它是如此之快,我想我必须遗漏一些东西.)

@print_timing
def parallelpoolrun(reps):
    pool = multiprocessing.Pool(processes=4)
    result = pool.apply_async(counter, (reps,))

结果:

C:\Users\hughdbrown\Documents\python\StackOverflow>python  1289813.py
serialrun took 57.000 ms
parallelrun took 3716.000 ms
parallelpoolrun took 128.000 ms
threadedrun took 58.000 ms

Answer 2

UNIX变体下的进程更轻量级.Windows进程很繁重,需要更多时间才能启动.线程是在Windows上进行多处理的推荐方法.

Answer 3

据说在Windows上创建进程比在linux上更昂贵.如果您在网站上搜索,您会找到一些信息.这里有一个,我发现很容易.