并行运行许多函数,并将所有结果收集到列表中

Question

我有一个CPU密集型功能:

def entity_intersections(ent, collidable):
    intersections = []

    for line1, line2 in product(ent.shape, collidable.shape):

        pair_intersections = find_intersections(line1 + ent.position, ent.velocity, ent.acceleration, line2 + collidable.position, collidable.velocity, collidable.acceleration, ent, collidable)
        intersections.extend(pair_intersections)

    return intersections

我希望让所有调用find_intersections并行运行,以便它们执行得更快,同时仍然将所有结果收集在一起(一旦所有执行完成).什么库允许我这样做,因为这find_intersections是一个纯函数？

将非常感谢如何生成这些并行执行以及将结果收集在一起的示例.

Answer 1

最简单的方法是使用multiprocessing模块:

class FindIntersectionsWrapper(object):
    def __init__(self, ent, collidable):
        self.ent = ent
        self.collidable = collidable
    def __call__(self, dims):
        line1, line2 = dims
        return find_intersections(
            line1 + self.ent.position, self.ent.velocity,
            self.ent.acceleration, line2 + self.collidable.position,
            self.collidable.velocity, self.collidable.acceleration, 
            self.ent, self.collidable)

def entity_intersections(ent, collidable):
    find_inter = FindIntersectionsWrapper(ent, collidable)
    pool = multiprocessing.Pool()
    return pool.map(find_inter, product(ent.shape, collidable.shape))

辅助函数find_intersections_wrapper()是必需的,因为Pool.map()期望具有单个参数的函数.

您可能希望将创建pool移出entity_intersections()以仅具有生成进程池一次的开销.

编辑:使用类而不是闭包,因为传递给的可调用对象Pool.map()必须在Windows上可选.