多处理for循环？

Question

我有一个数组(称为data_inputs)包含数百个天文图像文件的名称.然后操纵这些图像.我的代码工作,需要几秒钟来处理每个图像.但是,它一次只能执行一个图像,因为我通过for循环运行数组:

for name in data_inputs:
    sci=fits.open(name+'.fits')
    #image is manipulated

我没有理由在任何其他图像之前修改图像,因此可以在我的机器上使用所有4个核心,每个核心在不同的图像上运行for循环吗？

我已经阅读了关于multiprocessing模块的内容,但我不确定如何在我的情况下实现它.我渴望开始multiprocessing工作,因为最终我必须在10,000多张图片上运行它.

Answer 1

你可以简单地使用multiprocessing.Pool:

from multiprocessing import Pool

def process_image(name):
    sci=fits.open('{}.fits'.format(name))
    <process>

if __name__ == '__main__':
    pool = Pool()                         # Create a multiprocessing Pool
    pool.map(process_image, data_inputs)  # process data_inputs iterable with pool

Answer 2

你可以使用multiprocessing.Pool:

from multiprocessing import Pool
class Engine(object):
    def __init__(self, parameters):
        self.parameters = parameters
    def __call__(self, filename):
        sci = fits.open(filename + '.fits')
        manipulated = manipulate_image(sci, self.parameters)
        return manipulated

try:
    pool = Pool(8) # on 8 processors
    engine = Engine(my_parameters)
    data_outputs = pool.map(engine, data_inputs)
finally: # To make sure processes are closed in the end, even if errors happen
    pool.close()
    pool.join()

Answer 3

或者

with Pool() as pool: 
    pool.map(fits.open, [name + '.fits' for name in datainput])