Python - 在写入单个结果文件时进行多处理

Question

我是多处理程序包的新手,我无法完成任务.

我在对象列表上有很多计算要做.

我需要写下的结果也保存在这些对象中.

一旦过程完成计算(我至少工作的方式,等待所有计算完成),结果应该写在一个文件中.

import multiprocessing 
import time
import csv

class simpl():
    def __init__(self, name, val):
        self.name = name
        self.val = val

def pot_val(inpt):
    print("Process %s\t ..." % (inpt.name))
    old_v = inpt.val
    inpt.val *= inpt.val
    if old_v != 8:
            time.sleep(old_v)
    print("Process %s\t ... Done" % (inpt.name))


def mp_worker(inpt):
    pot_val(inpt)
    return inpt

def mp_handler(data_list):
    p = multiprocessing.Pool(4)
    with open('results.csv', 'a') as f:
        res = p.map_async(mp_worker, data_list)
        results = (res.get())
        for result in results:
            print("Writing result for ",result.name)
            writer= csv.writer(f, lineterminator = '\n', delimiter=";")
            writer.writerow((result.name, result.val))
if __name__=='__main__':
    data = []
    counter=0
    for i in range(10):
        data.append(simpl("name"+str(counter),counter))
        counter += 1    

    for d in data:
        print(d.name, d.val)
    mp_handler(data)

如何将计算结果同时写入一个文件,而不必等待所有进程完成？

Answer 1

你可以使用imap_unordered

def mp_handler(data_list):
    p = multiprocessing.Pool(4)
    with open('results.csv', 'a') as f:
        writer= csv.writer(f, lineterminator = '\n', delimiter=";")
        for result in p.imap_unordered(mp_worker, data_list):
            print("Writing result for ",result.name)
            writer.writerow((result.name, result.val))

用Python 3.3+做得更好

def mp_handler(data_list):
    with multiprocessing.Pool(4) as p:
        with open('results.csv', 'a') as f:
            writer= csv.writer(f, lineterminator = '\n', delimiter=";")
            for result in p.imap_unordered(mp_worker, data_list):
                print("Writing result for ",result.name)
                writer.writerow((result.name, result.val))