executor.map和非terating参数

Question

我试图将我的脚本从使用线程转换为更酷的多处理(使用python 3.2和concurrent.futures,但是这段代码崩溃了

        with ThreadPoolExecutor(max_workers=MAX_THREADS) as executor:
            for result in executor.map(lambda h:
                                       validate_hostname(h, pci_ids, options.verbose),
                                       get_all_hostnames()):

我收到错误_pickle.PicklingError: Can't pickle <class 'function'>: attribute lookup builtins.function failed.在阅读这个答案时,我认为问题在于不可能将lambda函数作为参数,executor.map()并且为了使executor.map()我需要开发一个参数函数,但是那些pci_ids并且options.verbose是变量所以我不能将它们指定为固定的帮助功能中的值.

有什么想法怎么办？

Answer 1

要避免pickle错误,必须validate在模块或脚本的顶层定义函数.

由于函数被传递给executor.map它只能接受一个参数,所以让该参数为3元组,(h, pci_ids, verbose).

def validate(arg):
    h, pci_ids, verbose = arg
    return validate_hostname(h, pci_ids, verbose)

with ThreadPoolExecutor(max_workers=MAX_THREADS) as executor:
    for result in executor.map(validate, [(host, pci_ids, options.verbose)
                                          for host in get_all_hostnames()]):