multiprocessing pool.map不按顺序处理列表

Question

我有这个脚本来并行处理一些URL：

import multiprocessing
import time

list_of_urls = []

for i in range(1,1000):
    list_of_urls.append('http://example.com/page=' + str(i))

def process_url(url):
    page_processed = url.split('=')[1]
    print 'Processing page %s'% page_processed
    time.sleep(5)

pool = multiprocessing.Pool(processes=4)
pool.map(process_url, list_of_urls)

该列表是有序的，但是当我运行它时，脚本不会按顺序从列表中选择URL：

import multiprocessing
import time

list_of_urls = []

for i in range(1,1000):
    list_of_urls.append('http://example.com/page=' + str(i))

def process_url(url):
    page_processed = url.split('=')[1]
    print 'Processing page %s'% page_processed
    time.sleep(5)

pool = multiprocessing.Pool(processes=4)
pool.map(process_url, list_of_urls)

相反，我希望它首先处理第1,2,3,4页，然后继续按照列表中的顺序进行处理。是否可以选择执行此操作？

Answer 1

如果您不传递参数 chunksize map 将使用此算法计算块：

chunksize, extra = divmod(len(iterable), len(self._pool) * 4)
if extra:
   chunksize += 1

它将您的可迭代对象切割成 task_batches 并在单独的进程上运行它。这就是为什么它不正常。解决方案是将块声明为 1。

import multiprocessing
import time

list_test = range(10)

def proces(task):
    print "task:", task
    time.sleep(1)

pool = multiprocessing.Pool(processes=3)
pool.map(proces, list_test, chunksize=1)

task: 0
task: 1
task: 2
task: 3
task: 4
task: 5
task: 6
task: 7
task: 8
task: 9