线程队列工作示例

Question

如何在以下代码中将打开的线程的最大值限制为20？我知道过去曾提出过一些类似的问题,但我特别想知道如何使用队列以及如果可能的工作示例做得最好.

    # b is a list with 10000 items
    threads = [threading.Thread(target=targetFunction, args=(ptf,anotherarg)) for ptf in b]
    for thread in threads:
        thread.start()

    for thread in threads:
        thread.join()

Answer 1

执行此操作的简单方法是使用queue.Queuefor工作并启动线程for _ in range(MAXTHREADS): threading.Thread(target=f, args=(the_queue,)).start().但是,我发现通过子类化更容易阅读Thread.你的旅费可能会改变.

import threading
import queue

class Worker(threading.Thread):
    def __init__(self, q, other_arg, *args, **kwargs):
        self.q = q
        self.other_arg = other_arg
        super().__init__(*args, **kwargs)
    def run(self):
        while True:
            try:
                work = self.q.get(timeout=3)  # 3s timeout
            except queue.Empty:
                return
            # do whatever work you have to do on work
            self.q.task_done()

q = queue.Queue()
for ptf in b:
    q.put_nowait(ptf)
for _ in range(20):
    Worker(q, otherarg).start()
q.join()  # blocks until the queue is empty.

如果你坚持使用一个函数,我建议用你targetFunction知道如何从队列中获取的东西包装你.

def wrapper_targetFunc(f, q, somearg):
    while True:
        try:
            work = q.get(timeout=3)  # or whatever
        except queue.Empty:
            return
        f(work, somearg)
        q.task_done()

q = queue.Queue()
for ptf in b:
    q.put_nowait(ptf)
for _ in range(20):
    threading.Thread(target=wrapper_targetFunc,
                     args=(targetFunction, q, otherarg)).start()
q.join()