下载使用队列扭曲的文件

Question

我想使用twisted和(例如)20个客户端线程从队列中下载许多文件.任何例子？

Answer 1

from twisted.internet.defer import inlineCallbacks, DeferredQueue

@inlineCallbacks
def worker(queue):
    while 1:
        url = yield queue.get() # wait for a url from the queue

        if url is None: # insert None into the queue to kill workers
            queue.put(None)
            return # done

        data = yield download(url) # download the file
        process(data) # do stuff with it


queue = DeferredQueue() # your queue

# make workers
MAX = 20
workers = [worker(queue) for _ in range(MAX)]