Python + Gevent + MultiProcessing:我的第一个程序非常简单的初学者问题

Question

有人知道这段代码有什么问题吗？它只是"加载"永远.没有输出."网站"是几十个字符串的列表.

num_worker_threads = 30

def mwRegisterWorker():
    while True:
        try:
            print q.get()
        finally:
            pass

q = multiprocessing.JoinableQueue()
for i in range(num_worker_threads):
     gevent.spawn(mwRegisterWorker)

for site in sites:
    q.put(site)

q.join()  # block until all tasks are done

Answer 1

gevent.spawn()创建greenlets而不是进程(更多:所有greenlet在单个OS线程中运行).所以multiprocessing.JoinableQueue这里不合适.

gevent基于协同多任务处理,即,在调用阻塞函数切换到gevent事件循环之前,其他greenlet将不会运行.例如,conn下面使用patched for gevent套接字方法,允许其他greenlet在等待来自站点的回复时运行.如果没有pool.join()这样就放弃了对运行事件循环的greenlet的控制,就不会建立连接.

要在向多个站点发出请求时限制并发性,您可以使用gevent.pool.Pool:

#!/usr/bin/env python
from gevent.pool import Pool
from gevent import monkey; monkey.patch_socket()
import httplib # now it can be used from multiple greenlets

import logging
info = logging.getLogger().info

def process(site):
    """Make HEAD request to the `site`."""
    conn = httplib.HTTPConnection(site)
    try:
        conn.request("HEAD", "/")
        res = conn.getresponse()
    except IOError, e:
        info("error %s reason: %s" % (site, e))
    else:
        info("%s %s %s" % (site, res.status, res.reason))
    finally:
        conn.close()

def main():
    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(msg)s")

    num_worker_threads = 2
    pool = Pool(num_worker_threads)    
    sites = ["google.com", "bing.com", "duckduckgo.com", "stackoverflow.com"]*3
    for site in sites:
        pool.apply_async(process, args=(site,))
    pool.join()

if __name__=="__main__":
   main()