Python中的异步HTTP调用

Question

我需要在Python中使用回调类功能,我多次向Web服务发送请求,每次都更改参数.我希望这些请求同时发生而不是顺序发生,所以我希望异步调用该函数.

它看起来像asyncore是我可能想要使用的,但我看到它的工作原理的例子看起来都像是矫枉过正,所以我想知道是否还有另一条道路我应该倒下.关于模块/流程的任何建议？理想情况下,我想以程序方式使用它们而不是创建类,但我可能无法绕过它.

Answer 1

从Python 3.2开始,您可以使用concurrent.futures启动并行任务.

看看这个ThreadPoolExecutor例子:

http://docs.python.org/dev/library/concurrent.futures.html#threadpoolexecutor-example

它产生线程来检索HTML并在收到响应时对其做出反应.

import concurrent.futures
import urllib.request

URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']

# Retrieve a single page and report the url and contents
def load_url(url, timeout):
    conn = urllib.request.urlopen(url, timeout=timeout)
    return conn.readall()

# We can use a with statement to ensure threads are cleaned up promptly
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    # Start the load operations and mark each future with its URL
    future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
    for future in concurrent.futures.as_completed(future_to_url):
        url = future_to_url[future]
        try:
            data = future.result()
        except Exception as exc:
            print('%r generated an exception: %s' % (url, exc))
        else:
            print('%r page is %d bytes' % (url, len(data)))

上面的例子使用了线程.还有一个ProcessPoolExecutor使用进程池而不是线程的类似代码:

http://docs.python.org/dev/library/concurrent.futures.html#processpoolexecutor-example

import concurrent.futures
import urllib.request

URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']

# Retrieve a single page and report the url and contents
def load_url(url, timeout):
    conn = urllib.request.urlopen(url, timeout=timeout)
    return conn.readall()

# We can use a with statement to ensure threads are cleaned up promptly
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    # Start the load operations and mark each future with its URL
    future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
    for future in concurrent.futures.as_completed(future_to_url):
        url = future_to_url[future]
        try:
            data = future.result()
        except Exception as exc:
            print('%r generated an exception: %s' % (url, exc))
        else:
            print('%r page is %d bytes' % (url, len(data)))

Answer 2

你知道eventlet吗？它允许您编写看似同步代码的内容,但让它在网络上异步操作.

以下是超级最小爬虫的示例:

urls = ["http://www.google.com/intl/en_ALL/images/logo.gif",
     "https://wiki.secondlife.com/w/images/secondlife.jpg",
     "http://us.i1.yimg.com/us.yimg.com/i/ww/beta/y3.gif"]

import eventlet
from eventlet.green import urllib2

def fetch(url):

  return urllib2.urlopen(url).read()

pool = eventlet.GreenPool()

for body in pool.imap(fetch, urls):
  print "got body", len(body)

Answer 3

扭曲的框架只是它的门票.但是如果你不想接受它,你也可以使用pycurl,libcurl的包装器,它有自己的异步事件循环并支持回调.