asyncio web scraping 101:使用aiohttp获取多个url

Question

在之前的问题中,其中一位作者aiohttp善意地建议使用以下新语法从aiohttp获取多个URL:async withPython 3.5

import aiohttp
import asyncio

async def fetch(session, url):
    with aiohttp.Timeout(10):
        async with session.get(url) as response:
            return await response.text()

async def fetch_all(session, urls, loop):
    results = await asyncio.wait([loop.create_task(fetch(session, url))
                                  for url in urls])
    return results

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    # breaks because of the first url
    urls = ['http://SDFKHSKHGKLHSKLJHGSDFKSJH.com',
            'http://google.com',
            'http://twitter.com']
    with aiohttp.ClientSession(loop=loop) as session:
        the_results = loop.run_until_complete(
            fetch_all(session, urls, loop))
        # do something with the the_results

但是,当其中一个session.get(url)请求中断(如上所述http://SDFKHSKHGKLHSKLJHGSDFKSJH.com)时,错误未得到处理,整个事情就会中断.

我寻找方法来插入关于结果的测试session.get(url),例如为a寻找位置try ... except ...,或者为a if response.status != 200:但我只是不理解如何使用async with,await以及各种对象.

由于async with仍然很新,所以没有很多例子.如果asyncio向导可以显示如何执行此操作,那么对很多人来说会非常有帮助.毕竟,大多数人想要测试的第一件事asyncio就是同时获得多个资源.

目标

我们的目标是检查the_results并快速查看:

• 这个网址失败了(为什么:状态代码,也许是异常名称),或者
• 这个网址工作,这是一个有用的响应对象

Answer 1

我会使用gather而不是wait,它可以将异常作为对象返回,而不会提升它们.然后,您可以检查每个结果,如果它是某个异常的实例.

import aiohttp
import asyncio

async def fetch(session, url):
    with aiohttp.Timeout(10):
        async with session.get(url) as response:
            return await response.text()

async def fetch_all(session, urls, loop):
    results = await asyncio.gather(
        *[fetch(session, url) for url in urls],
        return_exceptions=True  # default is false, that would raise
    )

    # for testing purposes only
    # gather returns results in the order of coros
    for idx, url in enumerate(urls):
        print('{}: {}'.format(url, 'ERR' if isinstance(results[idx], Exception) else 'OK'))
    return results

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    # breaks because of the first url
    urls = [
        'http://SDFKHSKHGKLHSKLJHGSDFKSJH.com',
        'http://google.com',
        'http://twitter.com']
    with aiohttp.ClientSession(loop=loop) as session:
        the_results = loop.run_until_complete(
            fetch_all(session, urls, loop))

测试:

$python test.py 
http://SDFKHSKHGKLHSKLJHGSDFKSJH.com: ERR
http://google.com: OK
http://twitter.com: OK

Answer 2

我距离asyncio专家还很远，但是您想捕获该错误，就需要捕获一个套接字错误：

async def fetch(session, url):
    with aiohttp.Timeout(10):
        try:
            async with session.get(url) as response:
                print(response.status == 200)
                return await response.text()
        except socket.error as e:
            print(e.strerror)

运行代码并打印the_results：

Cannot connect to host sdfkhskhgklhskljhgsdfksjh.com:80 ssl:False [Can not connect to sdfkhskhgklhskljhgsdfksjh.com:80 [Name or service not known]]
True
True
({<Task finished coro=<fetch() done, defined at <ipython-input-7-535a26aaaefe>:5> result='<!DOCTYPE ht...y>\n</html>\n'>, <Task finished coro=<fetch() done, defined at <ipython-input-7-535a26aaaefe>:5> result=None>, <Task finished coro=<fetch() done, defined at <ipython-input-7-535a26aaaefe>:5> result='<!doctype ht.../body></html>'>}, set())

您可以看到我们发现了错误，并且进一步的调用仍成功返回了html。

由于socket.error是 自python 3.3起已弃用的OSError别名，我们可能实际上应该捕获了OSError：

async def fetch(session, url):
    with aiohttp.Timeout(10):
        try:
            async with session.get(url) as response:
                return await response.text()
        except OSError as e:
            print(e)

如果您还想检查响应是否为200，也可以将if设置为try，然后可以使用reason属性获取更多信息：

async def fetch(session, url):
    with aiohttp.Timeout(10):
        try:
            async with session.get(url) as response:
                if response.status != 200:
                    print(response.reason)
                return await response.text()
        except OSError as e:
            print(e.strerror)