为什么 asyncio 使用 aiohttp 发出请求仍然使用线程

Question

我认为它ayncio的使用coroutine与线程无关，因为coroutine它是在程序调度程序下运行的一种“线程”，所以每个进程应该只有1个线程运行。但是当我运行使用 python-aiohttp 发出 100 万个请求中的示例时，代码如下所示：

# modified fetch function with semaphore
import random
import asyncio
from aiohttp import ClientSession

async def fetch(url, session):
    async with session.get(url) as response:
        delay = response.headers.get("DELAY")
        date = response.headers.get("DATE")
        print("{}:{} with delay {}".format(date, response.url, delay))
        return await response.read()


async def bound_fetch(sem, url, session):
    # Getter function with semaphore.
    async with sem:
        await fetch(url, session)


async def run(r):
    url = "http://localhost:8080/{}"
    tasks = []
    # create instance of Semaphore
    sem = asyncio.Semaphore(1000)

    # Create client session that will ensure we dont open new connection
    # per each request.
    async with ClientSession() as session:
        for i in range(r):
            # pass Semaphore and session to every GET request
            task = asyncio.ensure_future(bound_fetch(sem, url.format(i), session))
            tasks.append(task)

        responses = asyncio.gather(*tasks)
        await responses

number = 10000
loop = asyncio.get_event_loop()

future = asyncio.ensure_future(run(number))
loop.run_until_complete(future)

通过 Windows 的资源监视器，我发现代码在 1 个进程中创建了 25 个线程。

Answer 1

aiohttp 库默认使用线程进行并发 DNS 解析，以免阻塞 IO 循环，请参阅aiohttp/resolver.py。如果您想要异步 DNS 查找，则需要安装 python 包aiodns，该包又使用pycares.

然后你可以这样做：

async def fetch(url):
    resolver = aiohttp.AsyncResolver()
    connector = aiohttp.TCPConnector(resolver=resolver, family=socket.AF_INET)
    async with aiohttp.ClientSession(connector=connector) as session:
        async with session.get(url) as resp:
            if resp.status == 200:
                print("success!")

如果你想设置AsyncResolver为全局默认值，这对我来说适用于 aiohttp 2.2.3：

import aiohttp.resolver
aiohttp.resolver.DefaultResolver = aiohttp.resolver.AsyncResolver