了解aiohttp.TCPConnector池和连接限制

Question

我正在试验limit和limit_per_host参数aiohttp.connector.TCPConnector.

在下面的脚本中,我connector = aiohttp.connector.TCPConnector(limit=25, limit_per_host=5)转到aiohttp.ClientSession,然后打开2个请求到docs.aiohttp.org和3到github.com.

结果session.request是一个实例aiohttp.ClientResponse,在这个例子中我故意不.close()通过.close()或者调用它__aexit__.我认为这会使连接池保持打开状态,并减少与(host,ssl,port)三倍的可用连接数-1.

下表表示._available_connections()每个请求之后. 即使在完成对docs.aiohttp.org的第二次请求后,为什么数字仍为4？ 这两个连接可能仍然是开放的,尚未访问._content或已关闭.可用连接不应减少1吗？

After Request Num.        To                    _available_connections
1                         docs.aiohttp.org      4
2                         docs.aiohttp.org      4   <--- Why?
3                         github.com            4
4                         github.com            3
5                         github.com            2

此外,为什么._acquired_per_host只包含1个键？ 我想我可能会理解的方法TCPConnector; 是什么解释了上面的行为？

完整脚本:

import aiohttp


async def main():
    connector = aiohttp.connector.TCPConnector(limit=25, limit_per_host=5)

    print("Connector arguments:")
    print("_limit:", connector._limit)
    print("_limit_per_host:", connector._limit_per_host)
    print("-" * 70, end="\n\n")

    async with aiohttp.client.ClientSession(
        connector=connector,
        headers={"User-Agent": "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2225.0 Safari/537.36"},
        raise_for_status=True
    ) as session:

        # Make 2 connections to docs.aiohttp.org and 
        #      3 connections to github.com
        #
        # Note that these instances intentionally do not use
        # .close(), either explicitly or via __aexit__
        # in an async with block

        r1 = await session.request(
            "GET",
            "https://docs.aiohttp.org/en/stable/client_reference.html#connectors"
        )
        print_connector_attrs("r1", session)

        r2 = await session.request(
            "GET",
            "https://docs.aiohttp.org/en/stable/index.html"
        )
        print_connector_attrs("r2", session)

        r3 = await session.request(
            "GET",
            "https://github.com/aio-libs/aiohttp/blob/master/aiohttp/client.py"
        )
        print_connector_attrs("r3", session)

        r4 = await session.request(
            "GET",
            "https://github.com/python/cpython/blob/3.7/Lib/typing.py"
        )
        print_connector_attrs("r4", session)

        r5 = await session.request(
            "GET",
            "https://github.com/aio-libs/aiohttp"
        )
        print_connector_attrs("r5", session)


def print_connector_attrs(name: str, session: aiohttp.client.ClientSession):
    print("Connection attributes for", name, end="\n\n")
    conn = session._connector
    print("_conns:", conn._conns, end="\n\n")
    print("_acquired:", conn._acquired, end="\n\n")
    print("_acquired_per_host:", conn._acquired_per_host, end="\n\n")
    print("_available_connections:")
    for k in conn._acquired_per_host:
        print("\t", k, conn._available_connections(k))
    print("-" * 70, end="\n\n")


if __name__ == "__main__":
    import asyncio
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

输出粘贴在https://pastebin.com/rvfzMTe3上.我把它放在那里而不是在这里,因为线条很长而且不是很可以包装.

Answer 1

为了解决你的主要问题"为什么即使在完成对docs.aiohttp.org的第二次请求之后数字仍然是4？",aiohttp.connector.BaseConnector._release()调用时连接数将减少,如果你要使用async withon session.request() 或者这将被调用显式调用.close()或在您阅读响应内容后.read().或者就像docs.aiohttp.org的情况一样,当服务器发送EOF时发出请求(这发生在服务器不等待您流式传输响应内容但是将其全部发送以响应第一个请求时).这就是这里发生的事情.当你在断点中 aiohttp.connector.BaseConnector._release()检查并检查堆栈时,你可以亲眼看到,你会看到它aiohttp.http_parser.DeflateBuffer.feed_eof()被调用.那aiohttp.streams.StreamReade._eof_callbacks包含aiohttp.client_reqrep.ClientResponse._response_eof()了呼唤self._connection.release()

至于你的第二个问题"为什么._acquired_per_host只包含一把钥匙",这很奇怪.但我在文档中看不出任何相关内容.这是一个私有属性,所以我们不应该搞乱它.它可能只是命名为私有属性.