如何解决 OSError: [Errno 24] 使用 aiohttp 对三个不同 URL 执行 POST 调用时打开的文件过多？

Question

我有 csv 文件，其中一列用作在两个 POST 方法调用中传递的数据。我有三个不同的 URL，需要随后在代码中调用。我使用 aiohttp 定义会话并循环列长度（其单元格值在两个 POST 调用中发送）。从响应中获取的数据存储在一个列表中，并用于具有不同 url 的第三个 POST 调用。

前两个调用似乎执行得很好而且相当快（我打印了它们的响应并进行了验证），但是当执行到第三个调用时，出现此错误：

OSError：[Errno 24]打开的文件太多

我尝试了一些建议connector=aiohttp.TCPConnector(verify_ssl=False)在aiohttp.ClientSession(connector=aiohttp.TCPConnector(verify_ssl=False)) 但这不起作用的解决方案中指定。解决这个问题的最佳方法应该是什么？我当前使用的第二个 URL 是 HTTP URL，即 localhost url，最终将更改为基于 https 云的 url。

下面是描述这种情况的示例代码：

import aiohttp
import pandas as pd
import asyncio
import requests
import json
import time

start_time = time.time()

df = pd.read_csv('Sample.csv', header=None, dtype=str)
RequestBodyvalues = df.iloc[:, [0]].values

async def get_FirstUrlresponse(session, url, requestBody, headers):
    async with session.post(url, data = json.dumps(requestBody), headers = headers) as resp:
        response = await resp.json()
        return response

async def get_SecondUrlresponse(session, url, requestBody, headers):
    async with session.post(url, data = json.dumps(requestBody), headers = headers) as resp:
        response = await resp.json()
        return response

async def get_ThirdUrlresponse(session, url, requestBody, headers):
    async with session.post(url, data = json.dumps(requestBody), headers = headers) as resp:
        response = await resp.json()
        return response

async def main():
    async with aiohttp.ClientSession() as session:
        FirstUrlTasks = []
        SecondUrlTasks = []
        for reqBody in RequestBodyvalues:
            firstUrl = 'https://firstUrl.com/searchByValue'
            secondUrl = 'http://secondUrl.com/someEndpoint'
            requestBody = {'value': reqBody}
            headers = {'Authorization' : 'Bearer Token', 
            'content-type': 'application/json'}

            FirstUrlTasks.append(asyncio.ensure_future(get_FirstUrlresponse(session, firstUrl, requestBody, headers)))
            SecondUrlTasks.append(asyncio.ensure_future(get_SecondUrlresponse(session, secondUrl, requestBody, headers)))
        
        firstUrlResponses = await asyncio.gather(*FirstUrlTasks)
        secondUrlresponses = await asyncio.gather(*SecondUrlTasks)

        valuesForThridUrl = []
        for secondUrlresponse in secondUrlresponses:
            #Logic to fetch values to pass to Third Url stored in list

        ThirdUrlTasks = []
        for value in valuesForThridUrl:
            ThirdUrl = 'https://thirdUrl.com/someEndpoint'
            requestBody = {'reqBody': value}
            headers = {'Authorization' : 'Bearer Token', 
            'content-type': 'application/json'}

            ThirdUrlTasks.append(asyncio.ensure_future(get_ThirdUrlresponse(session, ThirdUrl, requestBody, headers)))

        thirdUrlresponses = await asyncio.gather(*ThirdUrlTasks)

asyncio.run(main())

Answer 1

使用此命令查看打开文件限制

ulimit -n

然后增加限制

ulimit -n NUM