小编Yi *_*hou的帖子

Python异步/等待下载URL列表

我正在尝试从FTP服务器下载超过30,000个文件,并且在使用异步IO进行一些谷歌搜索之后似乎是一个好主意.但是,下面的代码无法下载任何文件并返回超时错误.我真的很感激任何帮助!谢谢!

class pdb:
    def __init__(self):
        self.ids = []
        self.dl_id = []
        self.err_id = []


    async def download_file(self, session, url):
        try:
            with async_timeout.timeout(10):
                async with session.get(url) as remotefile:
                    if remotefile.status == 200:
                        data = await remotefile.read()
                        return {"error": "", "data": data}
                    else:
                        return {"error": remotefile.status, "data": ""}
        except Exception as e:
            return {"error": e, "data": ""}

    async def unzip(self, session, work_queue):
        while not work_queue.empty():
            queue_url = await work_queue.get()
            print(queue_url)
            data = await self.download_file(session, queue_url)
            id = queue_url[-11:-7]
            ID = id.upper()
            if …
Run Code Online (Sandbox Code Playgroud)

python asynchronous python-asyncio aiohttp

7
推荐指数
2
解决办法
4629
查看次数

标签 统计

aiohttp ×1

asynchronous ×1

python ×1

python-asyncio ×1