小编xss*_*ssl的帖子

我正在写蜘蛛来抓取网页.我知道asyncio可能是我最好的选择.所以我使用协同程序异步处理工作.现在我抓住了关于如何通过键盘中断退出程序的问题.所有工作完成后,该程序可能会关闭.源代码可以在python 3.5中运行,并在下面附上.

import asyncio
import aiohttp
from contextlib import suppress

class Spider(object):
    def __init__(self):
        self.max_tasks = 2
        self.task_queue = asyncio.Queue(self.max_tasks)
        self.loop = asyncio.get_event_loop()
        self.counter = 1

    def close(self):
        for w in self.workers:
            w.cancel()

    async def fetch(self, url):
        try:
            async with aiohttp.ClientSession(loop = self.loop) as self.session:
                with aiohttp.Timeout(30, loop = self.session.loop):
                    async with self.session.get(url) as resp:
                        print('get response from url: %s' % url)
        except:
            pass
        finally:
            pass

    async def work(self):
        while True:
            url = await self.task_queue.get()
            await self.fetch(url)
            self.task_queue.task_done()

    def assign_work(self): …