标签: python-asyncio

我正在尝试编写一个简单的Web爬虫来测试新的asyncio模块是如何工作的,但是我有些错误.我正在尝试使用单个URL启动爬网程序.该脚本应该下载该页面,<a>在页面上找到任何标签,并安排它们下载.我期望的输出是指示第一页已被下载的一堆行,然后是随机顺序的后续页面(即,当它们被下载时),直到完成所有内容,但实际上它们实际上是按顺序下载的.我对一般的异步和这个模块完全不熟悉所以我确信我只缺少一些基本概念.

到目前为止,这是我的代码:

import asyncio
import re
import requests
import time
from bs4 import BeautifulSoup
from functools import partial

@asyncio.coroutine
def get_page(url, depth=0):
    print('%s: Getting %s' % (time.time(), url))
    page = requests.get(url)
    print('%s: Got %s' % (time.time(), url))
    soup = BeautifulSoup(page.text)
    if depth < 2:
        for a in soup.find_all('a', href=re.compile(r'\w+\.html'))[:3]:
            u = 'https://docs.python.org/3/' + a['href']
            print('%s: Scheduling %s' % (time.time(), u))
            yield from get_page(u, depth+1)
    if depth == 0:
        loop.stop()
    return soup

root = 'https://docs.python.org/3/'
loop = asyncio.get_event_loop()
loop.create_task(get_page(root)) …

在课堂上Wizard,我想将属性设置wand为coroutine返回的值magic.

class Wizard:
    async def acquire_wand(self):
        self.wand = await magic()

但是,这段代码被认为是"糟糕的Python",因为wand它没有被定义__init__.但是我无法定义它__init__,因为它await可能只用于async繁琐的函数.

class Wizard:
    def __init__(self):
        self.wand = None

    async def acquire_wand(self):
        self.wand = await magic()

    async def perform_spell(self):
        if self.wand is None:
            await self.acquire_wand()
        self.wand.wave()

我可以设置wand要None在__init__和使用if self.wand is None:的地方访问它,但这似乎混乱和不便.

我怎样才能确保wand在整个班级中定义？

大家晚上好,我正在尝试创建互联网机器人,我在将我的脚本从python 3.4迁移到3.5或3.6+时遇到了问题.它使用asyncio并且在3.4 python上运行良好但是当我用python3.5 +启动时我得到了错误:RuntimeError: Cannot run the event loop while another loop is running

这是代码方案:

import multiprocessing as mp
import asyncio
import concurrent.futures
import aiohttp

def create_proccesses(separate_loop_creator, coro):
    proccesses = []
    for n in range(2):
        proc = mp.Process(target=separate_loop_creator, args=(coro,))
        proc.start()
        proccesses.append(proc)
    for p in proccesses:
        p.join()

def separate_loop_creator(coro):
    sep_loop = asyncio.new_event_loop()
    asyncio.set_event_loop(sep_loop)
    tasks = [asyncio.async(coro(sep_loop)) for _ in range(100)]
    try:
        sep_loop.run_until_complete(asyncio.wait(tasks))
        sep_loop.close()
    except Exception as err:
        print(err)
        for task in tasks:
            task.cancel()
        sep_loop.close()


@asyncio.coroutine
def manager(exe, loop):
    # some …

您是否认为可以使用asyncio在django中每n秒运行一次任务,这样主进程就不会被阻塞？

例如,在控制台中每5分钟打印一次,例如:

    import asyncio
    from random import randint

    async def do_stuff(something, howmany):   
        for i in range(howmany):
            print('We are doing  {}'.format(something))
            await asyncio.sleep(randint(0, 5))

    if __name__ == '__main__':
        loop = asyncio.get_event_loop()
        work = [
            asyncio.ensure_future(do_stuff('something', 5)),

        ]
        loop.run_until_complete(asyncio.gather(*work))

似乎django将在循环运行时停止工作.即使这可以在开发中工作,当网站在像apache或gunicorn这样的东西上线时它会如何表现？

以前，我用于asyncio.wait_for超时控制，效果很好。最近，我学习了aiohttp软件包，发现它用于asyncio_timeout.timeout超时控制。然后我阅读了asyncio_timeout 的github页面（https://github.com/aio-libs/async-timeout）。作者声称它运行的速度比快asyncio.wait_for。所以我有两个问题：

1. 能asyncio_timeout.timeout完全取代asyncio.wait_for吗？我是否应该全部更换asyncio.wait_for以提高速度？我正在编写一个websocket客户端，asyncio.wait_for当前控制websocket.recv它被频繁调用。
2. 在“用法示例”部分（https://github.com/aio-libs/async-timeout）中，似乎asyncio_timeout.timeout应该将其与一起使用async with。但是，在aiohttp帮助页面中，它们with代替async with（http://aiohttp.readthedocs.io/en/stable/）使用。那么哪一个是正确的呢？

我从文档中了解到，作为asyncio.Future子类，asyncio.Tasks将存储在它们中引发的异常，并且可以在我休闲时检索它们。

但是，在此示例代码中，立即引发了异常：

import asyncio

async def bad_task():
    raise Exception()

async def test():
    loop = asyncio.get_event_loop()
    task = loop.create_task(bad_task())
    await task

    # I would expect to get here
    exp = task.exception()
    # but we never do because the function exits on line 3

loop = asyncio.get_event_loop()
loop.run_until_complete(test())
loop.close()

输出示例（Python 3.6.5）：

python3 ./test.py
Traceback (most recent call last):
  File "./test.py", line 15, in <module>
    loop.run_until_complete(test())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "./test.py", line 9, in test
    await task
  File …

使用Python 3.6和asyncio和aiohttp我写了一个简单的异步程序:

from aiohttp import ClientSession
import asyncio, ssl, time

base_url = 'https://my-base-url.com/api'

async def fetch(session, id):
    query_params = {'qp1':'v1','qp2':'v2', 'id': id}
    async with session.get(base_url, params=query_params, ssl=ssl.SSLContext()) as response:
        res_json = await response.json()
        if response.status == 200:
            time.sleep(2)
            min_rating = res_json.get('minRating')
            max_rating = res_json.get('maxRating')
            print("id = %s, min = %s, max = %s" % (id, min_rating, max_rating))


async def run(ids):
    tasks = []
    async with ClientSession() as session:
        for id in ids:
            task = asyncio.ensure_future(fetch(session, id)) …

我想知道是否有任何方法可以使此脚本更快，例如立即创建1000个帐户，或者至少在几秒钟内创建一个帐户。我已经尝试过自己做一些异步的事情，但这是我所能做到的，我只是异步编程的初学者，所以可以提供任何帮助。

import asyncio
import aiohttp


async def make_numbers(numbers, _numbers):
    for i in range(numbers, _numbers):
        yield i

async def make_account():
   url = "https://example.com/sign_up.php"
   async with aiohttp.ClientSession() as session:
          async for x in make_numbers(35691, 5000000):
              async with  session.post(url, data ={
                    "terms": 1,
                    "captcha": 1,
                    "email": "user%s@hotmail.com" % str(x),
                    "full_name": "user%s" % str(x),
                    "password": "123456",
                    "username": "auser%s" % str(x)
              }) as response:
                    data = await response.text()
                    print("-> Creating account number %d" % x)
                    print (data)

loop = asyncio.get_event_loop()
try:
    loop.run_until_complete(make_account())
finally:
    loop.close()

我有一个将在其中包含aiohttp.ClientSession对象的类。

通常当您使用

async with aiohttp.ClientSession() as session:  
   # some code

由于调用了会话的__aexit__方法，因此该会话将关闭。

我无法使用上下文管理器，因为我想在对象的整个生命周期中保持会话的持久性。

这有效：

import asyncio
import aiohttp

class MyAPI:
    def __init__(self):
        self.session = aiohttp.ClientSession()

    def __del__(self):
        # Close connection when this object is destroyed
        print('In __del__ now')
        asyncio.shield(self.session.__aexit__(None, None, None))



async def main():
    api = MyAPI()

asyncio.run(main())

但是，如果在某些地方引发了异常，则在完成__aexit__方法之前将关闭事件循环。我该如何克服？

堆栈跟踪：

Traceback (most recent call last):
  File "/home/ron/.PyCharm2018.3/config/scratches/async.py", line 19, in <module>
    asyncio.run(main())
  File "/usr/local/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 568, in run_until_complete
    return future.result()
  File "/home/ron/.PyCharm2018.3/config/scratches/async.py", line …

目前我正在使用 async for _ in asyncgen(): pass

我正在寻找“快速路由”实现，用于同步生成器的方法是：

deque(maxlen=0).extend(generator)