标签: aiohttp

我想做并行的http请求任务asyncio,但我发现python-requests会阻塞事件循环asyncio.我发现了aiohttp,但它无法使用http代理提供http请求服务.

所以我想知道是否有办法在借助的帮助下进行异步http请求asyncio.

我已经使用Python asyncio和aiohttp 成功构建了一个RESTful微服务,它监听POST事件以从各种馈送器收集实时事件.

然后,它构建一个内存中结构,以便在嵌套的defaultdict/deque结构中缓存最后24h的事件.

现在我想定期检查该结构到光盘,最好使用泡菜.

由于内存结构可能> 100MB,我想避免在检查结构所需的时间内阻止我的传入事件处理.

我宁愿创建结构的快照拷贝(例如深度拷贝),然后花时间将其写入磁盘并在预设的时间间隔内重复.

我一直在寻找关于如何组合线程的例子(并且是一个线程甚至是最好的解决方案吗？)和asyncio用于那个目的但找不到能帮助我的东西.

任何开始使用的指针都非常感谢!

我正在使用aiohttp构建一个API服务器,将TCP请求发送到单独的服务器.发送TCP请求的模块是同步的,并且是出于我的目的的黑盒子.所以我的问题是这些请求阻止了整个API.我需要一种方法将模块请求包装在异步协程中,该协程不会阻止API的其余部分.

所以,仅仅使用sleep一个简单的例子,有没有办法以某种方式将耗时的同步代码包装在非阻塞协程中,如下所示:

async def sleep_async(delay):
    # After calling sleep, loop should be released until sleep is done
    yield sleep(delay)
    return 'I slept asynchronously'

Python版本：3.11

通过安装应用程序的依赖项pip install -r requirements.txt会出现以下错误：

socket.c -o build/temp.linux-armv8l-cpython-311/aiohttp/_websocket.o
aiohttp/_websocket.c:198:12: fatal error: 'longintrepr.h' file not found
#include "longintrepr.h"                                   
          ^~~~~~~                        1 error generated.
error: command '/data/data/com.termux/files/usr/bin/arm-linux-androideabi-clang' 
failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for aiohttp
Failed to build aiohttp
ERROR: Could not build wheels for aiohttp, which is required to install
pyproject.toml-based projects

此错误特定于 Python3.11版本。在 Python 上，3.10.6 …

我有一个异步功能,需要每隔N分钟使用apscheduller运行一次.下面有一个python代码

URL_LIST = ['<url1>',
            '<url2>',
            '<url2>',
            ]

def demo_async(urls):
    """Fetch list of web pages asynchronously."""
    loop = asyncio.get_event_loop() # event loop
    future = asyncio.ensure_future(fetch_all(urls)) # tasks to do
    loop.run_until_complete(future) # loop until done

async def fetch_all(urls):
    tasks = [] # dictionary of start times for each url
    async with ClientSession() as session:
        for url in urls:
            task = asyncio.ensure_future(fetch(url, session))
            tasks.append(task) # create list of tasks
        _ = await asyncio.gather(*tasks) # gather task responses

async def fetch(url, session):
    """Fetch a …

aiohttp的入门文档提供了以下客户端示例:

import asyncio
import aiohttp

async def fetch_page(session, url):
    with aiohttp.Timeout(10):
        async with session.get(url) as response:
            assert response.status == 200
            return await response.read()

loop = asyncio.get_event_loop()
with aiohttp.ClientSession(loop=loop) as session:
    content = loop.run_until_complete(
        fetch_page(session, 'http://python.org'))
    print(content)

他们为Python 3.4用户提供以下注释:

如果您使用的是Python 3.4,请使用@coroutine装饰器替换a yield和async def.

如果我遵循这些说明,我会得到:

import aiohttp
import asyncio

@asyncio.coroutine
def fetch(session, url):
    with aiohttp.Timeout(10):
        async with session.get(url) as response:
            return (yield from response.text())

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    with aiohttp.ClientSession(loop=loop) as session:
        html = loop.run_until_complete(
            fetch(session, 'http://python.org'))
        print(html)

但是,这不会运行,因为 …

我对HTTP服务器aiohttp与蟒蛇-3.6.如何web.Response()通过JSON(来自a dict)返回？

async def api_server(request):
    res = {"q": "qqq", "a": "aaa"}
    return web.Response(res) # <-- as JSON

是否可以使用asyncio进行多个循环？如果回答是肯定的,我该怎么做？我的用例是:*我从异步中的网站列表中提取网址*对于每个"子网址列表",我会在异步/抓取它们

提取网址的示例:

import asyncio
import aiohttp
from suburls import extractsuburls

@asyncio.coroutine
def extracturls(url):
    subtasks = []
    response = yield from aiohttp.request('GET', url)
    suburl_list = yield from response.text()
    for suburl in suburl_list:
        subtasks.append(asyncio.Task(extractsuburls(suburl)))
     loop = asyncio.get_event_loop()
     loop.run_until_complete(asyncio.gather(*subtasks))

 if __name__ == '__main__':
     urls_list = ['http://example1.com', 'http://example2.com']
     for url in url_list: 
          subtasks.append(asyncio.Task(extractsuburls(url)))  
     loop = asyncio.get_event_loop()
     loop.run_until_complete(asyncio.gather(*subtasks))
     loop.close()

如果我执行这段代码,当python尝试启动第二个循环时,我会发现一个错误,就是说循环已经在运行了.

PS:我的模块"extractsuburls"使用aiohttp来执行web请求.

编辑:

好吧,我试过这个解决方案:

import asyncio
import aiohttp
from suburls import extractsuburls

@asyncio.coroutine
def extracturls( url ):
    subtasks = []
    response = yield from aiohttp.request('GET', url) …

如何使用aiohttp在客户端设置每秒的最大请求数(限制它们)？

在之前的问题中,其中一位作者aiohttp善意地建议使用以下新语法从aiohttp获取多个URL:async withPython 3.5

import aiohttp
import asyncio

async def fetch(session, url):
    with aiohttp.Timeout(10):
        async with session.get(url) as response:
            return await response.text()

async def fetch_all(session, urls, loop):
    results = await asyncio.wait([loop.create_task(fetch(session, url))
                                  for url in urls])
    return results

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    # breaks because of the first url
    urls = ['http://SDFKHSKHGKLHSKLJHGSDFKSJH.com',
            'http://google.com',
            'http://twitter.com']
    with aiohttp.ClientSession(loop=loop) as session:
        the_results = loop.run_until_complete(
            fetch_all(session, urls, loop))
        # do something with the the_results

但是,当其中一个session.get(url) …