我是否会将concurrent.futures与Flask相结合,从而提升性能

Question

我想知道是否可以使用concurrent.futuresFlask.这是一个例子.

import requests
from flask import Flask
from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=10)
app = Flask(__name__)

@app.route("/path/<xxx>")
def hello(xxx):
    f = executor.submit(task, xxx)
    return "OK"

def task():
    resp = requests.get("some_url")
    # save to mongodb

app.run()

任务是IO绑定的,不需要返回值.请求不会经常发生,我估计最多10/s.

我测试了它,它工作.我想知道的是我是否可以通过这种方式使用多线程来提升性能.Flask会以某种方式阻止任务吗？

Answer 1

这取决于比Flask更多的因素,就像你在Flask(gunicorn,gevent,uwsgi,nginx等)面前使用的那些因素一样.如果您发现您对"some_url"的请求确实是一个瓶颈,将其推送到另一个线程可能会提升,但这又取决于您的个人情况; Web堆栈中的许多元素可以使进程"缓慢".

而不是对Flask进程进行多线程处理(可能很快变得复杂),将阻塞I/O推送到辅助进程可能是更好的解决方案.您可以将Redis消息发送到在asyncio事件循环上运行的进程,该进程可以很好地扩展.

app.py

from flask import Flask
import redis

r = redis.StrictRedis(host='127.0.0.1', port=6379)
app = Flask(__name__)

@app.route("/")
def hello():
    # send your message to the other process with redis
    r.publish('some-channel', 'some data')
    return "OK"

if __name__ == '__main__':
    app.run(port=4000, debug=True)

helper.py

import asyncio
import asyncio_redis
import aiohttp

@asyncio.coroutine
def get_page():
    # get some url
    req = yield from aiohttp.get('http://example.com')
    data = yield from req.read()

    # insert into mongo using Motor or some other async DBAPI
    #yield from insert_into_database(data) 

@asyncio.coroutine
def run():
    # Create connection
    connection = yield from asyncio_redis.Connection.create(host='127.0.0.1', port=6379)

    # Create subscriber.
    subscriber = yield from connection.start_subscribe()

    # Subscribe to channel.
    yield from subscriber.subscribe([ 'some-channel' ])

    # Inside a while loop, wait for incoming events.
    while True:
        reply = yield from subscriber.next_published()
        print('Received: ', repr(reply.value), 'on channel', reply.channel)
        yield from get_page()

    # When finished, close the connection.
    connection.close()

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(run())