FastAPI：由于 10 秒超时，某些请求失败

Question

我们在生产中部署了一个使用 FastAPI 的模型预测服务，不幸的是，一些请求由于 10 秒超时而失败。就并发请求而言，我们通常每秒仅加载大约 2/3 个请求，因此我认为这不会对 FastAPI 造成太大压力。我们尝试做的第一件事是将 FastAPI 框架与模型本身隔离，当我们执行一些跟踪时，我们注意到这个部分花费了大量时间（6 秒）starlette.exceptions:ExceptionMiddleware.__call__：。

\n

我们使用的 Gunicorn 配置似乎也没有帮助：

\n

"""gunicorn server configuration."""\nimport os\n\xe2\x80\x8b\nthreads = 2\nworkers = 4\ntimeout = 60\nkeepalive = 1800\ngraceful_timeout = 1800\nbind = f":{os.environ.get(\'PORT\', \'80\')}"\nworker_class = "uvicorn.workers.UvicornWorker"\n

\n

非常感谢有关上述部分的含义以及在不太繁重的负载下导致某些请求超时问题的原因的一些指导。

\n

\n

\n

Answer 1

关于上述部分含义的指导

here you have the official gunicorn config file with lot of explainations included.

since you use gunicorn to manager uvicorn workers, forcing your timeout to 60 sec should work just fine for lnog running tasks (you should think about using a asynchronous task queue or job queue like celery)

but what is returning your route ? first thing would be to see the error thrown by your api

starlette.exceptions:ExceptionMiddleware.call

Since you have expanded the list you can see that what take the most time (as expected) is not fastapi nor starlette but you function in app.api.routes.predictions.

so I wouldn't think that would be too much strain on FastAPI

it is not too much strain of fastapi since it is not involved in your request treatment. Remember that fastapi is "just" a framework so when your function take time it's your function/development that's at fault.

here it can be one or a combinaison of thoses things that cause long runing tasks:

• sync route
• blocking I/O function or treatment in your route function
• prediction algo that take a lot of time (too much maybe)
• bad worker class configuration for your type of treatment

当你经常做 AI 或 NLP 的东西时，如果需要大量的处理时间，关于在 api 中集成此类模型，你可以使用像celery这样的任务队列。如果你的 api 没有错误并且你的路由没有返回错误，只是需要花费很多时间，你应该考虑实现任务队列。