How to use python multiprocessing module in django view

Question

I have a simple function that go over a list of URLs, using GET to retrieve some information and update the DB (PostgresSQL) accordingly. The function works perfect. However, going over each URL one at a time talking too much time.

Using python, I'm able to do to following to parallel these tasks:

from multiprocessing import Pool

def updateDB(ip):
     code goes here...

if __name__ == '__main__':
    pool = Pool(processes=4)              # process per core
    pool.map(updateDB, ip)

This is working pretty well. However, I'm trying to find how do the same on django project. Currently I have a function (view) that go over each URL to get the information, and update the DB.

The only thing I could find is using Celery, but this seems to be a bit overpower for the simple task I want to perform.

Is there anything simple that i can do or do I have to use Celery?

Answer 1

尽管使用 Celery 似乎有点大材小用，但它是一种众所周知的执行异步任务的方法。本质上，Django 服务于 WSGI 请求-响应周期，它对多处理或后台任务一无所知。

以下是替代选项：

• Django 后台任务- 可能更适合您的情况。
• Redis队列

Answer 2

目前我有一个函数（视图）可以遍历每个 URL 以获取信息并更新数据库。

这意味着响应时间对您来说无关紧要，而不是在后台（异步）执行，如果响应时间减少 4（使用 4 个子进程/线程），您可以在前台执行。如果是这种情况，您可以简单地将示例代码放在您的视图中。喜欢

from multiprocessing import Pool

def updateDB(ip):
     code goes here...

def my_view(request):
    pool = Pool(processes=4)              # process per core
    pool.map(updateDB, ip)
    return HttpResponse("SUCCESS")

但是，如果您想在后台异步执行此操作，则应使用 Celery 或遵循 @BasicWolf 的建议之一。