Use django ORM with multiprocessing?

Question

I have a Django ORM database (mysql or sqlite), and would like to process each row with a fairly computationally intensive operation. What I have right now is something like:

entries = Entry.objects.filter(language='')
for e in entry:
    e.language = detect_language(e.text)
    e.save()

If the database was the bottleneck, I would use a transaction to speed it up. However, the detect_language function is what takes the most time. I could try to run the script multiple times in parallel, but that would introduce a race condition.

I think this could be done with multiprocessing using Pool.map() - the main process fetches the DB entries, the child processes run detect_language. I am not sure how to do it in detail, e.g. whether to save the entries in the child process or in the main process.

Is there anything to take care of when passing ORM objects between processes? Can you give a short example how to use the ORM with multiprocessing?

I've just tied it, and something like this seems to work fine. I am still wondering if there is any caveat here, or if this can be improved performance wise (e.g. batching the DB update):

def detect_and_save(obj):
    obj.language = detect_language(obj.text)
    obj.save()

with multiprocessing.Pool(processes=3) as pool:
    pool.map(detect_and_save, entries)

Answer 1

1. 您不需要传递完整的 ORM 对象 - 您只需传递函数所需的参数并将其结果保存到 ORM 对象即可。
2. 您可以使用bulk_update（可从Django 2.2 或作为第三方库获得）来保存在单个查询中。

texts = [e.text for e in entries]
with multiprocessing.Pool() as pool:
    languages = pool.map(detect_language, texts)
for e, l in zip(entries, languages):
    e.language = l
Entry.objects.bulk_update(entries, ['language'])