在 redis db 中迭代所有键和值的更快方法

Question

我有一个大约有 350,000 个键的数据库。目前我的代码只是遍历所有键并从数据库中获取它的值。

然而，这需要将近 2 分钟才能完成，这似乎很慢，redis-benchmark给了 100k reqs/3s。

我看过流水线，但我需要返回每个值，以便我最终得到一个键值对的字典。

目前，如果可能的话，我正在考虑在我的代码中使用线程来加快速度，这是处理这个用例的最佳方法吗？

这是我到目前为止的代码。

import redis, timeit
start_time = timeit.default_timer()
count = redis.Redis(host='127.0.0.1', port=6379, db=9)
keys = count.keys()

data = {}

for key in keys:
    value = count.get(key)
    if value:
        data[key.decode('utf-8')] = int(value.decode('utf-8'))

elapsed = timeit.default_timer() - start_time

print('Time to read {} records: '.format(len(keys)), elapsed)

Answer 1

First, the fastest way is doing all of this inside EVAL.

Next, recommended approach to iterate all keys is SCAN. It would not iterate faster than KEYS, but will allow Redis to process some other actions in between, so it will help with overall application behavior.

The script will be something like local data={} local i=1 local mykeys=redis.call(\"KEYS\",\"*\") for k=1,#mykeys do local tmpkey=mykeys[k] data[i]={tmpkey,redis.call(\"GET\",tmpkey)} i=i+1 end return data, but it will fail if you have keys inaccessible with GET (like sets, lists). You need to add error handling to it. If you need sorting, you can do it either in LUA directly, or later on the client side. The second will be slower, but would not let other users of redis instance wait.

Sample output:

127.0.0.1:6370> eval "local data={} local i=1 local mykeys=redis.call(\"KEYS\",\"*\") for k=1,#mykeys do local tmpkey=mykeys[k] data[i]={tmpkey,redis.call(\"GET\",tmpkey)} i=i+1 end return data" 0
1) 1) "a"
   2) "aval"
2) 1) "b"
   2) "bval"
3) 1) "c"
   2) "cval"
4) 1) "d"
   2) "dval"
5) 1) "e"
   2) "eval"
6) 1) "f"
   2) "fval"
7) 1) "g"
   2) "gval"
8) 1) "h"
   2) "hval"