如何调试/修复随机发生的Redis :: TimeoutError？

Question

我有一个运行的rails应用程序,它使用redis很多 - 但是 - 我Redis::TimeoutError不时会看到不少发生在这里和那里.在这种情况下没有模式.它既出现在Web应用程序中,也出现在后台作业中(使用sidekiq进行处理) - 不经常但偶尔会出现.

现在我不知道如何找出这个的根本原因,因此不知道如何解决它.

以下是我的设置的一些背景知识:

redis实例运行在单独的物理服务器上,该服务器连接到私有本地1Gbit网络中的Web服务器和后台服务器.所有服务器都运行ubuntu 12.04.redis版本是2.6.10.我使用像这样的初始化程序从我的rails应用程序(3.2)连接:

require 'redis'
require 'redis/objects'
REDIS = Redis.new(:url => APP_CONFIG['REDIS_URL'])
Redis.current = REDIS

这是输出redis-cli INFO:

# Server
redis_version:2.6.10
redis_git_sha1:00000000
redis_git_dirty:0
redis_mode:standalone
os:Linux 3.2.0-38-generic x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.6.3
process_id:28475
run_id:d89bbb1b81d3169c4228cf23c0988ae437d496a1
tcp_port:6379
uptime_in_seconds:14913365
uptime_in_days:172
lru_clock:1507056

# Clients
connected_clients:233
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:19

# Memory
used_memory:801637360
used_memory_human:764.50M
used_memory_rss:594706432
used_memory_peak:4295394784
used_memory_peak_human:4.00G
used_memory_lua:31744
mem_fragmentation_ratio:0.74
mem_allocator:jemalloc-3.3.0

# Persistence
loading:0
rdb_changes_since_last_save:23166
rdb_bgsave_in_progress:0
rdb_last_save_time:1378219310
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:4
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok

# Stats
total_connections_received:932395
total_commands_processed:3088408103
instantaneous_ops_per_sec:837
rejected_connections:0
expired_keys:31428
evicted_keys:3007
keyspace_hits:124093049
keyspace_misses:53060192
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:17651

# Replication
role:master
connected_slaves:1
slave0:192.168.0.2,6379,online

# CPU
used_cpu_sys:54000.21
used_cpu_user:73692.52
used_cpu_sys_children:36229.79
used_cpu_user_children:420655.84

# Keyspace
db0:keys=1498962,expires=1310

在我的redis配置中,我有以下设置:

\?daemonize yes
pidfile /var/run/redis/redis-server.pid
timeout 0
loglevel notice
databases 1
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /var/lib/redis
slave-serve-stale-data yes
slave-read-only yes
slave-priority 100
maxclients 1000
maxmemory 4GB
maxmemory-policy volatile-lru
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60 

Answer 1

这可能来自许多问题:

• 因为您使用SAVE命令(它在您的conf中设置)生成大量I/O并锤击服务器,尤其是在Amazon上使用EBS卷时.
• 因为你有一个Redis奴隶(和以前一样,SAVE在镜像之前做).
• 因为你KEY *在许多索引上使用的是非常慢的.

我写了一篇关于这些问题的文章,请看这里.