不可恢复的错误：ResponseError（'UNBLOCKED 强制解除阻塞操作的阻塞，实例状态已更改（主 -> 副本？）

Question

我正在使用一个 redis docker 容器在我的 python 应用程序中用作 celery 和 flaskk-socketio 的消息/代理队列，最近我的 redis 实例经常崩溃，而且它没有回来。

我不是 redis 专家，我还没有找到问题所在，这是日志，感谢您的帮助！

谢谢

[2019-08-09 03:44:29,930: CRITICAL/MainProcess] Unrecoverable error: ResponseError('UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)',)
Traceback (most recent call last):
  File "/server/env/lib/python3.6/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/server/env/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/server/env/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/server/env/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 322, in start
    blueprint.start(self)
  File "/server/env/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/server/env/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 598, in start
    c.loop(*c.loop_args())
  File "/server/env/lib/python3.6/site-packages/celery/worker/loops.py", line 121, in synloop
    connection.drain_events(timeout=2.0)
  File "/server/env/lib/python3.6/site-packages/kombu/connection.py", line 315, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/server/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 963, in drain_events
    get(self._deliver, timeout=timeout)
  File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 370, in get
    ret = self.handle_event(fileno, event)
  File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 352, in handle_event
    return self.on_readable(fileno), self
  File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 348, in on_readable
    chan.handlers[type]()
  File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 725, in _brpop_read
    **options)
  File "/server/env/lib/python3.6/site-packages/redis/client.py", line 680, in parse_response
    response = connection.read_response()
  File "/server/env/lib/python3.6/site-packages/redis/connection.py", line 629, in read_response
    raise response
redis.exceptions.ResponseError: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
[2019-07-15 07:04:53,803: WARNING/MainProcess] Traceback (most recent call last):
[2019-07-15 07:04:53,803: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 109, in wait
    listener.cb(fileno)
[2019-07-15 07:04:53,803: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/celery/worker/pidbox.py", line 122, in loop
    pass
[2019-07-15 07:04:53,804: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/connection.py", line 798, in __exit__
    self.release()
[2019-07-15 07:04:53,804: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/connection.py", line 372, in release
    self._close()
[2019-07-15 07:04:53,804: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/connection.py", line 338, in _close
    self._do_close_self()
[2019-07-15 07:04:53,804: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/connection.py", line 331, in _do_close_self
    self.transport.close_connection(self._connection)
[2019-07-15 07:04:53,804: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 953, in close_connection
    channel.close()
[2019-07-15 07:04:53,805: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 851, in close
    self.queue_delete(queue, client=client)
[2019-07-15 07:04:53,805: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 542, in queue_delete
    self._delete(queue, exchange, *meta, **kwargs)
[2019-07-15 07:04:53,805: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 810, in _delete
    queue or '']))
[2019-07-15 07:04:53,805: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/redis/client.py", line 1661, in srem
    return self.execute_command('SREM', name, *values)
[2019-07-15 07:04:53,805: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/redis/client.py", line 668, in execute_command
    return self.parse_response(connection, command_name, **options)
[2019-07-15 07:04:53,806: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/redis/client.py", line 680, in parse_response
    response = connection.read_response()
[2019-07-15 07:04:53,806: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/redis/connection.py", line 629, in read_response
    raise response
[2019-07-15 07:04:53,806: WARNING/MainProcess] redis.exceptions.ReadOnlyError: You can't write against a read only replica.
[2019-07-15 07:04:53,806: WARNING/MainProcess] Removing descriptor: 19

Answer 1

celery 代码崩溃可能因各种原因（redis 停机/维护、错误等）而发生，关键是确保你的 celery 进程从这些错误中恢复。

如果您使用 docker-compose，您可以按照本文档restart: unless-stopped中的建议进行添加。您可以在此处了解有关 docker-compose 重启策略的更多信息。

使用此选项，如果 celery 进程由于某种原因崩溃，docker-compose 将重新启动容器。