使用node-rdkafka重新连接到Kafka是缓慢且不一致的

Question

我有kafka和zookeeper在一些本地docker容器中运行.

我有一个node.js代码库,它使用node-rdkafka作为消费者连接到kafka.我们称这个代码库为"消费者"

消费者使用此处显示的代码连接到kafka:https: //github.com/Blizzard/node-rdkafka/blob/master/examples/consumer-flow.md

尝试连接到现有的Kafka实例时,我得到以下输出:

consumer ready.{"name":"rdkafka#consumer-1"}

但是接收和处理消息的代码永远不会触发,或者在开始触发之前需要几分钟(有时显然是5到10分钟).

为了确保信息的正在生产,我用的脚本kafka-console-consumer.sh与卡夫卡前来旁听的消息流.果然,PLENTY数据即将到来.

似乎连接有时会失败,有时需要很长时间才能连接.

这是这段时间内的kafka日志:

kafka_1      | [2017-04-27 20:55:37,963] INFO [Group Metadata Manager on Broker 1001]: Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
kafka_1      | [2017-04-27 20:55:48,576] WARN Attempting to send response via channel for which there is no open connection, connection id 1 (kafka.network.Processor)
kafka_1      | [2017-04-27 20:55:53,185] INFO [GroupCoordinator 1001]: Preparing to restabilize group video-caption-consumer with old generation 24 (kafka.coordinator.GroupCoordinator)
kafka_1      | [2017-04-27 20:56:18,577] INFO [GroupCoordinator 1001]: Stabilized group video-caption-consumer generation 25 (kafka.coordinator.GroupCoordinator)
kafka_1      | [2017-04-27 20:56:18,626] INFO [GroupCoordinator 1001]: Assignment received from leader for group video-caption-consumer for generation 25 (kafka.coordinator.GroupCoordinator)
kafka_1      | [2017-04-27 20:58:24,951] INFO [GroupCoordinator 1001]: Preparing to restabilize group video-caption-consumer with old generation 25 (kafka.coordinator.GroupCoordinator)
kafka_1      | [2017-04-27 20:58:51,643] INFO [GroupCoordinator 1001]: Stabilized group video-caption-consumer generation 26 (kafka.coordinator.GroupCoordinator)
kafka_1      | [2017-04-27 20:58:51,648] INFO [GroupCoordinator 1001]: Assignment received from leader for group video-caption-consumer for generation 26 (kafka.coordinator.GroupCoordinator)
kafka_1      | [2017-04-27 21:01:14,231] INFO [GroupCoordinator 1001]: Preparing to restabilize group video-caption-consumer with old generation 26 (kafka.coordinator.GroupCoordinator)
kafka_1      | [2017-04-27 21:01:16,414] INFO [GroupCoordinator 1001]: Stabilized group video-caption-consumer generation 27 (kafka.coordinator.GroupCoordinator)
kafka_1      | [2017-04-27 21:01:16,419] INFO [GroupCoordinator 1001]: Assignment received from leader for group video-caption-consumer for generation 27 (kafka.coordinator.GroupCoordinator)

它只是无限地重复这种模式.

想法？

为了它的价值,我也使用了kakfa节点模块一段时间了; 有类似的问题,虽然我不能确定是否同样的问题.

Answer 1

只要没有人有好的答案，我就会分享我的解决方案......

最终我停止在 docker 容器中运行 kafka，这似乎就是原因。相反，我按照“快速入门指南”的老方法运行它。

node-rdkafka这解决了我的问题，所以这个问题可能根本不是我的错。事实上，我们已经node-rdkafka在生产中成功运行了近一年，没有出现任何问题。

最终我没有硬性的、令人满意的答案，只有怀疑和很多试图让kafka在容器中工作的痛苦回忆。当我最终放弃 dockerized kafka 时，我松了口气。似乎只有我的一些同事分享了我的问题。