Kafka控制台消费者错误"分区上的偏移提交失败"

Question

我正在使用a kafka-console-consumer探测kafka主题.

间歇性地,我收到此错误消息,然后是2个警告:

[2018-05-01 18:14:38,888] ERROR [Consumer clientId=consumer-1, groupId=console-consumer-56648] Offset commit failed on partition my-topic-0 at offset 444: The coordinator is not aware of this member. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

[2018-05-01 18:14:38,888] WARN [Consumer clientId=consumer-1, groupId=console-consumer-56648] Asynchronous auto-commit of offsets {my-topic-0=OffsetAndMetadata{offset=444, metadata=''}} failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

[2018-05-01 18:14:38,888] WARN [Consumer clientId=consumer-1, groupId=console-consumer-56648] Synchronous auto-commit of offsets {my-topic-0=OffsetAndMetadata{offset=447, metadata=''}} failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

它在警告日志中建议:

这意味着后续调用poll()的时间长于配置的max.poll.interval.ms,这通常意味着轮询循环花费了太多时间进行消息处理.您可以通过增加会话超时或通过max.poll.records减少poll()中返回的批量的最大大小来解决此问题.

所以,我要么增加max.poll.interval.ms要么减少max.poll.records.

请告知每种方法的含义,以及哪种方法在不同情况下更受青睐？

Answer 1

如果增加max.poll.interval.ms"可以花时间处理大批记录",那么如果您可以比较小的批处理更有效地处理更大批量,那么您将获得吞吐量.

减少max.poll.records说"减少记录,以便有足够的时间来处理它们",并且有利于延迟而不是吞吐量.

还要考虑两者都配置正常,但其他一些因素导致poll循环中出现性能问题.我会在更改配置之前先探索它,这样就不会掩盖更大的问题.