我正在使用a kafka-console-consumer探测kafka主题.
间歇性地,我收到此错误消息,然后是2个警告:
[2018-05-01 18:14:38,888] ERROR [Consumer clientId=consumer-1, groupId=console-consumer-56648] Offset commit failed on partition my-topic-0 at offset 444: The coordinator is not aware of this member. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2018-05-01 18:14:38,888] WARN [Consumer clientId=consumer-1, groupId=console-consumer-56648] Asynchronous auto-commit of offsets {my-topic-0=OffsetAndMetadata{offset=444, metadata=''}} failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll …我们使用Kafka Streams来消费，处理和产生消息，而在PROD env上，我们面临着多个主题的错误：
ERROR org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=app-xxx-StreamThread-3-consumer, groupId=app]
Offset commit failed on partition xxx-1 at offset 13920:
The request timed out.[]
对于负载较小的主题，这些错误很少发生，但是对于负载较高（和峰值）的主题，每个主题每天都会发生数十次错误。主题具有多个分区（例如10个）。似乎此问题不会影响数据处理（尽管有性能），因为在引发异常（即使对于相同的偏移量可能是多个错误）之后，使用者随后重新读取消息并成功处理它。
我看到此错误消息由于PR而出现在kafka-clients版本中，但是在同一用例的早期版本中（在消费者上），类似的消息（）被记录为级别。对于我来说，将日志级别更新为这种用例的警告会更合乎逻辑。1.0.0kafka-clientsErrors.REQUEST_TIMED_OUTOffset commit for group {} failed: {}debug
如何解决这个问题？可能是根本原因？也许更改使用者属性或分区设置可以帮助摆脱此类问题。
我们使用以下实现来创建Kafka流：
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> stream = builder.<String, String>stream(topicName);
stream.foreach((key, value) -> processMessage(key, value));
Topology topology = builder.build();
StreamsConfig streamsConfig = new StreamsConfig(consumerSettings);
new KafkaStreams(streamsTopology, streamsConfig);
我们的Kafka消费者设置：
bootstrap.servers: xxx1:9092,xxx2:9092,...,xxx5:9092
application.id: app
state.dir: /tmp/kafka-streams/xxx
commit.interval.ms: 5000 # also …