Kafka Consumer标记协调员2147483647死亡

Question

我使用Kafka Server 0.9与消费者kafka-client版本0.9和kafka-producer 0.8.2.

每件事情都很有效,除非我得到很多关于协调员死于消费者的信息

2016-02-25 19:30:45.046  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.048  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.049  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.050  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.051  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.052  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.053  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.054  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.055  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.056  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.057  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.058  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.059  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.060  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.061  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.062  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.062  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.063  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.064  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.065  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.066  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.067  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.068  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.068  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.069  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.070  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.071  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.072  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.072  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.073  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.074  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.075  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.075  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.076  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.077  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.078  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.079  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.079  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.080  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.081  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.082  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.083  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.083  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.084  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.085  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.086  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.086  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.087  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.088  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.089  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.089  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.090  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.091  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.093  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.094  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.094  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.

我还注意到生产者每隔10分钟就会断开连接,如下所示

2016-03-12 15:55:36 INFO  [pool-1-thread-1] - Fetching metadata from broker id:0,host:192.168.72.30,port:9092 with correlation id 41675 for 1 topic(s) Set(act)
2016-03-12 15:55:36 INFO  [pool-1-thread-1] - Connected to 192.168.72.30:9092 for producing
2016-03-12 15:55:36 INFO  [pool-1-thread-1] - Disconnecting from 192.168.72.30:9092
2016-03-12 15:55:36 INFO  [pool-1-thread-1] - Disconnecting from kafkauk.XXXXXXXXXX.co:9092
2016-03-12 15:55:36 INFO  [pool-1-thread-1] - Connected to kafkauk.XXXXXXXXXX.co:9092 for producing

这是我的制作人配置

metadata.broker.list=192.168.72.30:9092
serializer.class=kafka.serializer.StringEncoder
request.required.acks=1
linger.ms=2000
batch.size=500

和消费者配置

bootstrap.servers: kafkauk.xxxxxxxx.co:9092
group.id: cdrServer
client.id: cdrServer
enable.auto.commit: true
auto.commit.interval.ms: 1000
session.timeout.ms: 30000
key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
value.deserializer: org.apache.kafka.common.serialization.StringDeserializer

我无法弄清楚这些是什么意思,我应该忽略它们,或者我在配置中遗漏了一些东西

---

在我将消费者的kafka更改为调试级别后,我发现了以下内容

2016-03-13 18:21:55.586 DEBUG 5469 --- [      cdrServer] org.apache.kafka.clients.NetworkClient   : Node 2147483647 disconnected.
2016-03-13 18:21:55.586  INFO 5469 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-03-13 18:21:55.586 DEBUG 5469 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Issuing group metadata request to broker 0
2016-03-13 18:21:55.586 DEBUG 5469 --- [      cdrServer] org.apache.kafka.clients.NetworkClient   : Sending metadata request ClientRequest(expectResponse=true
, callback=null, request=RequestSend(header={api_key=3,api_version=0,correlation_id=183025,client_id=cdrServer}, body={topics=[act]}), isInitiatedByNetworkCli
ent, createdTimeMs=1457893315586, sendTimeMs=0) to node 0
2016-03-13 18:21:55.591 DEBUG 5469 --- [      cdrServer] org.apache.kafka.clients.Metadata        : Updated cluster metadata version 296 to Cluster(nodes = [N
ode(0, kafkauk.xxxxxxxxx.co, 9092)], partitions = [Partition(topic = act, partition = 0, leader = 0, replicas = [0,], isr = [0,]])
2016-03-13 18:21:55.592 DEBUG 5469 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Group metadata response ClientResponse(receivedTimeMs=1457
893315592, disconnected=false, request=ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFu
tureCompletionHandler@1e2de777, request=RequestSend(header={api_key=10,api_version=0,correlation_id=183024,client_id=cdrServer}, body={group_id=cdrServer}), c
reatedTimeMs=1457893315586, sendTimeMs=1457893315586), responseBody={error_code=0,coordinator={node_id=0,host=kafkauk.xxxxxxxx.co,port=9092}})

我不确定这是一个网络问题,因为它确实每9分钟发生一次

---

更新

我发现那是直接相关的

connections.max.idle.ms: 300000

什么,然后我会断开此值

Answer 1

当消费者客户端和协调器之间存在网络通信错误时,标记协调器死机(当协调器死亡且组需要重新平衡时,也会发生这种情况).有多种情况(偏移提交请求,获取偏移等)可能导致此问题.我建议你研究造成这种情况的原因

Answer 2

我遇到了同样的问题.最后在遵循Shannon关于TRACING日志的建议之后,我使用了:

logging.level.org.apache.kafka=TRACE

要找出我的客户试图解决Euler:9092协调员...本地名称!所以我注释掉,改变listeners和advertised.listeners价值观server.properties的文件.它现在正在运作!:-)

Answer 3

就我而言，当我尝试手动分配分区时，该消息位于日志中。在我阅读了新消费者的 api 文档后，请遵循以下通知：

消费者还可以使用 allocate(Collection) 手动分配特定分区（类似于旧的“简单”消费者）。在这种情况下，动态分区分配和消费者组协调将被禁用。

也就是说，如果您有这样的代码：

    KafkaConsumer<String, String> consumer = new KafkaConsumer(props);
    consumer.assign( Arrays.asList(
            new TopicPartition("topic", 0),
            new TopicPartition("topic", 1)
    ));

那么消息“Marking the coordinator 2147483647 dead”总是会放入我们的日志中。