使用Kafka 0.8.1和Zookeeper 3.4.6时遇到LeaderNotAvailableException

Question

我按照他们的网站安装了稳定版的kafka(0.8.1和2.9.2 Scala),并使用3节点zookeeper集合(3.4.6)运行它.我尝试创建一个测试主题,但一直看到没有为该主题的分区分配任何领导者:

[kafka_2.9.2-0.8.1]$ ./bin/kafka-topics.sh --zookeeper <zookeeper_ensemble> --describe --topic test-1
Topic:test-1    PartitionCount:1    ReplicationFactor:3 Configs:
    Topic: test-1   Partition: 0    **Leader: none**    Replicas: 0,1,2 **Isr:** 

我试图使用控制台生成器写入主题,但遇到了LeaderNotAvailableException异常:

[kafka_2.9.2-0.8.1]$ ./kafka-console-producer.sh --broker-list <broker_list> --topic test-1

hello world

[2014-04-22 11:58:48,297] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -> 
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2014-04-22 11:58:48,321] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -> 
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2014-04-22 11:58:48,322] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler)

[2014-04-22 11:58:48,445] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -> 
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2014-04-22 11:58:48,467] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -> 
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2014-04-22 11:58:48,467] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler)

[2014-04-22 11:58:48,590] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -> 
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2014-04-22 11:58:48,612] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -> 
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2014-04-22 11:58:48,612] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler)

[2014-04-22 11:58:48,731] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -> 
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2014-04-22 11:58:48,753] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -> 
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2014-04-22 11:58:48,754] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler)

[2014-04-22 11:58:48,876] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -> 
No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

[2014-04-22 11:58:48,877] ERROR Failed to send requests for topics test-1 with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)

[2014-04-22 11:58:48,878] ERROR Error in handling batch of 1 events (kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
    at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
    at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
    at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
    at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
    at scala.collection.immutable.Stream.foreach(Stream.scala:547)
    at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
    at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)

我还应该声明这最初工作了几天,然后突然发现任何创建的主题都有这个缺失的领导者问题.

Answer 1

Kafka使用外部协调框架(默认情况下为Zookeeper)来维护配置.现在配置似乎与Kafka日志数据不同步.在这种情况下,我将删除受影响的主题数据和相关的Zookeeper数据.

对于测试环境:

1. 停止Kafka-server和Zookeeper-server
2. 删除两个服务的数据目录,默认情况下是/tmp/kafka-log和/tmp/zookeeper.
3. 启动Kafka-server和Zookeeper-server再
4. 创建一个新主题

现在,您可以再次使用该主题.

对于生产环境:

由于Kafka主题存储在不同的目录中,因此应删除特定目录.您还应该/brokers/{broker_id}/topics/{broken_topic}使用Zookeeper客户端从Zookeeper中删除.

在您做任何愚蠢的事情之前,请仔细阅读Kafka文档以确保配置结构.Kafka正在推出删除主题功能(KAFKA-330),以便更容易解决问题.

Answer 2

我遇到过同样的问题.事实证明,Kafka要求机器的主机名可以解析为连接回自身.

我在我的机器上更新了主机名,并在重新启动zookeeper和kafka之后,可以正确写入主题.

Answer 3

我已经通过添加进入解决了这个问题/etc/hosts对于127.0.0.1具有完全合格的主机名:

127.0.0.1       x4239433.your.domain.com x4239433

制片人和消费者开始工作正常.

Answer 4

我有同样的问题.最后我不得不删除停止Kafka节点,然后按照这里有关如何删除Kafka主题的建议.一旦我摆脱了破碎的主题,我就能够再次成功启动Kafka.

我想知道是否有更好的方法,以及如何避免将来发生这种情况.

Answer 5

我遇到了同样的问题，解决了JDK从1.7到1.6的问题