Kafka "num.partitions" setting in service.properties does not take effect

Question

We use Kafka in Docker container. We create topics automatically if the topic does not exist when producing or consuming messages. We want 3 partitions for the topics, so set

num.partitions=3

in file /etc/kafka/server.properties in the Kafka container. However, it does not take effect. After doing the setting and restarting the container, then try subscribing or publishing on some non-existential topics, the topics are created, but only with one partition.

We tried this on containers created from image confluentinc/cp-kafka:5.1.0 and also on containers created from image confluentinc/cp-enterprise-kafka:5.3.1, and the behaviors were the same.

We tested creating topics with command:

kafka-topics --create --topic my_topic --zookeeper zookeeper:2181 --replication-factor 1 --partitions 3

This correctly created the topic with three partitions. But we need Kafka to create multi-partition topics automatically.

What could cause the problem? Or how to make Kafka auto-create multi-partition topics?

We do not have any dynamic configs. This is verified by running the following commands:

kafka-configs --bootstrap-server kafka:9092 --entity-type brokers --entity-default --describe
kafka-configs --bootstrap-server kafka:9092 --entity-type brokers --entity-name 0 (or other ids) --describe

Those commands return empty results.

Answer 1

这个答案来得有点晚了，但我一直在使用 docker 镜像来解决同样的问题：confluenceinc/cp-enterprise-kafka:5.3.4。

我的解决方案是在我的 docker-compose 中添加一个新的环境变量： KAFKA_NUM_PARTITIONS: 3（或您想要的分区）

这将自动将属性num.partitions添加到 /etc/kafka/ 目录下的kafka.properties文件中。

修改 /etc/kafka/server.properties 中的属性num.partitions对我来说也不起作用。