__consumer_offsets主题内容的含义

Question

我正在使用 kafka-dump-log 工具查看 __consumer_offsets 主题的内容。有人可以解释该内容的确切含义吗（offset、createtime、isvalid、keysize、valuesize、compresscodec、 ProducerId、 ProducerEpoch、isTransactional、headerKeys、key、offset、payload）？

[root@localhost kafka_2.11-2.0.0]# bin/kafka-dump-log.sh --files data/kafka/__consumer_offsets-32/00000000000000036360.log --value-decoder-class "kafka.serializer.StringDecoder" --offsets-decoder
    Dumping data/kafka/__consumer_offsets-32/00000000000000036360.log
    Starting offset: 36360
    offset: 36360 position: 0 CreateTime: 1623678492136 isvalid: true keysize: 19 valuesize: 28 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: 0 isTransactional: false headerKeys: [] key: offset::b0147:test:48 payload: 35
    offset: 36361 position: 0 CreateTime: 1623678492136 isvalid: true keysize: 19 valuesize: 28 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: 1 isTransactional: false headerKeys: [] key: offset::b0147:test:64 payload: 34
    offset: 36362 position: 0 CreateTime: 1623678492136 isvalid: true keysize: 19 valuesize: 28 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: 2 isTransactional: false headerKeys: [] key: offset::b0147:test:14 payload: 34
    ...
    offset: 36460 position: 5497 CreateTime: 1623678503139 isvalid: true keysize: 9 valuesize: 24 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: [] key: {"metadata":"b0147"} payload: {"protocolType":"consumer","protocol":null,"generationId":2,"assignment":"{}"}
    offset: 36461 position: 5598 CreateTime: 1623678554656 isvalid: true keysize: 19 valuesize: -1 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: []
    ...

Answer 1

您看到的字段并不特定于__consumer_offsets，它们只是日志中有关每个消息/批次的元数据，您将获得所有主题的它们，请参阅https://kafka.apache.org/documentation/#recordbatch

如果你想探索它的内容，__consumer_offsets你需要知道它包含两种类型的记录：

• OffsetMetadata：这是有关组提交的偏移量的信息
• GroupMetadata：这是有关消费者组成员的信息

要显示 OffsetMetadata，您可以使用：

./bin/kafka-console-consumer.sh \
  --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" \
  --bootstrap-server localhost:9092 --topic __consumer_offsets

要显示 GroupMetadata，您可以使用：

./bin/kafka-console-consumer.sh \
  --formatter "kafka.coordinator.group.GroupMetadataManager\$GroupMetadataMessageFormatter" \
  --bootstrap-server localhost:9092 --topic __consumer_offsets