使用Apache Kafka的Spring Boot：无法读取消息

Question

我目前正在使用Kafka侦听器设置Spring Boot应用程序。我试图只编码消费者。对于生产者，我现在暂时从Kafka控制台手动发送消息。我按照示例进行操作：http : //www.source4code.info/2016/09/spring-kafka-consumer-producer-example.html

我尝试将其作为Spring Boot应用程序运行，但看不到收到的任何消息。在我的本地Kafka主题中已经有一些消息。

C:\software\kafka_2.11-0.10.1.0\kafka_2.11-0.10.1.0\kafka_2.11-0.10.1.0\bin\wind
ows>kafka-console-producer.bat --broker-list localhost:9092 --topic test
this is a message
testing again

我的Spring Boot应用程序是：

@EnableDiscoveryClient
@SpringBootApplication
public class KafkaApplication {

    /**
     * Run the application using Spring Boot and an embedded servlet engine.
     * 
     * @param args
     *            Program arguments - ignored.
     */
    public static void main(String[] args) {
        // Tell server to look for registration.properties or registration.yml
        System.setProperty("spring.config.name", "kafka-server");

        SpringApplication.run(KafkaApplication.class, args);
    }
}

Kafka的配置是：

package kafka;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.IntegerDeserializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.annotation.EnableKafka;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.config.KafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
import org.springframework.kafka.listener.ConcurrentMessageListenerContainer;

import java.util.HashMap;
import java.util.Map;

@Configuration
@EnableKafka
public class KafkaConsumerConfig {
    @Bean
    KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory();
        factory.setConsumerFactory(consumerFactory());
        //factory.setConcurrency(1);
        //factory.getContainerProperties().setPollTimeout(3000);
        return factory;
    }

    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        return new DefaultKafkaConsumerFactory(consumerConfigs());
    }

    @Bean
    public Map<String, Object> consumerConfigs() {
        Map<String, Object> propsMap = new HashMap();
        propsMap.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        //propsMap.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
        //propsMap.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "100");
        //propsMap.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "15000");
        propsMap.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class);
        propsMap.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        //propsMap.put(ConsumerConfig.GROUP_ID_CONFIG, "group1");
        //propsMap.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        return propsMap;
    }

    @Bean
    public Listener listener() {
        return new Listener();
    }
}

Kafka的监听器是：

package kafka;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;

import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.IOException;
import java.util.concurrent.CountDownLatch;
import java.util.logging.Logger;

public class Listener {

    protected Logger logger = Logger.getLogger(Listener.class
            .getName());

    public CountDownLatch getCountDownLatch1() {
        return countDownLatch1;
    }

    private CountDownLatch countDownLatch1 = new CountDownLatch(1);

    @KafkaListener(topics = "test")
    public void listen(ConsumerRecord<?, ?> record) {
        logger.info("Received message: " + record);
        System.out.println("Received message: " + record);
        countDownLatch1.countDown();
    }
}

我是第一次尝试。如果我做错任何事，请告诉我。任何帮助将不胜感激。

Answer 1

您没有设置ConsumerConfig.AUTO_OFFSET_RESET_CONFIG，因此默认值为“最新”。将其设置为“最早”，以便使用者将接收主题中已有的消息。

ConsumerConfig.AUTO_OFFSET_RESET_CONFIG仅在使用者组尚未具有主题分区的偏移量时才生效。如果您已经使用“最新”设置运行了使用者，那么使用其他设置再次运行使用者将不会更改偏移量。使用者必须使用其他组，以便Kafka将为该组分配偏移量。