Kafka输入到logstash插件

Question

我试图从3个服务器的kafka集群读入logstash,以便将其写入syslog服务器.我已写入syslog,但即使在logstash文档站点上,我也无法找到如何从kafka读入logstash.

Logtash版本:5.2.2 Kafka版本:0.10.2 scala版本:2_11

我去看了api版本,LOGSTASH_CORE_PLUGIN_API ="2.1.12"

这是我尝试使用的配置

input {
#    file
#    {
#        path => "/opt/logstash/NOTICE.TXT"
#
#        #DEBUG below
#        #path => "../fsdfdstt.log"
#        start_position => "beginning"
#        sincedb_path => "/dev/null"
#    }
        kafka
        {
                zk_connect => "localhost:2181"
                topic_id => "kafkatest2"
        }
}
output
{
        syslog
        {
                host => ["targetserver"]
                port => port#
        }
}

但这是我得到的错误......

[2017-03-15T10:24:17,000][ERROR][logstash.inputs.kafka    ] Unknown setting 'zk_connect' for kafka
[2017-03-15T10:24:17,008][ERROR][logstash.inputs.kafka    ] Unknown setting 'topic_id' for kafka
[2017-03-15T10:24:17,015][ERROR][logstash.agent           ] Cannot load an invalid configuration {:reason=>"Something is wrong with your configuration."}

另外,我发现有些人在kafka输入中使用'group_id'.我不确定我的群集是否有groupid.

谢谢,Karan

Answer 1

你的kafka输入配置必须是这样的:

    kafka
    {
            bootstrap_servers => "localhost:9092"
            topics => "kafkatest2"
    }

您不再连接到Zookeeper,而是直接连接到您的某个Kafka代理.此外,topic_id应该读取设置topics.

您可以在以下网址找到最新的配置选项:https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html