Logstash - csv output headers

Question

I'm trying to request database with logstash jdbc plugins and returns a csv output file with headers with logstash csv plugin.

I spent a lot of time on logstash documentation but I'm still missing a point.

With the following logstash configuration, the results give me a file with headers for each row. I couldn't find a way to add the headers for only the first row in the logstash configuration.

Helps very much appreciated.

Output file

_object$id;_object$name;_object$type;nb_surveys;csat_score
2;Jeff Karas;Agent;2;2  
_object$id;_object$name;_object$type;nb_surveys;csat_score
3;John Lafer;Agent;2;2;2;2;$2;2
_object$id;_object$name;_object$type;nb_surveys;csat_score
4;Michele Fisher;Agent;2;2
_object$id;_object$name;_object$type;nb_surveys;csat_score
5;Chad Hendren;Agent;2;78

file: simple-out.conf

input {
    jdbc {
        jdbc_connection_string => "jdbc:postgresql://localhost:5432/postgres"
        jdbc_user => "postgres"
        jdbc_password => "postgres"
        jdbc_driver_library => "/tmp/drivers/postgresql/postgresql_jdbc.jar"
        jdbc_driver_class => "org.postgresql.Driver"
        statement_filepath => "query.sql"
    }
}
output {
    csv {
        fields => ["_object$id","_object$name","_object$type","nb_surveys","csat_score"]
        path => "output/%{team}/output-%{team}.%{+yyyy.MM.dd}.csv"
        csv_options => {
        "write_headers" => true
        "headers" =>["_object$id","_object$name","_object$type","nb_surveys","csat_score"]
        "col_sep" => ";"
        }
    }
}

Thanks

Answer 1

您在输出中获得多个标头的原因是因为 Logstash 没有事件之间的全局/共享状态的概念，每个项目都是独立处理的，因此每次 CSV 输出插件运行时，它的行为都与第一个相同并写入标头。

我遇到了同样的问题，并找到了使用ruby 过滤器的init选项在 logstash 启动时执行一些代码的解决方案。

这是一个示例 logstash 配置：

# csv-headers.conf

input {
    stdin {}
}
filter {
    ruby {
        init => "
            begin
                @@csv_file    = 'output.csv'
                @@csv_headers = ['A','B','C']
                if File.zero?(@@csv_file) || !File.exist?(@@csv_file)
                    CSV.open(@@csv_file, 'w') do |csv|
                        csv << @@csv_headers
                    end
                end
            end
        "
        code => "
            begin
                event['@metadata']['csv_file']    = @@csv_file
                event['@metadata']['csv_headers'] = @@csv_headers
            end
        "
    }
    csv {
        columns => ["a", "b", "c"]
    }
}
output {
    csv {
        fields => ["a", "b", "c"]
        path   => "%{[@metadata][csv_file]}"
    }
    stdout {
        codec => rubydebug {
            metadata => true
        }
    }
}

如果您使用该配置运行 Logstash：

echo "1,2,3\n4,5,6\n7,8,9" | ./bin/logstash -f csv-headers.conf

您将获得一个output.csv包含以下内容的文件：

A,B,C
1,2,3
4,5,6
7,8,9

这也是线程安全的，因为它仅在启动时运行代码，因此您可以使用多个工作线程。

希望能帮助到你！