Logstash配置:如何从ElastciSearch输出插件调用部分更新API？

Question

我正在尝试创建一个数据管道,其中Logstash jdbc插件每5分钟用SQL查询获取一些数据,ElasticSearch输出插件将数据从输入插件放入ElasticSearch服务器.换句话说,我希望此输出插件部分更新ElasticSearch服务器中的现有文档.我的Logstash配置文件如下所示:

input {
    jdbc {
        jdbc_driver_library => "/Users/hello/logstash-2.3.2/lib/mysql-connector-java-5.1.34.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        jdbc_connection_string => "jdbc:mysql://localhost:13306/mysqlDB”
        jdbc_user => “root”
        jdbc_password => “1234”
        last_run_metadata_path => "/Users/hello/.logstash_last_run_display"
        statement => "SELECT * FROM checkout WHERE checkout_no between :sql_last_value + 1 and :sql_last_value + 5 ORDER BY checkout_no ASC"
        schedule => “*/5 * * * *"
        use_column_value => true
        tracking_column => “checkout_no”
    }
}

output {
    stdout { codec => json_lines }

    elasticsearch {
        action => "update"
        index => "ecs"
        document_type => “checkout”
        document_id => “%{checkout_no}"
        hosts => ["localhost:9200"]
    }
}

问题是ElasticSearch输出插件似乎不会调用部分更新API,例如/ {index}/{type}/{id}/_ update.手动只是列出了诸如动作index,delete,create,update,但它并没有提到每一个动作调用,REST API URL,即)是否update操作调用/ {指数}/{类型}/{ID}/_更新或/ {指数}/{type}/{id} api(upsert).我想从弹性搜索输出插件中调用部分更新api？可能吗？

Answer 1

设置两者doc_as_upsert => true并action => "update"在我的生产脚本中工作.

output {

  elasticsearch {
    hosts => ["es_host"]
    document_id => "%{id}" # !!! the id here MUST be the same
    index => "logstash-my-index"
    timeout => 30
    workers => 1
    doc_as_upsert => true
    action => "update"
  }
}