Logstash 在与 AWS Elasticservice 连接时抛出 401

Question

我有 AWS Elastic 服务域设置，我正在尝试通过 Logstash 将一些数据从 ec2 实例推送到 AWS elasticservice。

我已经使用Open access policy - Allow all traffic. 我还启用Fine Grained Control并设置了一个master user帐户来访问 AWS ES 服务。

Kibana对于常规调用来说，这一切都可以正常工作cURL，但是logstash 会因请求发送到https://<my_es_hostname>/_license响应而失败401。

我无法弄清楚为什么会发生这个电话。当我尝试在浏览器中点击这个时，我得到{"Message":"Your request: '/_license' is not allowed."}

这是我从logstash 获得的示例日志：

[INFO ] 2021-06-02 11:40:18.858 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://<host-name-partxxxx>.us-east-2.es.amazonaws.com:443"]}
[INFO ] 2021-06-02 11:40:19.902 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://<username>:<password>@<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/]}}
[WARN ] 2021-06-02 11:40:20.760 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://<username>:<password>@<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/"}
[INFO ] 2021-06-02 11:40:21.371 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (7.10.2) {:es_version=>7}
[WARN ] 2021-06-02 11:40:21.380 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[ERROR] 2021-06-02 11:40:21.443 [[main]-pipeline-manager] elasticsearch - Unable to get license information {:url=>"https://<username>:<password>@<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/_license'"}
[ERROR] 2021-06-02 11:40:21.449 [[main]-pipeline-manager] elasticsearch - Could not connect to a compatible version of Elasticsearch {:url=>"https://<username>:<password>@<host-name-partxxxx>.us-east-2.es.amazonaws.com:443/"}

这是我的 Logstash 配置：

input {
  jdbc {
     jdbc_driver_class => "org.postgresql.Driver"
     jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/postgresql-42.2.20.jar"
     ...
     <other properties to fetch data>
 }
}
output {
  elasticsearch {
    hosts => ["https://<host-name-partxxxx>.us-east-2.es.amazonaws.com:443"]
    user => "username"
    password => "password"
    ilm_enabled => false
    index => "my_index"
    document_id => "%{id}"
    doc_as_upsert => true
 }
}

Answer 1

问题在于 Logstash Elasticsearch 输出插件尝试验证 URL 上的许可证<hostname>/_license。

\n

请参阅LS 应始终执行 ES 许可证检查 \xc2\xb7 Issue #1004 \xc2\xb7logstash-plugins/logstash-output-elasticsearch以了解报告的问题/修复。

\n

当该修复程序发布后，您可以按照以下步骤使其正常工作：

\n

\n
• 前往/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch
\n
• 打开文件->根据上述 github 问题中建议的修复license_checker.rb更改方法。appropriate_license使方法appropriate_license()在设置时返回 true OSS。
\n

\n