CJW*_*CJW 6 elasticsearch logstash
我已将 ElasticSearch 集群从 6.6 更新到 7.6.2，作为升级的一部分，删除了_all 字段和默认字段。
我更新了集群，检查了日志，果然，我的模板之一抱怨这些字段存在。我的所有其他模板仍然正确收集数据并将数据推送到 ElasticSearch/Kibana。
因此，我使用 API 控制台对 /_template/logstash-qa01-stats 运行 PUT 来更新模板：
{
"template" : "logstash-qa01-stats-*",
"settings" : {
"index.refresh_interval" : "10s"
},
"mappings" : {
"_default_" : {
"_all" : { "enabled" : false, "norms" : false },
"dynamic_templates" : [ {
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": { "type": "text", "index": true }
}
}, {
"byte_fields" : {
"match" : "*",
"match_mapping_type" : "long",
"mapping" : { "type" : "double", "doc_values": true }
}
}, {
"short_fields" : {
"match" : "*",
"match_mapping_type" : "long",
"mapping" : { "type" : "double", "doc_values": true }
}
}, {
"integer_fields" : {
"match" : "*",
"match_mapping_type" : "long",
"mapping" : { "type" : "double", "doc_values": true }
}
}, {
"long_fields" : {
"match" : "*",
"match_mapping_type" : "long",
"mapping" : { "type" : "double", "doc_values": true }
}
}, {
"float_fields": {
"match": "*",
"match_mapping_type": "double",
"mapping": { "type": "double", "doc_values": true }
}
}],
"properties" : {
"@timestamp": { "type" : "date", "format": "date_optional_time" },
"@version" : { "type" : "keyword", "index": true },
"type_instance" : { "type" : "text", "index" : true, "norms" : false,
"fields" : {
"raw" : { "type": "text", "index" : true }
}
}
}
}
}
}
到：
{
"index_pattern": "logstash-qa01-stats-*",
"settings": {
"index.refresh_interval": "10s"
},
"mappings": {
"dynamic_templates": [{
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"type": "string",
"index": true,
"doc_values": true
}
}
}, {
"byte_fields": {
"match": "*",
"match_mapping_type": "long",
"mapping": {
"type": "double",
"doc_values": true
}
}
}, {
"short_fields": {
"match": "*",
"match_mapping_type": "long",
"mapping": {
"type": "double",
"doc_values": true
}
}
}, {
"integer_fields": {
"match": "*",
"match_mapping_type": "long",
"mapping": {
"type": "double",
"doc_values": true
}
}
}, {
"long_fields": {
"match": "*",
"match_mapping_type": "long",
"mapping": {
"type": "double",
"doc_values": true
}
}
}, {
"float_fields": {
"match": "*",
"match_mapping_type": "double",
"mapping": {
"type": "double",
"doc_values": true
}
}
}],
"properties": {
"@timestamp": {
"type": "date",
"format": "date_optional_time"
},
"@version": {
"type": "keyword",
"index": true
},
"type_instance": {
"type": "text",
"index": true,
"norms": false,
"fields": {
"raw": {
"type": "string",
"index": true,
"doc_values": true,
"ignore_above": 256
}
}
}
}
}
}
控制台接受了这一点并回复 200 OK。
我返回日志以确保数据正在处理，现在我得到的是：
[2020-04-15T20:28:52,111][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>64}
[2020-04-15T20:28:52,663][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/][Manticore::SocketTimeout] Read timed out {:url=>https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/, :error_message=>"Elasticsearch Unreachable: [https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2020-04-15T20:28:52,664][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>64}
[2020-04-15T20:28:56,245][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/, :path=>"/"}
[2020-04-15T20:28:56,346][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/"}
一遍又一遍地。除非我重新启动logstash，否则不会将数据记录到ElasticSearch/Kibana，然后推送丢失的数据，但随后停止发送新数据。
这个集群多年来一直很好，我不明白模板的更改怎么会严重破坏它。
我能够访问我的弹性域并获得以下信息：
{
"name" : "instance-0000000105",
"cluster_name" : "ea33a63cd7ed235230b7aedcd62e4c37",
"cluster_uuid" : "Tl84gLdZRI26oPe9FRWVuw",
"version" : {
"number" : "7.6.2",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
"build_date" : "2020-03-26T06:34:37.794943Z",
"build_snapshot" : false,
"lucene_version" : "8.4.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
我的其他环境使用相同的 ElasticSearch 集群，仅当我更新在其环境中报告问题的同一模板时才会受到影响。
有人对这里可能发生的事情有什么建议吗？