Filebeat 多行 kubernetes 容器日志不起作用

Question

嗨，我在使用 filebeat 和 logstash解析 kubernetes 容器多行时遇到了一些问题。kubernetes 日志文件位于 /var/log/containers/*.log 和 json 行结构中。

是不是我的配置有问题？我错过了什么？

filebeat.yml：

filebeat:
  # List of prospectors to fetch data.
  prospectors:
    -
      paths:
        - /var/log/containers/*.log
      fields: {log_type: containers}
      ignore_older: 5m
      symlinks: true
      json.message_key: log
      json.keys_under_root: true
      json.add_error_key: true
      multiline.pattern: '^\d{4}-\d{2}-\d{2}'
      multiline.match: after
      multiline.negate: true
      document_type: kube-logs
  registry_file: /var/log/containers/filebeat_registry
output:
  logstash:
    hosts: ["logstash-logging:5044"]

logstash.conf:

input {
  beats {
    port => 5044
  }
}

filter {
  if [type] == "kube-logs" {

    date {
      match => ["time", "ISO8601"]
      remove_field => ["time"]
    }
    json {
      source => "message"
    }

    grok {
      match => [ "log", "<SOME_PATTERN>" ]
      overwrite => [ "message" ]
    }

}

Kubernetes 容器：

{"log":"11:11:17,740 |-INFO in ch.qos.logback.core.joran.action.mapWAR - Attaching appender named [FILE-LOG] to Logger[ROOT]\n","stream":"stdout","time":"2017-05-09T11:11:17.742837362Z"}
{"log":"11:11:17,740 |-INFO in ch.qos.logback.classic.joran.action.mapWAR - End of configuration.\n","stream":"stdout","time":"2017-05-09T11:11:17.742840277Z"}
{"log":"11:11:17,741 |-INFO in ch.qos.logback.classic.joran.mapWAR - Registering current configuration as safe fallback point\n","stream":"stdout","time":"2017-05-09T11:11:17.742843277Z"}
{"log":"\n another line","stream":"stdout","time":"2017-05-09T11:11:17.742846485Z"}
{"log":"09-May-2017 11:11:17.756 INFO [localhost-startStop-1] org.apache.catalina.startup.mapWAR nice","stream":"stderr","time":"2017-05-09T11:11:17.756924376Z"}
{"log":"09-May-2017 11:11:17.757 INFO [localhost-startStop-1] org.apache.catalina.startup.mapWAR great","stream":"stderr","time":"2017-05-09T11:11:17.757465828Z"}

Answer 1

我想你还是需要把线放在一起，你能试试这个吗？使用 { 因为日志以 { 开头，而不是您的时间戳格式。

filebeat.prospectors:
- paths:
    - input.json
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match:  after

processors:
- decode_json_fields:
    fields: ['message']
    target: json

output.console.pretty: true