更新：流畅地将 JSON 日志解析为文本

Question

我正在使用 fluidd 将日志发送到 ES。应用程序日志采用 JSON 格式。这是其中之一

{"@timestamp":"2022-05-06T06:02:10.669Z", "log.level": "INFO", "message":"INFO Health check ok", "ecs.version": "1.2.0","service.name":"spring-boot-application","event.dataset":"spring-boot-application","process.thread.name":"http-nio-9001-exec-7","log.logger":"com.app.designer.rules.controller.RulesController","transaction.id":"15e36b5ef6cc69dd","trace.id":"fd275718e25c061d309954773985b101"}

在 Kibana 中，我可以看到日志

JSON日志是一个key的值log。但是，我们需要 JSON 日志键值，正如Field and Value 请建议的那样。流畅的位配置图是：

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-designer
data:
  fluent-bit-service.conf: |
    [SERVICE]
        Flush        1
        Daemon       Off
        Log_Level    error
        Parsers_File parsers.conf

  fluent-bit-input.conf: |
    [INPUT]
        Name              tail
        Path              /app/logs/designer-logs.log.json
        Multiline         On
        Parser_Firstline  multiline_pattern
        Refresh_Interval  5
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Path_key          FileName

  fluent-bit-filter.conf: |
    [FILTER]
        Name    modify
        Match   *
        Add     PodName ${POD_NAME}
        Add     Namespace ${NAMESPACE_NAME}

  fluent-bit-output.conf: |
    [OUTPUT]   
       Name    forward
       Match   *
       Host    {{ .Values.efkCluster }}-fluentd-aggregator.{{ .Values.efkCluster }}
       Port    24224


  fluent-bit.conf: |
    @INCLUDE fluent-bit-service.conf
    @INCLUDE fluent-bit-input.conf
    @INCLUDE fluent-bit-filter.conf
    @INCLUDE fluent-bit-output.conf

  parsers.conf: |
    [PARSER]
        Name         multiline_pattern
        Format       regex
        Regex        ^\[(?<LogDate>\d{2}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}\.\d{3}\+\d{4})\] (?<LogLevel>[^ ]*) * (?<Thread>\S+) (?<Username>[^\s]*?) (?<Logger>[A-Za-z0-9$_.]+) -(\r\n|\r|\n)*(?<Message>.*)
        time_key     LogDate
        time_format  %d-%m-%y %H:%M:%S.%L%z

Answer 1

我将解析器配置从正则表达式更改为json 解析器以使其正常工作。

请找到 Fluentbit 配置图：

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-designer
data:
  fluent-bit-service.conf: |
    [SERVICE]
        Flush        1
        Daemon       Off
        Log_Level    error
        Parsers_File parsers.conf

  fluent-bit-input.conf: |
    [INPUT]
        Name              tail
        Path              /app/logs/app.log
        Parser            json_parser
        Refresh_Interval  5
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Path_key          FileName

  fluent-bit-filter.conf: |
    [FILTER]
        Name    modify
        Match   *
        Add     PodName ${POD_NAME}
        Add     Namespace ${NAMESPACE_NAME}

  fluent-bit-output.conf: |
    [OUTPUT]   
       Name    forward
       Match   *
       Host    {{ .Values.efkCluster }}-fluentd-aggregator.{{ .Values.efkCluster }}
       Port    24224


  fluent-bit.conf: |
    @INCLUDE fluent-bit-service.conf
    @INCLUDE fluent-bit-input.conf
    @INCLUDE fluent-bit-filter.conf
    @INCLUDE fluent-bit-output.conf

  parsers.conf: |
    [PARSER]
        Name         json_parser
        Format       json
        Time_Key     time
        Time_Format  %Y-%m-%dT%H:%M:%S.%L
        Time_Keep    On
        # Command      |  Decoder | Field | Optional Action
        # =============|==================|=================
        Decode_Field_As   escaped_utf8    log    do_next
        Decode_Field_As   json       log

在基巴纳