grok 解析多行，例如异常堆栈跟踪

Question

我的日志文件中有多种（三种）类型的日志。其中一种类型有一些自己的打印 + 异常堆栈跟踪。该示例如下所示：

Multiple lines example:
    2018-04-27 10:53:17 [http-nio-8088-exec-4] - ERROR - app-info-exception-info - params:{"cardid":"111111111","txamt":10,"ip":"192.168.16.89","stationcode":"0002","inputuserid":1,"organcode":"99999"} java.lang.NullPointerException: null
        at com.datalook.group.BusinessHandler.handler(BusinessHandler.java:93) ~[classes/:?]
        at com.datalook.group.BusinessGroupController.businessGroup(BusinessGroupController.java:51) [classes/:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]

我有一个模式来解析它，它是：

#pattern:
(?<timestamp>[\d\-\s\:]+)\s\[(?<threadname>[\w\-\d]+)\]\s-\s(?<loglevel>[\w]+)\s\-\s(?<appinfo>app-info-exception-info)\s-\s(?<params>params):(?<jsonstr>[\"\w\d\,\:\.\{\}]+)\s(?<exceptionname>[\w\d\.]+Exception):\s(?<exceptiondetail>[\w\d\.]+)\n\t(?<extralines>at[\s\w\.\d\~\?\n\t\(\)\_\[\]\/\:\-]+)\n

模式在解析多行异常堆栈跟踪时有错误（实际上不是错误，但没有完全解析或按预期解析），主要在最后两部分（exceptiondetail（在这种情况下为 null）和 extralines（那些以空格或制表符开头的行加上 'at '，或第一行堆栈跟踪之后的行））。有比我更好的主意吗？

在 filebeat.yml 中，我配置了以下内容：

# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  multiline.pattern: '^[[:space:]]'

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  multiline.negate: false

  multiline.match: after

任何改进解析多行（异常堆栈跟踪）的想法？

Answer 1

怎么样让它更简单？使用?将额外数据（所有行以 开头at）分配GREEDYDATA到单个字段中(?m)？

例如，如果这是您的日志，

2018-04-27 10:53:17 [http-nio-8088-exec-4] - ERROR - app-info-exception-info - params:{"cardid":"111111111","txamt":10,"ip":"192.168.16.89","stationcode":"0002","inputuserid":1,"organcode":"99999"} java.lang.NullPointerException: null
        at com.datalook.group.BusinessHandler.handler(BusinessHandler.java:93) ~[classes/:?]
        at com.datalook.group.BusinessGroupController.businessGroup(BusinessGroupController.java:51) [classes/:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]

您可以将其解析为，

%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:threadname}\] - %{LOGLEVEL:loglevel} - app-info-exception-info - params:%{SPACE}\{\"%{DATA:jsondata}\"\} %{DATA:excentionname}: %{DATA:exceptiondetail}\n(?m)%{GREEDYDATA:extralines}

这将输出，

{
  "timestamp": [
    [
      "2018-04-27 10:53:17"
    ]
  ],
  "YEAR": [
    [
      "2018"
    ]
  ],
  "MONTHNUM": [
    [
      "04"
    ]
  ],
  "MONTHDAY": [
    [
      "27"
    ]
  ],
  "HOUR": [
    [
      "10",
      null
    ]
  ],
  "MINUTE": [
    [
      "53",
      null
    ]
  ],
  "SECOND": [
    [
      "17"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "threadname": [
    [
      "http-nio-8088-exec-4"
    ]
  ],
  "loglevel": [
    [
      "ERROR"
    ]
  ],
  "SPACE": [
    [
      ""
    ]
  ],
  "jsondata": [
    [
      "cardid":"111111111","txamt":10,"ip":"192.168.16.89","stationcode":"0002","inputuserid":1,"organcode":"99999"
    ]
  ],
  "excentionname": [
    [
      "java.lang.NullPointerException"
    ]
  ],
  "exceptiondetail": [
    [
      "null"
    ]
  ],
  "extralines": [
    [
      "        at com.datalook.group.BusinessHandler.handler(BusinessHandler.java:93) ~[classes/:?]\n        at com.datalook.group.BusinessGroupController.businessGroup(BusinessGroupController.java:51) [classes/:?]\n        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]\n        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]\n        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]"
    ]
  ]
}

您也可以用(?m)with替换以%{SPACE}将开头的每一行at分成自己的字段。