gunicorn 访问日志格式

Question

我打算在 kubernetes 上通过 gunicorn 运行烧瓶。为了正确记录日志，我想在 json 中输出我的所有日​​志。

目前我正在使用 minikube 和https://github.com/inovex/kubernetes-logging进行测试，以流畅地收集日志。

由于使用 Flask 和 gunicorn 进行 JSON 格式的日志记录，我设法正确格式化了错误日志（回溯）

我仍在为访问日志格式而苦苦挣扎。我指定了以下 gunicorn 访问日志格式：

access_log_format = '{"remote_ip":"%(h)s","request_id":"%({X-Request-Id}i)s","response_code":"%(s)s","request_method":"%(m)s","request_path":"%(U)s","request_querystring":"%(q)s","request_timetaken":"%(D)s","response_length":"%(B)s"}'

并且生成的日志是json格式的。但是消息部分（基于 access_log_format 的格式）现在包含转义的双引号，并且不会被 fluentd / ELK 解析到它的各个字段中

{"tags": [], "timestamp": "2017-12-07T11:50:20.362559Z", "level": "INFO", "host": "ubuntu", "path": "/usr/local/lib/python2.7/dist-packages/gunicorn/glogging.py", "message": "{\"remote_ip\":\"127.0.0.1\",\"request_id\":\"-\",\"response_code\":\"200\",\"request_method\":\"GET\",\"request_path\":\"/v1/records\",\"request_querystring\":\"\",\"request_timetaken\":\"19040\",\"response_length\":\"20\"}", "logger": "gunicorn.access"}

谢谢 Jpw

Answer 1

最简单的解决方案是将外部单引号更改为双引号，将内部双引号更改为单引号，如下所述。

--access-logformat  "{'remote_ip':'%(h)s','request_id':'%({X-Request-Id}i)s','response_code':'%(s)s','request_method':'%(m)s','request_path':'%(U)s','request_querystring':'%(q)s','request_timetaken':'%(D)s','response_length':'%(B)s'}"

以下是示例日志

{'remote_ip':'127.0.0.1','request_id':'-','response_code':'404','request_method':'GET','request_path':'/test','request_querystring':'','request_timetaken':'6642','response_length':'233'}
{'remote_ip':'127.0.0.1','request_id':'-','response_code':'200','request_method':'GET','request_path':'/','request_querystring':'','request_timetaken':'881','response_length':'20'}

Answer 2

我正在寻找在日志配置文件中有用的东西。另外，我不喜欢手动构建 json 格式。
解决方案：gunicorn 的所有日志记录参数都可以在记录的参数中找到。因此，让我们从那里获取字段，并让 pythonjsonlogger 为我们完成其余的工作。

格式化类

from pythonjsonlogger.jsonlogger import JsonFormatter, merge_record_extra


class GunicornLogFormatter(JsonFormatter): 
    def add_fields(self, log_record, record, message_dict):
        """
        This method allows us to inject gunicorn's args as fields for the formatter
        """
        super(GunicornLogFormatter, self).add_fields(log_record, record, message_dict)
        for field in self._required_fields:
            if field in self.rename_fields:
                log_record[self.rename_fields[field]] = record.args.get(field)
            else:
                log_record[field] = record.args.get(field)

示例日志记录配置文件

version: 1
disable_existing_loggers: False
formatters:
  gunicorn_json:
    (): GunicornLogFormatter
    format: '%(h)s %(r)s %({x-request-id}i)s'
    datefmt: '%Y-%m-%dT%H:%M:%S%z'
    rename_fields:
      levelname: level
      '{x-request-id}i': request_id
      r: request

handlers:
  json-gunicorn-console:
    class: logging.StreamHandler
    level: INFO
    formatter: gunicorn_json
    stream: ext://sys.stdout

loggers:
  gunicorn.access:
    level: INFO
    handlers:
      - json-gunicorn-console

样本日志

{"h": "127.0.0.1", "request": "GET /login?next=/ HTTP/1.1", "request_id": null}
{"h": "127.0.0.1", "request": "GET /static/css/style.css HTTP/1.1", "request_id": null}
{"h": "127.0.0.1", "request": "GET /some/random/path HTTP/1.1", "request_id": null}
{"h": "127.0.0.1", "request": "GET /some/random/path HTTP/1.1", "request_id": "123123123123123123"}