将字符串输出格式化为 JSON

Question

我正在使用 FastAPI 和 Structlog，想要测试日志格式并将其从纯文本/字符串转换为 JSON 格式，以便日志聚合器平台具有更好的可读性和处理能力。面临某些日志输出以 JSON 形式提供但以纯字符串形式存在的情况。

电流输出

INFO:     127.0.0.1:62154 - "GET /api/preface HTTP/1.1" 200 OK
INFO:     127.0.0.1:62154 - "GET /loader.json HTTP/1.1" 200 OK
INFO:     127.0.0.1:62155 - "GET /hello_world HTTP/1.1" 200 OK
{"key":"test_key","message":"Push to NFS Success","event":"Testing Fast API..","logger":"test_my_api","filename":"main.py","func_name":"Hello_World","process":23760,"module":"docker","thread":23140,"pathname":"D:\\my_work\\fast_api\\main.py","process_name":"SpawnProcess-1","level":"info","time-iso":"2023-06-30T15:25:03.113400Z"}

预期输出：

    {
    "level": "INFO",
    "IP": "127.0 .0 .1: 62154",
    "method": "GET",
    "endpoint": "/loader.json",
    "protocol": "HTTP / 1.1",
    "status_code": 200,
    "status": "OK"
}
    {
    "level": "INFO",
    "IP": "127.0 .0 .1: 62155",
    "method": "GET",
    "endpoint": "/api/preface",
    "protocol": "HTTP / 1.1",
    "status_code": 200,
    "status": "OK"
}

 {
    "level": "INFO",
    "IP": "127.0 .0 .1: 62155",
    "method": "GET",
    "endpoint": "/hello_world",
    "protocol": "HTTP / 1.1",
    "status_code": 200,
    "status": "OK"
}
    {"key":"test_key","message":"Push to NFS Success","event":"Testing Fast API..","logger":"test_my_api","filename":"main.py","func_name":"Hello_World","process":23760,"module":"docker","thread":23140,"pathname":"D:\\my_work\\fast_api\\main.py","process_name":"SpawnProcess-1","level":"info","time-iso":"2023-06-30T15:25:03.113400Z"}

我在这里缺少什么？谢谢 ！

结构体.py

import orjson
import structlog
import logging

## Added only the necessary context.
class StructLogTest:
    def __init__(self, logging_level=logging.DEBUG, logger_name="test"):
        self.logging_level = logging_level
        self.logger_name = logger_name
        StructLogTest.logger_name_var = self.logger_name
        self.configure_structlog(self.logging_level, self.logger_name)

    def logger_name(_, __, event_dict):
        event_dict["test_log"] = StructLogTest.logger_name_var
        return event_dict


    @staticmethod
    def configure_structlog(logging_level, logger_name):
        structlog.configure(
            processors=[
                StructLogTest.logger_name,
                structlog.threadlocal.merge_threadlocal,
                structlog.processors.CallsiteParameterAdder(),
                structlog.processors.add_log_level,
                structlog.stdlib.PositionalArgumentsFormatter(),
                structlog.processors.StackInfoRenderer(),
                structlog.processors.format_exc_info,
                structlog.processors.TimeStamper(fmt="iso", utc=True, key="time-iso"),
                structlog.processors.JSONRenderer(serializer=orjson.dumps),
            ],
            wrapper_class=structlog.make_filtering_bound_logger(logging_level),
            context_class=dict,
            logger_factory=structlog.BytesLoggerFactory(),
        )
        return structlog

    def define_Logger(self, *args, **kwargs):
        return structlog.get_logger(*args, **kwargs)

    def info(self, message, *args, **kwargs):
        return structlog.get_logger().info(message, *args, **kwargs)
    
    and other methods so on..


    

主要.py

from struct import StructLogTest
from fastapi import APIRouter
import requests
from requests.auth import HTTPBasicAuth
from requests import Response

log = StructLogTest(logger_name="test_my_api")
log = log.get_Logger()

@router.get("/hello_world")
def Hello_World():
    logg = log.bind(key=test_key)
    logg.info(
        "Testing Fast API..",
        message=some_other_meaningful_function.dump(),
    )
    return {" Hello World !! "}

Answer 1

Structlog 是一个与 stdlib 完全独立的日志框架logging。它不会为您配置 stdlib 日志框架。Uvicorn 使用 stdlib 日志记录，并且您无法更改分叉和编辑源代码以使用其他日志记录框架。

\n

不过，您在这里确实有一些选择。最简单的就是使用 json 格式化程序配置 stdlib 日志记录，例如 create uvicorn-logconfig.ini：

\n

[loggers]\nkeys=root, uvicorn, gunicorn\n\n[handlers]\nkeys=access_handler\n\n[formatters]\nkeys=json\n\n[logger_root]\nlevel=INFO\nhandlers=access_handler\npropagate=1\n\n[logger_gunicorn]\nlevel=INFO\nhandlers=access_handler\npropagate=0\nqualname=gunicorn\n\n[logger_uvicorn]\nlevel=INFO\nhandlers=access_handler\npropagate=0\nqualname=uvicorn\n\n[handler_access_handler]\nclass=logging.StreamHandler\nformatter=json\nargs=()\n\n[formatter_json]\nclass=pythonjsonlogger.jsonlogger.JsonFormatter\n

\n

它使用python-json-logger中的 stdlib json 格式化程序。现在，使用如下命令运行您的应用程序：

\n

uvicorn --log-config=uvicorn-logconfig.ini main:router\n

\n

来自 uvicorn 和您自己的应用程序代码的日志将呈现为 JSON，前者通过 stdlib 格式化程序，后者通过 structlog 处理器。

\n

这是我在启动应用程序时看到的内容，向http://127.0.0.1:8000/hello_world发出请求以从 structlog 获取“Testing Fast API...”事件，然后使用 Ctrl 关闭 uvicorn +C:

\n

$ uvicorn --log-config=uvicorn-logconfig.ini main:router\n{"message": "Started server process [54894]", "color_message": "Started server process [\\u001b[36m%d\\u001b[0m]"}\n{"message": "Waiting for application startup."}\n{"message": "Application startup complete."}\n{"message": "Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)", "color_message": "Uvicorn running on \\u001b[1m%s://%s:%d\\u001b[0m (Press CTRL+C to quit)"}\n{"test_key":"test_val","message":"some_message","event":"Testing Fast API..","pathname":"/private/tmp/s/main.py","lineno":42,"process_name":"MainProcess","func_name":"Hello_World","filename":"main.py","thread_name":"AnyIO worker thread","thread":12930912256,"module":"main","process":54894,"level":"info","time-iso":"2023-07-02T19:22:58.085382Z"}\n{"message": "127.0.0.1:58519 - \\"GET /hello_world HTTP/1.1\\" 200"}\n^C{"message": "Shutting down"}\n{"message": "Waiting for application shutdown."}\n{"message": "Application shutdown complete."}\n{"message": "Finished server process [54894]", "color_message": "Finished server process [\\u001b[36m%d\\u001b[0m]"}\n

\n

这是 structlog 文档中推荐的最简单方法（请参阅标准库部分中的“不要集成”小节）：

\n

\n

最直接的选项是将标准库配置logging得足够接近structlog正在记录的内容，然后将其保留。

\n

由于这些通常是来自第三方的日志条目，而这些第三方不利用structlog \xe2\x80\x99s 功能，因此令人惊讶的是，这通常是一种完全足够的方法。

\n

例如，如果您在生产中记录 JSON，请配置日志记录以使用python-json-logger使其也打印 JSON，然后调整配置以匹配其输出。

\n

\n

由于 stdlib 日志记录是高度可配置的，并且 structlog 也高度灵活，因此还有其他方法可以更紧密地集成 structlog 和 stdlib。如果有兴趣，请参阅使用基于structlog的格式化程序进行渲染logging。

\n