如何在python中优雅地记录多个非常相似的事件？

Question

使用pythons logging模块,有没有办法将多个事件收集到一个日志条目中？一个理想的解决方案是扩展python的logging模块或自定义格式化程序/过滤器,因此收集相同类型的日志记录事件在后台发生,并且不需要在代码体中添加任何内容(例如,在每次调用日志记录函数时).

这是一个生成大量相同或非常相似的日志记录事件的示例:

import logging

for i in range(99999): 
    try:
        asdf[i]   # not defined!
    except NameError:
        logging.exception('foo') # generates large number of logging events
    else: pass

# ... more code with more logging ...

for i in range(88888): logging.info('more of the same %d' % i)

# ... and so on ...

所以我们有相同的例外99999次并记录它.如果日志只是说:

ERROR:root:foo (occured 99999 times)
Traceback (most recent call last):
  File "./exceptionlogging.py", line 10, in <module>
    asdf[i]   # not defined!
NameError: name 'asdf' is not defined

INFO:root:foo more of the same (occured 88888 times with various values)

Answer 1

您应该编写消息聚合/统计类而不是尝试挂钩日志系统的单例,但我想您可能有一个使用日志记录的现有代码库.

我还建议你应该实例化你的记录器,而不是总是使用默认的root.在Python记录食谱具有广泛的解释和例子.

以下课程应该按照你的要求做.

import logging
import atexit
import pprint

class Aggregator(object):
    logs = {}

    @classmethod
    def _aggregate(cls, record):
        id = '{0[levelname]}:{0[name]}:{0[msg]}'.format(record.__dict__)
        if id not in cls.logs: # first occurrence
            cls.logs[id] = [1, record]
        else: # subsequent occurrence
            cls.logs[id][0] += 1

    @classmethod
    def _output(cls):
        for count, record in cls.logs.values():
            record.__dict__['msg'] += ' (occured {} times)'.format(count)
            logging.getLogger(record.__dict__['name']).handle(record)

    @staticmethod
    def filter(record):
        # pprint.pprint(record)
        Aggregator._aggregate(record)
        return False

    @staticmethod
    def exit():
        Aggregator._output()



logging.getLogger().addFilter(Aggregator)
atexit.register(Aggregator.exit)

for i in range(99999): 
    try:
        asdf[i]   # not defined!
    except NameError:
        logging.exception('foo') # generates large number of logging events
    else: pass

# ... more code with more logging ...
for i in range(88888): logging.error('more of the same')

# ... and so on ...    

请注意,在程序退出之前,您不会获得任何日志.

运行它的结果是:

    ERROR:root:foo (occured 99999 times)
    Traceback (most recent call last):
      File "C:\work\VEMS\python\logcount.py", line 38, in 
        asdf[i]   # not defined!
    NameError: name 'asdf' is not defined
    ERROR:root:more of the same (occured 88888 times)

Answer 2

创建一个计数器并仅记录它count=1，然后递增并在finally块中写出（以确保无论应用程序崩溃和烧毁有多严重，它都会被记录）。如果您因不同原因出现相同的异常，这当然可能会带来问题，但您始终可以搜索行号以验证它是相同的问题或类似的问题。一个最小的例子：

name_error_exception_count = 0
try:
    for i in range(99999): 
        try:
            asdf[i]   # not defined!
        except NameError:
            name_error_exception_count += 1
            if name_error_exception_count == 1:
                logging.exception('foo') 
        else: pass
except Exception:
    pass  # this is just to get the finally block, handle exceptions here too, maybe
finally:
    if name_error_exception_count > 0:
        logging.exception('NameError exception occurred {} times.'.format(name_error_exception_count))