如何为Scrapy项目设置RotatingFileHandler（或最大大小）

Question

我试图确保日志文件不会超过 1048576 字节。由于Scrapy使用pythonlogging模块，我试图将Scrapy的默认值更改FileHandler为RotatingFileHandler.

我在 Scrapy 文档或设置中找不到任何有关此内容的信息。我的想法是重写__init__()蜘蛛的方法（我有一只蜘蛛）。

def __init__(self, *a, **kw):
    self.logger.logger.root.handlers[-1] = RotatingFileHandler('log.log',maxBytes=1024,backupCount=1)
    super(MainSpider, self).__init__(*a, **kw)

但是 scrapy 在蜘蛛初始化之前记录数据。

2017-10-26 13:16:15 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: engine_bot)
2017-10-26 13:16:15 [scrapy.utils.log] INFO: Overridden settings:
....

有没有办法让这个工作或设置 Scrapy 默认日志文件的最大大小？

Answer 1

您必须按照https://doc.scrapy.org/en/latest/topics/logging.html#module-scrapy.utils.log中所述进行configure_logging调用。install_root_handler=False

import logging
from logging.handlers import RotatingFileHandler

from scrapy.utils.log import configure_logging


# Disable default Scrapy log settings.
configure_logging(install_root_handler=False)

# Define your logging settings.
log_format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
log_level = logging.INFO  # Or better yet get `LOG_LEVEL` from settings.
log_file = 'log.log'  # Or better yet get `LOG_FILE` from settings.

logging.basicConfig(
    format=log_format,
    level=log_level
)

rotating_file_log = RotatingFileHandler(log_file, maxBytes=1024, backupCount=1)
rotating_file_log.setFormatter(logging.Formatter(log_format))

root_logger = logging.getLogger()
root_logger.addHandler(rotating_file_log)

这样，您将登录到控制台（根据basicConfig默认设置；不过，可以删除此处理程序）并旋转文件。