多个处理器记录到同一个旋转文件

Question

我用我的nginx + uwsgi + django网站遇到了问题.我知道django + uwsgi没什么特别的,应该是日志模块本身的东西.

在我的网站中,我使用RotatingFileHandler来记录特殊条目,但是,当uwsgi运行多个工作者处理器时,今天我发现,多个日志文件正在同时更改.例如,这是文件片段:

[root@speed logs]# ls -lth
total 18M
-rw-rw-rw- 1 root root  2.1M Sep 14 19:44 backend.log.7
-rw-rw-rw- 1 root root  1.3M Sep 14 19:43 backend.log.6
-rw-rw-rw- 1 root root  738K Sep 14 19:43 backend.log.3
-rw-rw-rw- 1 root root  554K Sep 14 19:43 backend.log.1
-rw-rw-rw- 1 root root 1013K Sep 14 19:42 backend.log.4
-rw-rw-rw- 1 root root  837K Sep 14 19:41 backend.log.5
-rw-rw-rw- 1 root root  650K Sep 14 19:40 backend.log.2
-rw-rw-rw- 1 root root  656K Sep 14 19:40 backend.log
-rw-r--r-- 1 root root   10M Sep 13 10:11 backend.log.8
-rw-r--r-- 1 root root     0 Aug 21 15:53 general.log
[root@speed-app logs]#

实际上,我将旋转文件设置为10M perfile和最多10个文件.

我google了很多很多人之前打过这个,好像登录模块本身不能支持这个.

我发现有人提到了ConcurrentLogHandler(https://pypi.python.org/pypi/ConcurrentLogHandler/0.9.1).以前有人用这个人吗？我看到它基于文件锁,我不知道这个家伙的表现是不是很好.

或者任何人都有更好的想法将多个uwsig实例记录到相同的旋转文件？

谢谢.韦斯利

Answer 1

Just for the heck of it, here is a complete solution example which uses python StreamHandler, uWSGI "daemonized file logging", and logrotate daemon to log to file with rotation.

As you will see, uWSGI logging captures stdout/stderr from your app and redirects it either to stdout/stderr (by default) or to other logger/handlers as defined.

Setup Django/uWSGI

Your Django settings.py

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'default': {
            'format': '%(asctime)s - %(process)s - %(levelname)s - %(name)s : %(message)s',
        },
    },
    'handlers': {
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
        },
    },
    'root': {
        'handlers': ['console'],
        'level': 'DEBUG',
    },
}

Somewhere in your code

log = logging.getLogger(__name__)
log.info("test log!")

Run uWSGI with some logging params

$ uwsgi --http :9090 --chdir=`pwd -P` --wsgi-file=wsgi.py \
    --daemonize=test.log \  # daemonize AND set log file
    --log-maxsize=10000  \  # a 10k file rotate
    --workers=4             # start 4 workers

Output

Excerpt of test.log

*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 79755)
spawned uWSGI worker 1 (pid: 79813, cores: 1)
spawned uWSGI worker 2 (pid: 79814, cores: 1)
spawned uWSGI worker 3 (pid: 79815, cores: 1)
spawned uWSGI worker 4 (pid: 79816, cores: 1)
spawned uWSGI http 1 (pid: 79817)
2015-10-12 07:55:48,458 - 79816 - INFO - testapp.views : test log!
2015-10-12 07:55:51,440 - 79813 - INFO - testapp.views : test log!
2015-10-12 07:55:51,965 - 79814 - INFO - testapp.views : test log!
2015-10-12 07:55:52,810 - 79815 - INFO - testapp.views : test log!

In the same dir, after a while:

-rw-r-----   1 big  staff   1.0K Oct 12 09:56 test.log
-rw-r-----   1 big  staff    11K Oct 12 09:55 test.log.1444636554

Logrotate

Alternatively, to handle rotating the files yourself, omit the --log-maxsize parameter and use a logrotate config file (/etc/logrotate.d/uwsgi-test-app):

/home/demo/test_django/*log {
    rotate 10
    size 10k
    daily
    compress
    delaycompress
}

Please note, the above values are for example sake, you probably don't want the rotate size at 10k. For more info on the logrotate format, see an example blog post.