在Django admin中上传文件时出现UnicodeEncodeError

Question

我的问题类似于此处报道的问题,但建议的解决方案对我不起作用.我正在尝试通过Django管理应用程序上传名为"Testaråäö.txt"的文件.

我在Debian 6服务器上运行Django 1.3.1和Gunicorn 0.13.4以及Nginx 0.7.6.7.数据库是PostgreSQL 8.4.9.其他Unicode数据保存到数据库没有问题,所以我猜问题必须以某种方式与文件系统.

我已经定了

http {
    charset utf-8;
}

在我的nginx.conf中.LC_ALL和LANG设置为'sv_SE.UTF-8'.运行'locale'验证这一点.我甚至尝试在我的nginx init脚本中设置LC_ALL和LANG,以确保正确设置了语言环境.

这是追溯:

Traceback (most recent call last):

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/contrib/admin/options.py", line 307, in wrapper
return self.admin_site.admin_view(view)(*args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/utils/decorators.py", line 93, in _wrapped_view
response = view_func(request, *args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/views/decorators/cache.py", line 79, in _wrapped_view_func
response = view_func(request, *args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/contrib/admin/sites.py", line 197, in inner
return view(request, *args, **kwargs)

File "/srv/django/letebo/app/cms/admin.py", line 81, in change_view
return super(PageAdmin, self).change_view(request, obj_id)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/utils/decorators.py", line 28, in _wrapper
return bound_func(*args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/utils/decorators.py", line 93, in _wrapped_view
response = view_func(request, *args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/utils/decorators.py", line 24, in bound_func
return func(self, *args2, **kwargs2)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/transaction.py", line 217, in inner
res = func(*args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/contrib/admin/options.py", line 985, in change_view
self.save_formset(request, form, formset, change=True)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/contrib/admin/options.py", line 677, in save_formset
formset.save()

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/forms/models.py", line 482, in save
return self.save_existing_objects(commit) + self.save_new_objects(commit)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/forms/models.py", line 613, in save_new_objects
self.new_objects.append(self.save_new(form, commit=commit))

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/forms/models.py", line 717, in save_new
obj.save()

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/base.py", line 460, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/base.py", line 504, in save_base
self.save_base(cls=parent, origin=org, using=using)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/base.py", line 543, in save_base
for f in meta.local_fields if not isinstance(f, AutoField)]

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/fields/files.py", line 255, in pre_save
file.save(file.name, file, save=False)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/fields/files.py", line 92, in save
self.name = self.storage.save(name, content)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/core/files/storage.py", line 48, in save
name = self.get_available_name(name)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/core/files/storage.py", line 74, in get_available_name
while self.exists(name):

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/core/files/storage.py", line 218, in exists
return os.path.exists(self.path(name))

File "/srv/.virtualenvs/letebo/lib/python2.6/genericpath.py", line 18, in exists
st = os.stat(path)

UnicodeEncodeError: 'ascii' codec can't encode characters in position 52-54: ordinal not in range(128)

更新:我尝试在打开调试的情况下运行Gunicorn,文件上传没有任何问题.我想这肯定意味着问题出在Nginx上.尽管如此,仍然可以打败我.以下是Gunicorn和Nginx的原始响应标题,如果它有任何意义:

Gunicorn:

HTTP/1.1 302 FOUND
Server: gunicorn/0.13.4
Date: Thu, 09 Feb 2012 14:50:27 GMT
Connection: close
Transfer-Encoding: chunked
Expires: Thu, 09 Feb 2012 14:50:27 GMT
Vary: Cookie
Last-Modified: Thu, 09 Feb 2012 14:50:27 GMT
Location: http://my-server.se:8000/admin/cms/page/15/
Cache-Control: max-age=0
Content-Type: text/html; charset=utf-8
Set-Cookie: messages="yada yada yada"; Path=/

Nginx的:

HTTP/1.1 500 INTERNAL SERVER ERROR
Server: nginx/0.7.67
Date: Thu, 09 Feb 2012 14:50:57 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Vary: Cookie

500

Answer 1

似乎运行django服务的服务是一个常见的问题,无论是apache,gunicorn + supervisor,daemontools等......并不总是使用正确的环境变量.

在这种情况下,gunicorn由主管启动,因此您必须明确告诉主管使用环境选项使用utf-8:

environment=LANG=en_US.UTF-8, LC_ALL=en_US.UTF-8, LC_LANG=en_US.UTF-8

Answer 2

当尝试上传包含非 ASCII 字符的文件名时，我遇到了与 genericpath.py 相同的问题，给出了 UnicodeEncodeError 。

我使用的是 nginx、uwsgi 和 django 以及 python 2.7。

本地一切正常，但服务器上不行

以下是我采取的步骤：

1. 添加到/etc/nginx/nginx.conf（没有修复问题）

   http {
        charset utf-8;
   }
   

2. 我将此行添加到 etc/default/locale （没有解决问题）

   LANGUAGE="en_US.UTF-8"
   

3. 我按照“成功”标题下列出的说明进行操作 https://code.djangoproject.com/wiki/ExpectedTestFailures （没有解决问题）

   aptitude install language-pack-en-base
   

4. 在此票证中找到 https://code.djangoproject.com/ticket/17816 ，建议测试服务器上的视图以了解区域设置信息发生的情况

姜戈视图

import locale

def locales(request):
    """Display the locales"""
    locales = "Current locale: %s %s -- Default locale: %s %s" % (
        locale.getlocale() + locale.getdefaultlocale())
    default_encoding =  sys.getdefaultencoding()
    file_system_encoding = sys.getfilesystemencoding()

    context = {
        'locales': locales,
        'default_encoding': default_encoding,
        'file_system_encoding': file_system_encoding,  # affects file uploads
    }
    return render(request, 'testing/locales.html', context)

Django 模板

<h2>Locales</h2>
<p>{{ locales }}</p>

<h2>Default Encoding</h2>
<p>{{ default_encoding }}</p>

<h2>File System Encoding</h2>
<p>{{ file_system_encoding }}</p>

对我来说，问题是我的 Ubuntu 服务器上没有语言环境，也没有默认语言环境（尽管我的本地 OSX 开发机器上确实有它们），那么具有非 ASCII 文件名/路径的文件将无法正确上传，Python 会引发 UnicodeEncodeError ，但仅在生产服务器上。

解决方案

我将其添加到我的网站和网站管理员 uwsgi 配置文件中，例如 /etc/uwsgi-emperor/vassals/my-site-config-ini 文件

env = LANG=en_US.utf8

更新

转移到 docker 后，我又开始遇到同样的错误。进一步研究后，我意识到在 uwsgi django 实例 import sys; sys.getfilesystemencoding()或{{ file_system_encoding }}以上实例中返回 ANSI_X3.4-1968，但是如果我启动自己的 python 实例并运行，import sys; sys.getfilesystemencoding()我将得到 UTF-8。ANSI_X3.4-1968 格式是引发 UnicodeEncodeError 的原因。

因此，除了上面这个答案中列出的 uwsgi 解决方案之外，我还必须将其添加到我的 Django dockerfile 中

ENV LANG en_US.UTF-8
RUN apt-get update && install -y locales && \
    sed -i -e "s/# $LANG.*/$LANG UTF-8/" /etc/locale.gen && \
    locale-gen --purge &&\
    update-locale LANG=$LANG

或者，如果您不使用 docker，则可以在 shell 中运行它

$ export LANG=en_US.UTF-8

然后运行上面的 RUN 命令（但没有 RUN）。

参考：

http://stackoverflow.com/a/37246853/3003438
https://docs.djangoproject.com/en/dev/howto/deployment/wsgi/uwsgi/