在lxml中过滤无效unicode字符的中心方法？

Question

众所周知,XML文档中不允许使用某些字符范围.我知道过滤掉这些字符的解决方案(如[1],[2]).

按照不要重复自己的原则,我宁愿在一个中心点实现其中一个解决方案 - 现在,我必须在输入之前清理任何可能不安全的文本lxml.有没有办法实现这一点,例如通过继承lxml过滤器类,捕获一些异常或设置配置开关？

---

编辑:希望有点澄清这个问题,这里是一个示例代码:

from lxml import etree

root = etree.Element("root")
root.text = u'\uffff'
root.text += u'\ud800' 

print(etree.tostring(root))

root.text += '\x02'.decode("utf-8")

执行此操作会得到结果

<root>&#65535;&#55296;</root>

Traceback (most recent call last):
  File "[…]", line 9, in <module>
    root.text += u'\u0002'
  File "lxml.etree.pyx", line 953, in lxml.etree._Element.text.__set__ (src/lxml/lxml.etree.c:44956)
  File "apihelpers.pxi", line 677, in lxml.etree._setNodeText (src/lxml/lxml.etree.c:20273)
  File "apihelpers.pxi", line 1395, in lxml.etree._utf8 (src/lxml/lxml.etree.c:26485)
ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters

如您所见,2字节抛出异常,但lxml很高兴地逃避了另外两个超出范围的字符.真正的麻烦在于

s = "<root>&#65535;&#55296;</root>"
root = etree.fromstring(s)

也抛出一个例外.在我看来,这种行为有点令人不安,特别是因为它产生了无效的XML文档.

---

事实证明,这可能是一个2比3的问题.使用python3.4,上面的代码抛出异常

Traceback (most recent call last):
  File "[…]", line 5, in <module>
    root.text += u'\ud800'
  File "lxml.etree.pyx", line 953, in lxml.etree._Element.text.__set__ (src/lxml/lxml.etree.c:44971)
  File "apihelpers.pxi", line 677, in lxml.etree._setNodeText (src/lxml/lxml.etree.c:20273)
  File "apihelpers.pxi", line 1387, in lxml.etree._utf8 (src/lxml/lxml.etree.c:26380)
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 1: surrogates not allowed

唯一剩下的问题是\uffff角色,它lxml仍然乐于接受.

Answer 1

只需在用 LXML 解析字符串之前过滤该字符串即可：从 XML 中清除无效字符（要点为 lawlesst）。

我用你的代码试过了；它似乎有效，但您需要更改要点以导入re和sys！

from lxml import etree
from cleaner import invalid_xml_remove

root = etree.Element("root")
root.text = u'\uffff'
root.text += u'\ud800' 

print(etree.tostring(root))

root.text += invalid_xml_remove('\x02'.decode("utf-8"))