如何获取XML文档并使用Python扭曲解析它？

Question

我想要一种快速的方法来获取URL并在流式传输时解析它.理想情况下,这应该是超快的.我选择的语言是Python.我有一种直觉,扭曲可以做到这一点,但我找不到一个例子.

Answer 1

如果您需要以流方式处理HTTP响应,则有几个选项.

你可以通过以下方式完成downloadPage:

from xml.sax import make_parser
from twisted.web.client import downloadPage

class StreamingXMLParser:
    def __init__(self):
        self._parser = make_parser()

    def write(self, bytes):
        self._parser.feed(bytes)

    def close(self):
        self._parser.feed('', True)

parser = StreamingXMLParser()
d = downloadPage(url, parser)
# d fires when the response is completely received

这是有效的,因为downloadPage将响应主体写入传递给它的类文件对象.在这里,使用write和close方法传递对象满足该要求,但是逐步将数据解析为XML而不是将其放在磁盘上.

另一种方法是在HTTPPageGetter关卡中挂钩. HTTPPageGetter是内部使用的协议getPage.

class StreamingXMLParsingHTTPClient(HTTPPageGetter):
    def connectionMade(self):
        HTTPPageGetter.connectionMade(self)
        self._parser = make_parser()

    def handleResponsePart(self, bytes):
        self._parser.feed(bytes)

    def handleResponseEnd(self):
        self._parser.feed('', True)
        self.handleResponse(None) # Whatever you pass to handleResponse will be the result of the Deferred below.

factory = HTTPClientFactory(url)
factory.protocol = StreamingXMLParsingHTTPClient
reactor.connectTCP(host, port, factory)
d = factory.deferred
# d fires when the response is completely received

最后,很快就会有一个新的HTTP客户端API.由于这不是任何版本的一部分,它不像前两种方法那样直接有用,但它更好一些,所以我会把它包含在内,让你了解未来会带来什么.:)新API允许您指定接收响应正文的协议.所以你会做这样的事情:

class StreamingXMLParser(Protocol):
    def __init__(self):
        self.done = Deferred()

    def connectionMade(self):
        self._parser = make_parser()

    def dataReceived(self, bytes):
        self._parser.feed(bytes)

    def connectionLost(self, reason):
        self._parser.feed('', True)
        self.done.callback(None)

from twisted.web.client import Agent
from twisted.internet import reactor

agent = Agent(reactor)
d = agent.request('GET', url, headers, None)
def cbRequest(response):
    # You can look at the response headers here if you like.
    protocol = StreamingXMLParser()
    response.deliverBody(protocol)
    return protocol.done
d.addCallback(cbRequest) # d fires when the response is fully received and parsed