解析原始HTTP标头

Question

我有一串原始HTTP,我想表示对象中的字段.有没有办法解析HTTP字符串中的各个标头？

'GET /search?sourceid=chrome&ie=UTF-8&q=ergterst HTTP/1.1\r\nHost: www.google.com\r\nConnection: keep-alive\r\nAccept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5\r\nUser-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.45 Safari/534.13\r\nAccept-Encoding: gzip,deflate,sdch\r\nAvail-Dictionary: GeNLY2f-\r\nAccept-Language: en-US,en;q=0.8\r\n
[...]'

Answer 1

标准库中有很好的工具可用于解析RFC 821标头,也可用于解析整个HTTP请求.下面是一个示例请求字符串(请注意,Python将其视为一个大字符串,即使我们在多行中将其分解为可读性),我们可以将其提供给我的示例:

request_text = (
    b'GET /who/ken/trust.html HTTP/1.1\r\n'
    b'Host: cm.bell-labs.com\r\n'
    b'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n'
    b'Accept: text/html;q=0.9,text/plain\r\n'
    b'\r\n'
)

正如@TryPyPy指出的那样,你可以Message用来解析标题 - 虽然我们应该添加一下,BaseHTTPRequestHandler一旦你完成创建它,结果对象就像一个标题字典:

from email.parser import BytesParser
request_line, headers_alone = request_text.split(b'\r\n', 1)
headers = BytesParser().parsebytes(headers_alone)

print(len(headers))     # -> "3"
print(headers.keys())   # -> ['Host', 'Accept-Charset', 'Accept']
print(headers['Host'])  # -> "cm.bell-labs.com"

但是,当然,这会忽略请求行,或者让您自己解析它.事实证明,有一个更好的解决方案.

如果您使用它,标准库将为您解析HTTP BytesIO().虽然它的文档有点模糊 - 标准库中的整套HTTP和URL工具存在问题 - 要使它解析字符串所需要做的就是(a)将字符串包装成a raw_requestline,(b)读取error_code以便它可以被解析,并且(c)捕获在解析期间发生的任何错误代码,而不是让它尝试将它们写回客户端(因为我们没有!).

所以这是我们对标准库类的专业化:

from http.server import BaseHTTPRequestHandler
from io import BytesIO

class HTTPRequest(BaseHTTPRequestHandler):
    def __init__(self, request_text):
        self.rfile = BytesIO(request_text)
        self.raw_requestline = self.rfile.readline()
        self.error_code = self.error_message = None
        self.parse_request()

    def send_error(self, code, message):
        self.error_code = code
        self.error_message = message

再一次,我希望标准库人员已经意识到HTTP解析应该以一种不需要我们编写九行代码来正确调用它的方式进行分解,但你能做什么呢？以下是如何使用这个简单的类:

# Using this new class is really easy!

request = HTTPRequest(request_text)

print(request.error_code)       # None  (check this first)
print(request.command)          # "GET"
print(request.path)             # "/who/ken/trust.html"
print(request.request_version)  # "HTTP/1.1"
print(len(request.headers))     # 3
print(request.headers.keys())   # ['Host', 'Accept-Charset', 'Accept']
print(request.headers['host'])  # "cm.bell-labs.com"

如果解析过程中出错,None则不会Message:

# Parsing can result in an error code and message

request = HTTPRequest(b'GET\r\nHeader: Value\r\n\r\n')

print(request.error_code)     # 400
print(request.error_message)  # "Bad request syntax ('GET')"

我更喜欢使用这样的标准库,因为我怀疑他们已经遇到并解决了任何可能会让我感到困惑的边缘情况,如果我尝试使用正则表达式重新实现Internet规范.

Answer 2

mimetools自Python 2.3以来已被弃用,并完全从Python 3(链接)中删除.

以下是Python 3中应该如何做的:

import email
import io
import pprint

# […]

request_line, headers_alone = request_text.split('\r\n', 1)
message = email.message_from_file(io.StringIO(headers_alone))
headers = dict(message.items())
pprint.pprint(headers, width=160)

Answer 3

如果你剥离GET线条,这似乎工作正常:

import mimetools
from StringIO import StringIO

he = "Host: www.google.com\r\nConnection: keep-alive\r\nAccept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5\r\nUser-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.45 Safari/534.13\r\nAccept-Encoding: gzip,deflate,sdch\r\nAvail-Dictionary: GeNLY2f-\r\nAccept-Language: en-US,en;q=0.8\r\n"

m = mimetools.Message(StringIO(he))

print m.headers

解析示例并将第一行中的信息添加到对象的方法是:

import mimetools
from StringIO import StringIO

he = 'GET /search?sourceid=chrome&ie=UTF-8&q=ergterst HTTP/1.1\r\nHost: www.google.com\r\nConnection: keep-alive\r\n'

# Pop the first line for further processing
request, he = he.split('\r\n', 1)    

# Get the headers
m = mimetools.Message(StringIO(he))

# Add request information
m.dict['method'], m.dict['path'], m.dict['http-version'] = request.split()    

print m['method'], m['path'], m['http-version']
print m['Connection']
print m.headers
print m.dict