Python:我如何发送多个HTTP请求并接收响应？

Question

如何以最快的方式发送1000个请求？我知道您可以发送多个请求grequests:

urls = [
    'sample.url/1',
    'sample.url/2',
    ...
]
request = (grequests.get(u) for u in urls)
print grequests.map(request)

但回报不是内容.我需要的是获取json数据,例如:

request = (grequests.get(u) for u in urls)
content = grequests.json(request)

Answer 1

返回的项目不是内容,但它们确实包含内容.您可以像这样获取所有内容:

result = grequests.map(request)
content = '\n'.join(r.content for r in result) # raw content
text = '\n'.join(r.text for r in result)       # decoded content

您可以像这样解析json:

result = grequests.map(request)
json = [r.json() for r in result]

示例程序:

import grequests
import pprint

urls = [
    'http://httpbin.org/user-agent',
    'http://httpbin.org/headers',
    'http://httpbin.org/ip',
]

requests = (grequests.get(u) for u in urls)
responses = grequests.map(requests)

json = [response.json() for response in responses]
pprint.pprint(json)

text = '\n'.join(response.text for response in responses)
print(text)