使用HTTPAdapter的Python请求暂停了几个小时

Question

我有一个特殊的URL,我的代码暂停了几个小时(超过3个小时).我似乎无法理解为什么会这样做.

该URL是http://www.etudes.ccip.fr/maintenance_site.php.

直接的requests.get()即时工作,但每当我有一个HTTPAdapter时,代码似乎几乎无限期地睡眠

import requests
from requests.adapters import HTTPAdapter    

url = 'http://www.etudes.ccip.fr/maintenance_site.php'
session = requests.Session()
session.mount('http://', HTTPAdapter(max_retries=2))
session.get(url, timeout=2)

Answer 1

初始化的适配器使用下面的设置重试

    if max_retries == DEFAULT_RETRIES:
        self.max_retries = Retry(0, read=False)
    else:
        self.max_retries = Retry.from_int(max_retries)

如果你看一下初始化

def __init__(self, total=10, connect=None, read=None, redirect=None, status=None,
             method_whitelist=DEFAULT_METHOD_WHITELIST, status_forcelist=None,
             backoff_factor=0, raise_on_redirect=True, raise_on_status=True,
             history=None, respect_retry_after_header=True):

默认值respect_retry_after_header是True.False在你的情况下你需要这个.如果使用curl检查响应

$ curl -I http://www.etudes.ccip.fr/maintenance_site.php
HTTP/1.1 503 Service Temporarily Unavailable
Date: Thu, 23 Nov 2017 14:15:49 GMT
Server: Apache
Status: 503 Service Temporarily Unavailable
Retry-After: 3600
Expires: Sat, 26 Jul 1997 05:00:00 GMT
Cache-Control: pre-check=0, post-check=0, max-age=0
Pragma: no-cache
Connection: close
Content-Type: text/html; charset=ISO-8859-1

您希望将respect_retry_after_header其设置为False.这可以通过创建适配器然后修改此行为来完成

import requests
from requests.adapters import HTTPAdapter

url = 'http://www.etudes.ccip.fr/maintenance_site.php'
session = requests.Session()

adapter = HTTPAdapter(max_retries=2)
adapter.max_retries.respect_retry_after_header = False

session.mount('http://', adapter)

session.get(url, timeout=2)

Answer 2

响应中的retry-after标头是将连接设置为休眠3600秒的问题.请参阅urllib3中的retry.py.

    def sleep(self, response=None):
    """ Sleep between retry attempts.

    This method will respect a server's ``Retry-After`` response header
    and sleep the duration of the time requested. If that is not present, it
    will use an exponential backoff. By default, the backoff factor is 0 and
    this method will return immediately.
    """

    if response:
        slept = self.sleep_for_retry(response)
        if slept:
            return

    self._sleep_backoff()

解决方案是设置max_retries = 0.这样可以避免等待2x3600秒直到您的应用程序结束.