使用requests/urllib3在每次重试尝试时添加回调函数

Question

我已经在这里和这里建议requests使用重试机制进行会话. urllib3.util.retry

现在,我试图找出添加回调函数的最佳方法,该函数将在每次重试尝试时调用.

为了更好地解释自己,如果Retry对象或请求get方法有办法添加回调函数,那就太好了.也许是这样的:

import requests
from requests.packages.urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

def retry_callback(url):
    print url   

s = requests.Session()
retries = Retry(total=5, status_forcelist=[ 500, 502, 503, 504 ])
s.mount('http://', HTTPAdapter(max_retries=retries))

url = 'http://httpstat.us/500'
s.get(url, callback=retry_callback, callback_params=[url])

我知道,对于打印网址,我可以使用日志记录,但这只是一个简单的例子,用于更复杂的使用.
如果不是最好的python编码,请原谅我,但我希望它足够清楚.

谢谢.

Answer 1

您可以继承Retry该类，以添加该功能。

Retry对于给定的连接尝试，这是与实例的完整交互流：

• Retry.increment()每当引发异常或返回30x重定向响应或Retry.is_retry()方法返回true时，使用当前方法，URL，响应对象（如果有）和异常（如果引发了）调用当前方法。
  • .increment() 将重新引发该错误（如果有），并且该对象已配置为不重试该特定类别的错误。
  • .increment()调用Retry.new()以创建更新后的实例，并更新所有相关计数器，并history用新RequestHistory()实例（名为元组）修改属性。
  • .increment()MaxRetryError如果Retry.is_exhausted()调用的返回值为Retry.new()true，将引发异常。is_exhausted()当其跟踪的任何计数器降到0以下（None忽略设置为的计数器）时，返回true 。
  • .increment()返回新Retry实例。
• 返回值Retry.increment()替换Retry跟踪的旧实例。如果存在重定向，则将Retry.sleep_for_retry()调用（如果有Retry-After头，则处于睡眠状态），否则Retry.sleep()被调用（调用self.sleep_for_retry()以遵守Retry-After头，否则，如果存在退避策略，则仅处于睡眠状态）。然后，使用新Retry实例进行递归连接调用。

这给了您3个不错的回调点；在的开始处.increment()，创建新Retry实例时以及在上下文管理器中super().increment()，让回调否决异常或在退出时更新返回的重试策略。

这是勾住开始的.increment()样子：

import logging

logger = getLogger(__name__)

class CallbackRetry(Retry):
    def __init__(self, *args, **kwargs):
        self._callback = kwargs.pop('callback', None)
        super(CallbackRetry, self).__init__(*args, **kwargs)
    def new(self, **kw):
        # pass along the subclass additional information when creating
        # a new instance.
        kw['callback'] = self._callback
        return super(CallbackRetry, self).new(**kw)
    def increment(self, method, url, *args, **kwargs):
        if self._callback:
            try:
                self._callback(url)
            except Exception:
                logger.exception('Callback raised an exception, ignoring')
        return super(CallbackRetry, self).increment(method, url, *args, **kwargs)

请注意，该url说法是真的只是URL路径，省略了请求的网址部分（你必须提取从_pool参数，它有.scheme，.host和.port属性）。

演示：

>>> def retry_callback(url):
...     print('Callback invoked with', url)
...
>>> s = requests.Session()
>>> retries = CallbackRetry(total=5, status_forcelist=[500, 502, 503, 504], callback=retry_callback)
>>> s.mount('http://', HTTPAdapter(max_retries=retries))
>>> s.get('http://httpstat.us/500')
Callback invoked with /500
Callback invoked with /500
Callback invoked with /500
Callback invoked with /500
Callback invoked with /500
Callback invoked with /500
Traceback (most recent call last):
  File "/.../lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 732, in urlopen
    body_pos=body_pos, **response_kw)
  File "/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 732, in urlopen
    body_pos=body_pos, **response_kw)
  File "/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 732, in urlopen
    body_pos=body_pos, **response_kw)
  [Previous line repeated 1 more times]
  File "/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 712, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
  File "<stdin>", line 8, in increment
  File "/.../lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='httpstat.us', port=80): Max retries exceeded with url: /500 (Caused by ResponseError('too many 500 error responses',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../lib/python3.6/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/.../lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/.../lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/.../lib/python3.6/site-packages/requests/adapters.py", line 499, in send
    raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPConnectionPool(host='httpstat.us', port=80): Max retries exceeded with url: /500 (Caused by ResponseError('too many 500 error responses',))

在.new()方法中添加一个钩子将使您可以为下一次尝试调整策略，还可以对.history属性进行自省，但不能避免重新引发异常。