使用UrlFetch时无法处理DeadlineExceededError

Question

我有这个基本的实用程序类并行获取(可能)缩短的URL并返回一个具有最终URL的字典.它使用此博客文章中描述的wait_any功能.

class UrlFetcher(object):

  @classmethod
  def fetch_urls(cls,url_list):
    rpcs = []
    for url in url_list:
      rpc = urlfetch.create_rpc(deadline=5.0)
      urlfetch.make_fetch_call(rpc, url,method = urlfetch.HEAD)
      rpcs.append(rpc)

    result = {}
    while len(rpcs) > 0:
      rpc = apiproxy_stub_map.UserRPC.wait_any(rpcs)
      rpcs.remove(rpc)
      request_url = rpc.request.url()
      try:
        final_url = rpc.get_result().final_url
      except AttributeError:
        final_url = request_url
      except DeadlineExceededError:
        logging.error('Handling DeadlineExceededError for url: %s' %request_url)
        final_url  = None
      except (DownloadError,InvalidURLError):
        final_url  = None        
      except UnicodeDecodeError: #Funky url with very evil characters
        final_url = unicode(rpc.get_result().final_url,'utf-8')

      result[request_url] = final_url

    logging.info('Returning results: %s' %result)
    return result

即使我尝试处理DeadlineExceededError,应用程序日志也会显示.

2011-04-20 17:06:17.755
UrlFetchWorker started
E 2011-04-20 17:06:22.769
The API call urlfetch.Fetch() took too long to respond and was cancelled.
Traceback (most recent call last):
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 636, in __call__
    handler.post(*groups)
  File "/base/data/home/apps/tweethitapp/1.349863151373877476/tweethit/handlers/taskworker.py", line 80, in post
    result_dict = UrlFetcher.fetch_urls(fetch_targets)
  File "/base/data/home/apps/tweethitapp/1.349863151373877476/tweethit/utils/rpc.py", line 98, in fetch_urls
    final_url = rpc.get_result().final_url
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 592, in get_result
    return self.__get_result_hook(self)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/urlfetch.py", line 345, in _get_fetch_result
    rpc.check_success()
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 558, in check_success
    self.__rpc.CheckSuccess()
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 133, in CheckSuccess
    raise self.exception
DeadlineExceededError: The API call urlfetch.Fetch() took too long to respond and was cancelled.
W 2011-04-20 17:06:22.858
Found 1 RPC request(s) without matching response (presumably due to timeouts or other errors)

我在这里错过了什么？有没有其他方法来处理DeadlineExceededError？

或者是否有不同类型的DeadlineExceededErrors并且我导入错误的一个？我正在使用:from google.appengine.runtime import DeadlineExceededError

Answer 1

根据内联文档google.appengine.runtime.DeadlineExceededError:

请求达到其总时间限制时引发异常.

不要与runtime.apiproxy_errors.DeadlineExceededError混淆.当单个API调用花费太长时间时会引发该问题.

这是一个很好的演示,为什么你应该使用合格的导入(from google.appengine import runtime,然后参考runtime.DeadlineExceededError)!

Answer 2

正如你猜到的和其他人所说,你想要一个不同的DeadlineExceededError:

来自于2012年6月的https://developers.google.com/appengine/articles/deadlineexceedederrors:

目前,Python运行时有几个名为DeadlineExceededError的错误:>

google.appengine.runtime.DeadlineExceededError:如果整体请求超时(通常在60秒后,或10分钟后,任务队列请求),则会引发;

google.appengine.runtime.apiproxy_errors.DeadlineExceededError:如果RPC超过其截止日期,则引发此问题.这通常是5秒,但可以使用'deadline'选项设置某些API;

google.appengine.api.urlfetch_errors.DeadlineExceededError:如果URLFetch超时则引发.

捕捉google.appengine.api.urlfetch_errors.DeadlineExceededError似乎对我有用.另外值得注意的是(至少在dev app server 1.7.1中),它urlfetch_errors.DeadlineExceededError是一个子类DownloadError,这是有意义的:

class DeadlineExceededError(DownloadError):
  """Raised when we could not fetch the URL because the deadline was exceeded.

  This can occur with either the client-supplied 'deadline' or the system
  default, if the client does not supply a 'deadline' parameter.
  """