Python机械化连接失败的问题

Question

我正在运行一个爬虫,它在MAC和Ubuntu系统中运行良好.当我在Windows中运行它时,它会跟踪堆栈跟踪:

  File "build\bdist.win32\egg\mechanize\_mechanize.py", line 203, in open
    return self._mech_open(url, data, timeout=timeout)
  File "build\bdist.win32\egg\mechanize\_mechanize.py", line 230, in _mech_open
    response = UserAgentBase.open(self, request, data)
  File "build\bdist.win32\egg\mechanize\_opener.py", line 193, in open
    response = urlopen(self, req, data)
  File "build\bdist.win32\egg\mechanize\_urllib2_fork.py", line 344, in _open
    '_open', req)
  File "build\bdist.win32\egg\mechanize\_urllib2_fork.py", line 332, in _call_chain
    result = func(*args)
  File "build\bdist.win32\egg\mechanize\_urllib2_fork.py", line 1170, in https_open
    return self.do_open(conn_factory, req)
  File "build\bdist.win32\egg\mechanize\_urllib2_fork.py", line 1118, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>

Answer 1

我非常怀疑问题是否与您的系统有关.要解决这个问题,我建议:

1. 检查您的机械化版本.旧版本可能以不同方式处理请求.
2. 调查set_handle_robots和addheaders.问题可能与每个IP的请求过多有关.(我怀疑这是真正的问题.)
3. 检查错误处理.如上面的评论所述,您应该添加适当的错误处理.

如果上述解决方案均无效,请提供导致错误的代码.