即使页面存在,Python urllib2.urlopen也会返回302错误

Question

我正在使用Python函数urllib2.urlopen来阅读http://www.bad.org.uk/网站但我仍然收到302错误,即使我访问该网站时它加载正常.任何人都知道为什么？

import socket 

headers = { 'User-Agent' : 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)' }

socket.setdefaulttimeout(10)

try:
    req = urllib2.Request('http://www.bad.org.uk/', None, headers)
    urllib2.urlopen(req)
    return True         # URL Exist
except ValueError, ex:
    print 'URL: %s not well formatted' % 'http://www.bad.org.uk/'
    return False        # URL not well formatted
except urllib2.HTTPError, ex:
    print 'The server couldn\'t fulfill the request for %s.' % 'http://www.bad.org.uk/'
    print 'Error code: ', ex.code
    return False
except urllib2.URLError, ex:
    print 'We failed to reach a server for %s.' % 'http://www.bad.org.uk/'
    print 'Reason: ', ex.reason
    return False        # URL don't seem to be alive

错误打印:

The server couldn't fulfill the request for http://www.bad.org.uk//site/1/default.aspx.
Error code:  302

Answer 1

禁用cookie时,http://www.bad.org.uk/上的页面将被破坏.

http://www.bad.org.uk/返回:

HTTP/1.1 302 Found
Location: http://www.bad.org.uk/DesktopDefault.aspx
Set-Cookie: Esperantus_Language_bad=en-GB; path=/
Set-Cookie: Esperantus_Language_rainbow=en-GB; path=/
Set-Cookie: PortalAlias=rainbow; path=/
Set-Cookie: refreshed=true; expires=Thu, 04-Nov-2010 16:21:23 GMT; path=/
Set-Cookie: .ASPXAUTH=; expires=Mon, 11-Oct-1999 23:00:00 GMT; path=/; HttpOnly
Set-Cookie: portalroles=; expires=Mon, 11-Oct-1999 23:00:00 GMT; path=/

如果我在没有设置这些cookie的情况下请求http://www.bad.org.uk/DesktopDefault.aspx ,它会给出另一个302并重定向到自己.

urllib2忽略cookie并发送没有cookie的新请求,因此它会导致该URL的重定向循环.要处理这个问题,您需要添加一个cookie处理程序:

import urllib2
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
response = opener.open('http://www.bad.org.uk')
print response.read()