urlopen返回有效链接的重定向错误

Question

我正在python中构建一个断开的链接检查器,它正在成为一个苦差事,构建逻辑,用于正确识别使用浏览器访问时无法解析的链接.我找到了一组链接,我可以用我的刮刀一致地重现重定向错误,但在浏览器中访问时它会完美地解析.我希望我能在这里找到一些见解.

import urllib
import urllib.request
import html.parser
import requests
from requests.exceptions import HTTPError
from socket import error as SocketError

try:
    req=urllib.request.Request(url, None, {'User-Agent': 'Mozilla/5.0 (X11; Linux i686; G518Rco3Yp0uLV40Lcc9hAzC1BOROTJADjicLjOmlr4=) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36','Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8','Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3','Accept-Encoding': 'gzip, deflate, sdch','Accept-Language': 'en-US,en;q=0.8','Connection': 'keep-alive'})
    response = urllib.request.urlopen(req)
    raw_response = response.read().decode('utf8', errors='ignore')
    response.close()
except urllib.request.HTTPError as inst:
    output = format(inst)


print(output)

在这种情况下,可靠地返回此错误的URL示例是" http://forums.hostgator.com/want-see-your-sites-dns-propagating-t48838.html ".它在访问时完美解析,但上面的代码将返回以下错误:

HTTP Error 301: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was:
Moved Permanently

任何想法如何我可以正确地识别这些链接功能,而不是盲目地忽略该网站的链接(可能会错过真正破坏的链接)？

Answer 1

您会收到无限循环错误,因为您要抓取的页面使用cookie并在客户端未发送cookie时重定向.当你不允许使用cookie时,你会得到与大多数其他刮刀工具以及浏览器相同的错误.

你需要a http.cookiejar.CookieJar和a urllib.request.HTTPCookieProcessor来避免重定向循环:

import urllib
import urllib.request
import html.parser
import requests
from requests.exceptions import HTTPError
from socket import error as SocketError
from http.cookiejar import CookieJar

try:
    req=urllib.request.Request(url, None, {'User-Agent': 'Mozilla/5.0 (X11; Linux i686; G518Rco3Yp0uLV40Lcc9hAzC1BOROTJADjicLjOmlr4=) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36','Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8','Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3','Accept-Encoding': 'gzip, deflate, sdch','Accept-Language': 'en-US,en;q=0.8','Connection': 'keep-alive'})
    cj = CookieJar()
    opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
    response = opener.open(req)
    raw_response = response.read().decode('utf8', errors='ignore')
    response.close()
except urllib.request.HTTPError as inst:
    output = format(inst)
    print(output)