Scrapy句柄302响应代码

Question

我正在使用一个简单的CrawlSpider实现来抓取网站.默认情况Scrapy下,302重定向到目标位置,并忽略最初请求的链接.在特定网站上,我遇到了一个302重定向到另一个页面的页面.我的目标是记录原始链接(响应302)和目标位置(在HTTP响应头中指定)并在parse_item方法中处理它们CrawlSpider.请指导我,我该怎么做到这一点？

我碰到的解决方案提的使用dont_redirect=True或REDIRECT_ENABLE=False,但我真的不希望忽略重定向,其实我也想看看(即不能忽视)重定向页面为好.

例如:我访问http://www.example.com/page1哪个发送302重定向HTTP响应并重定向到http://www.example.com/page2.默认情况下,scrapy忽略page1,跟随page2并处理它.我想同时处理page1和page2在parse_item.

编辑 我已经handle_httpstatus_list = [500, 404]在蜘蛛的类定义中使用来处理500和404响应代码parse_item,但302如果我在其中指定它,则同样不起作用handle_httpstatus_list.

Answer 1

Scrapy 1.0.5(我编写这些行的最新官方)不在handle_httpstatus_list内置的RedirectMiddleware中使用 - 请参阅此问题.从Scrapy 1.1.0(1.1.0rc1可用),问题得到解决.

即使你禁用重定向,你仍然可以模仿它在回调中的行为,检查Location标题并返回Request重定向

蜘蛛示例:

$ cat redirecttest.py
import scrapy


class RedirectTest(scrapy.Spider):

    name = "redirecttest"
    start_urls = [
        'http://httpbin.org/get',
        'https://httpbin.org/redirect-to?url=http%3A%2F%2Fhttpbin.org%2Fip'
    ]
    handle_httpstatus_list = [302]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, dont_filter=True, callback=self.parse_page)

    def parse_page(self, response):
        self.logger.debug("(parse_page) response: status=%d, URL=%s" % (response.status, response.url))
        if response.status in (302,) and 'Location' in response.headers:
            self.logger.debug("(parse_page) Location header: %r" % response.headers['Location'])
            yield scrapy.Request(
                response.urljoin(response.headers['Location']),
                callback=self.parse_page)

控制台日志:

$ scrapy runspider redirecttest.py -s REDIRECT_ENABLED=0
[scrapy] INFO: Scrapy 1.0.5 started (bot: scrapybot)
[scrapy] INFO: Optional features available: ssl, http11
[scrapy] INFO: Overridden settings: {'REDIRECT_ENABLED': '0'}
[scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
[scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
[scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
[scrapy] INFO: Enabled item pipelines: 
[scrapy] INFO: Spider opened
[scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
[scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
[scrapy] DEBUG: Crawled (200) <GET http://httpbin.org/get> (referer: None)
[redirecttest] DEBUG: (parse_page) response: status=200, URL=http://httpbin.org/get
[scrapy] DEBUG: Crawled (302) <GET https://httpbin.org/redirect-to?url=http%3A%2F%2Fhttpbin.org%2Fip> (referer: None)
[redirecttest] DEBUG: (parse_page) response: status=302, URL=https://httpbin.org/redirect-to?url=http%3A%2F%2Fhttpbin.org%2Fip
[redirecttest] DEBUG: (parse_page) Location header: 'http://httpbin.org/ip'
[scrapy] DEBUG: Crawled (200) <GET http://httpbin.org/ip> (referer: https://httpbin.org/redirect-to?url=http%3A%2F%2Fhttpbin.org%2Fip)
[redirecttest] DEBUG: (parse_page) response: status=200, URL=http://httpbin.org/ip
[scrapy] INFO: Closing spider (finished)

请注意,您需要http_handlestatus_list使用302,否则,您将看到此类日志(来自HttpErrorMiddleware):

[scrapy] DEBUG: Crawled (302) <GET https://httpbin.org/redirect-to?url=http%3A%2F%2Fhttpbin.org%2Fip> (referer: None)
[scrapy] DEBUG: Ignoring response <302 https://httpbin.org/redirect-to?url=http%3A%2F%2Fhttpbin.org%2Fip>: HTTP status code is not handled or not allowed