如何停止scrapy爬虫

Question

如果满足某些条件我想停止蜘蛛我试着这样做: raise CloseSpider('Some Text')和

sys.exit("SHUT DOWN EVERYTHING!")

但它并没有停止.这是代码编写引发异常而不是返回也不会工作,因为蜘蛛继续爬行:

import scrapy
from scrapy.http import Request

from tutorial.items import DmozItem
from scrapy.exceptions import CloseSpider
import sys

class DmozSpider(scrapy.Spider):
    name = "tutorial"
    allowed_domain = ["jabong.com"]
    start_urls = [
            "http://www.jabong.com/women/shoes/sandals/?page=1"
        ]

    page_index = 1

    def parse(self,response):
        products = response.xpath('//li')

        if products:
            for product in products:
                item = DmozItem()
                item_url = product.xpath('@data-url').extract()
                item_url = "http://www.jabong.com/" + item_url[0] if item_url else ''   
                if item_url:
                        request=Request(url=item_url,callback=self.parse_page2,meta={"item":item},
                                headers={"Accept":
                        "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"})
                    request.meta['item'] = item
                    yield request
        else:
            return

        self.page_index += 1
            if self.page_index:
                    yield Request(url="http://www.jabong.com/women/shoes/sandals/?page=%s" %                        (self.page_index),
            headers={"Referer": "http://www.jabong.com/women/shoes/sandals/",
                 "X-Requested-With": "XMLHttpRequest"},
                        callback=self.parse)

    def parse_page2(self, response):
        sizes=[]
        item = response.meta['item']
        item['site_name'] = 'jabong'
        item['tags'] = ''
        yield item

更新:即使我举起shutpider而不是返回,我的蜘蛛也没有停止

Answer 1

这return也将在这里工作,而不是强制,但由于爬行逻辑 - 因为你不再产生任何请求.
但请记住,您可能会将其解释为"蜘蛛不关闭"实际上是管道中已经开始处理的剩余请求,并且需要更多时间来完成处理.因此,蜘蛛不会停止return执行的同一瞬间,因为管道中仍有请求.当它们全部被处理时,如果没有创建新的蜘蛛,蜘蛛最终会停止.