Scrapy crawler无法完成解析函数中的所有循环

Question

我在我的抓取工具中有这个代码

class StackSpider(InitSpider):
    name = 'stack'
    allowed_domains = ['sitepoint.com']
    start_urls = ["http://www.sitepoint.com"]
    start_page = "http://www.sitepoint.com"
    item = StackItem()

    def init_request(self):

        return Request(url=self.start_page, callback=self.parse)

    def parse(self, response):

        hxs = HtmlXPathSelector(response)
        sites = hxs.select('//div[@class="headline_area"]')
        items = []


        ivar = 1
        for site in sites[:5]:
            item = StackItem()
            log.msg(' LOOP' +str(ivar)+ '', level=log.ERROR)
            item['title'] ="yoo ma"
            request =  Request("http://www.sitepoint.com/getting-to-know-css3-selectors-structural-pseudo-classes/",  callback=self.test1)
            request.meta['item'] = item
            ivar = ivar + 1
            yield request


    def test1(self, response):
        log.msg('  LOOP 2 \n', level=log.ERROR)
        item = response.meta['item']
        item['desc'] = "test4"
        return item

我根据文档做了它,但它只适用于一个循环.我的意思是我只能在登录屏幕上看到

LOOP1
LOOP2

它应该重复3次

我尝试了返回和收益的不同组合

1. return request 并return item给出输出 LOOP1 LOOP2
2. yield request并return item给出输出 LOOP1 LOOP1 LOOP1 LOOP2
3. yield request并yield item给出输出 LOOP1 LOOP1 LOOP1 LOOP2
4. return request并yield item给出输出 LOOP1 LOOP2

我怎样才能得到 LOOP 1 LOOP2 LOOP1 LOOP2 AND so on

Answer 1

问题出在你的循环中

for site in sites[:5]:

您在循环中多次请求1个相同的URL.

默认情况下,Scrapy会过滤相同的请求并忽略它们.

如果您想多次请求相同的URL,则需要进行设置 dont_filter=True

            request = Request("http://www.sitepoint.com/getting-to-know-css3-selectors-structural-pseudo-classes/",
            dont_filter=True,
            callback=self.test1)

那应该重复3次