Scrapy - 抓取整个网站

Question

我无法抓取整个网站,Scrapy只是在表面爬行,我想爬得更深.谷歌搜索最后5-6小时,没有任何帮助.我的代码如下:

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from scrapy.item import Item
from scrapy.spider import BaseSpider
from scrapy import log

class ExampleSpider(CrawlSpider):
    name = "example.com"
    allowed_domains = ["example.com"]
    start_urls = ["http://www.example.com/"]
    rules = [Rule(SgmlLinkExtractor(allow=()), 
                  follow=True),
             Rule(SgmlLinkExtractor(allow=()), callback='parse_item')
    ]
    def parse_item(self,response):
        self.log('A response from %s just arrived!' % response.url)

请帮忙 !!!!

谢谢,阿比拉姆

Answer 1

规则短路,意味着链接满足的第一个规则将是应用的规则,您的第二个规则(带回调)将不会被调用.

将您的规则更改为:

rules = [Rule(SgmlLinkExtractor(), callback='parse_item', follow=True)]