from scrapy.spider import Spider
from scrapy.selector import Selector
from dirbot.items import Website
class DmozSpider(Spider):
name = "dmoz"
allowed_domains = ["dmoz.org"]
start_urls = [
"http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",
]
def parse(self, response):
"""
The lines below is a spider contract. For more info see:
http://doc.scrapy.org/en/latest/topics/contracts.html
@url http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/
@scrapes name
"""
sel = Selector(response)
sites = sel.xpath('//ul[@class="directory-url"]/li')
items = []
for site in sites:
item = Website()
item['name'] = site.xpath('a/text()').extract()
item['url'] = site.xpath('a/@href').extract()
item['description'] = site.xpath('text()').re('-\s[^\n]*\\r')
items.append(item)
return …我试图以非常基本的方式抓取网站.但Scrapy没有抓取所有链接.我会解释如下情况─
main_page.html - >包含指向a_page.html,b_page.html,c_page.html的
链接a_page.html - >包含指向a1_page.html的链接,a2_page.html
b_page.html - >包含指向b1_page.html,b2_page.html
c_page的链接.html - >包含指向c1_page.html的链接,c2_page.html
a1_page.html - >包含指向b_page.html的链接
a2_page.html - >包含指向c_page.html的链接
b1_page.html - >包含指向a_page.html的链接
b2_page.html - >包含指向c_page.html的链接
c1_page.html - >包含指向a_page.html的链接
c2_page.html - >包含指向main_page.html的链接
我在CrawlSpider中使用以下规则 -
Rule(SgmlLinkExtractor(allow = ()), callback = 'parse_item', follow = True))
但抓取结果如下 -
DEBUG:Crawled(200)http://localhost/main_page.html>(referer:None)2011-12-05 09:56:07 + 0530 [test_spider] DEBUG:Crawled(200)http:// localhost/a_page. html>(referer: http://localhost/main_page.html )2011-12-05 09:56:07 + 0530 [test_spider] DEBUG:Crawled(200)http://localhost/a1_page.html>(referer:http ://localhost/a_page.html )2011-12-05 09:56:07 + 0530 [test_spider] DEBUG:Crawled(200)http://localhost/b_page.html>(referer:http:// localhost/a1_page .html)2011-12-05 09:56:07 + 0530 [test_spider] DEBUG:Crawled(200)http://localhost/b1_page.html>(referer:http://localhost/b_page.html)2011-12 -05 09:56:07 …