小编Jas*_*ren的帖子

我的第一个问题:)

我试图抓住我的学校网站,查找所有可能的网页.但我无法将链接转换为文本文件.我有正确的权限,所以这不是问题.

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from scrapy.item import Item
from scrapy.spider import BaseSpider

class hsleidenSpider(CrawlSpider):
        name = "hsleiden1"
        allowed_domains = ["hsleiden.nl"]
        start_urls = ["http://hsleiden.nl"]

        # allow=() is used to match all links
        rules = [
        Rule(SgmlLinkExtractor(allow=()), follow=True),
        Rule(SgmlLinkExtractor(allow=()), callback='parse_item')
        ]

        def parse_item(self, response):
                x = HtmlXPathSelector(response)

                filename = "hsleiden-output.txt"
                open(filename, 'ab').write(response.url)

所以我只在hsleiden.nl页面上扫描.我想将response.url放入文本文件hsleiden-output.txt中.

有没有办法做到这一点？