如何分析一个沙哑的python脚本？

Question

以示例脚本

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class MySpider(CrawlSpider):
    name = 'example.com'
    allowed_domains = ['example.com']
    start_urls = ['http://www.example.com']

    rules = (
        # Extract links matching 'category.php' (but not matching 'subsection.php')
        # and follow links from them (since no callback means follow=True by default).
        Rule(LinkExtractor(allow=('category\.php', ), deny=('subsection\.php', ))),

        # Extract links matching 'item.php' and parse them with the spider's method parse_item
        Rule(LinkExtractor(allow=('item\.php', )), callback='parse_item'),
    )

    def parse_item(self, response):
        self.logger.info('Hi, this is an item page! %s', response.url)
        item = scrapy.Item()
        item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)')
        item['name'] = response.xpath('//td[@id="item_name"]/text()').extract()
        item['description'] = response.xpath('//td[@id="item_description"]/text()').extract()
        return item

谁能让我知道实际上如何对此脚本进行性能分析？

谢谢

Answer 1

这取决于您想要分析的内容：

• 如果是一般调试，请参见调试Spiders

• 如果是内存使用，请参阅调试内存泄漏

• 如果是爬取率等统计信息，请参见统计信息收集

• 如果您想在运行时与蜘蛛进行交互，Telnet 控制台可能是最佳选择。

当然，还有日志记录。