我spider_idle在接收信号时的几个蜘蛛之间有共同的行为,我想将此行为移动到扩展中。
我的分机已经成功侦听spider_opened并spider_closed发出信号。但是,spider_idle没有接收到信号。
这是我的扩展程序(为简洁起见进行了编辑):
import logging
import MySQLdb
import MySQLdb.cursors
from scrapy import signals
logger = logging.getLogger(__name__)
class MyExtension(object):
def __init__(self, settings, stats):
self.settings = settings
self.stats = stats
@classmethod
def from_crawler(cls, crawler):
# instantiate the extension object
ext = cls(crawler.settings, crawler.stats)
# connect the extension object to signals
crawler.signals.connect(ext.spider_opened, signal=signals.spider_opened)
crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed)
crawler.signals.connect(ext.spider_idle, signal=signals.spider_idle)
# return the extension object
return ext
def spider_opened(self, spider):
logger.info("start logging spider %s", spider.name)
def spider_closed(self, spider): …Run Code Online (Sandbox Code Playgroud)