相关疑难解决方法(0)

我想获得与此命令行相同的结果:scrapy crawl linkedin_anonymous -a first = James -a last = Bond -o output.json

我的脚本如下:

import scrapy
from linkedin_anonymous_spider import LinkedInAnonymousSpider
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

spider = LinkedInAnonymousSpider(None, "James", "Bond")
process = CrawlerProcess(get_project_settings())
process.crawl(spider) ## <-------------- (1)
process.start()

我发现(1)中的process.crawl()创建了另一个LinkedInAnonymousSpider,其中第一个和最后一个是None(打印在(2)中),如果是这样,那么就没有创建对象蜘蛛的意义了,怎么可能首先传递参数,最后传递给process.crawl()？

linkedin_anonymous:

from logging import INFO

import scrapy

class LinkedInAnonymousSpider(scrapy.Spider):
    name = "linkedin_anonymous"
    allowed_domains = ["linkedin.com"]
    start_urls = []

    base_url = "https://www.linkedin.com/pub/dir/?first=%s&last=%s&search=Search"

    def __init__(self, input = None, first= None, last=None):
        self.input = input  # source file name
        self.first = …