Scrapy将多余的数据从csv文件传递到解析

Question

我的小蜘蛛会浏览一个csv文件，并使用该csv文件中的地址运行start_urls，如下所示：

 from csv import DictReader
   with open('addresses.csv') as rows:
     start_urls=['http://www.example.com/search/?where='+row["Address"].replace(',','').replace(' ','+') for row in DictReader(rows)]

但是.csv文件还包含电子邮件和其他信息。如何将这些额外信息传递到解析中，以将其添加到新文件中？

import scrapy
from csv import DictReader

with open('addresses.csv') as rows:
  names=[row["Name"].replace(',','') for row in DictReader(rows)]
  emails=[row["Email"].replace(',','') for row in DictReader(rows)]
  start_urls=['http://www.example.com/search/?where='+row["Address"].replace(',','').replace(' ','+') for row in DictReader(rows)]

 def parse(self,response):
   yield{
     'name': FROM CSV,
     'email': FROM CSV,
     'address' FROM SCRAPING: 
     'city' FROM SCRAPING: 
    }

Answer 1

import scrapy
from csv import DictReader

class MySpider(scrapy.Spider):

    def start_requests(self):

        with open('addresses.csv') as rows:

            for row in DictReader(rows):

                name=row["Name"].replace(',','')
                email=row["Email"].replace(',','')

                link = 'http://www.example.com/search/?where='+row["Address"].replace(',','').replace(' ','+')

                yield Request(url = link, 
                        callback = self.parse, 
                        method = "GET", 
                        meta={'name':name, 'email':email}
                    )


    def parse(self,response):
        yield{
         'name': resposne.meta['name'],
         'email': respose.meta['email'],
         'address' FROM SCRAPING: 
         'city' FROM SCRAPING: 
        }

• 打开您的CSV文件。
• 在start_requests方法内部对其进行迭代。
• 将参数传递给回调函数，使用meta变量，您可以在中传递Python字典meta。

注意： 请记住，这start_requests不是我的自定义定义方法，而是Python Scrapy的方法。请参阅https://doc.scrapy.org/en/latest/topics/spiders.html#scrapy.spiders.Spider.start_requests