如何将数据从scrapy插入到mysql中

Question

我试图从亚马逊使用scrapy获取数据我可以用CSV获取数据但我无法在mysql数据库中插入数据请找到我的代码 我的蜘蛛是

import scrapy
from craigslist_sample.items import AmazonDepartmentItem
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor

class AmazonAllDepartmentSpider(scrapy.Spider):

    name = "amazon"
    allowed_domains = ["amazon.com"]
    start_urls = [
        "http://www.amazon.com/gp/site-directory/ref=nav_sad/187-3757581-3331414"
    ]
    def parse(self, response):
        for sel in response.xpath('//ul/li'):
            item = AmazonDepartmentItem()
            item['title'] = sel.xpath('a/text()').extract()
            item['link'] = sel.xpath('a/@href').extract()
            item['desc'] = sel.xpath('text()').extract()
        return item

我的管道代码是

import sys
import MySQLdb
import hashlib
from scrapy.exceptions import DropItem
from scrapy.http import Request

class MySQLStorePipeline(object):


    host = 'rerhr.com'
    user = 'amazon'
    password = 'sads23'
    db = 'amazon_project'

    def __init__(self):
        self.connection = MySQLdb.connect(self.host, self.user, self.password, self.db)
        self.cursor = self.connection.cursor()

    def process_item(self, item, spider):   
        try:
            self.cursor.execute("""INSERT INTO amazon_project.ProductDepartment (ProductDepartmentLilnk) 
                            VALUES (%s)""",
                           (
                            item['link'].encode('utf-8')))

            self.connection.commit()

        except MySQLdb.Error, e:
            print "Error %d: %s" % (e.args[0], e.args[1])
        return item

当我运行以下命令

scrapy crawl amazon -o items.csv -t csv

然后我可以在我的CSV中获取数据,但是当我运行时

scrapy爬行亚马逊

上面的代码我无法在mysql中插入数据 所以,请帮助我,我们必须做什么,然后我可以在mysql中插入数据

谢谢

Answer 1

问题实际上是在parse()回调中.extract()调用返回一个列表,因此,所有项字段值都成为列表.然后,item['link'].encode('utf-8')管道中的调用失败,因为encode()列表上没有方法.

一个快速而简单的解决方法是获取extract()调用结果的第一个元素:

def parse(self, response):
    for sel in response.xpath('//ul/li'):
        item = AmazonDepartmentItem()
        item['title'] = sel.xpath('a/text()').extract()[0]
        item['link'] = sel.xpath('a/@href').extract()[0]
        item['desc'] = sel.xpath('text()').extract()[0]
        yield item

请注意,我还return item用a 替换了表达式yield item并将其放在循环中.

更好的方法是ItemLoader使用输入和输出处理器定义:

from scrapy.contrib.loader import ItemLoader
from scrapy.contrib.loader.processor import TakeFirst

class ProductLoader(ItemLoader):
    default_output_processor = TakeFirst()

仅供参考,这是做什么的TakeFirst():

从接收的值返回第一个非null /非空值,因此它通常用作单值字段的输出处理器.它不接收任何构造函数参数,也不接受Loader上下文.

然后,该parse()方法将转换为:

def parse(self, response):
    for sel in response.xpath('//ul/li'):
        l = ItemLoader(item=AmazonDepartmentItem(), selector=sel)
        l.add_xpath('title', 'a/text()')
        l.add_xpath('link', 'a/@href')
        l.add_xpath('desc', 'text()')
        yield l.load_item()