Scrapy Cloud中的“ AttributeError：'SelectorList'对象没有属性'get'”

Question

我正在使用Scrapy设置刮板，该刮板在笔记本电脑上效果很好。但是，当我在刮cloud的云上尝试使用同一蜘蛛时，此消息就会出现：

File "/usr/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/tmp/unpacked-eggs/__main__.egg/ccv_spiders/spiders/gmitem.py", line 31, in parse
    data["marque"] = caritem.css("div.make::text").get().strip().split(" ", 2)[1]
AttributeError: 'SelectorList' object has no attribute 'get'

这是我的代码：

def start_requests(self):
        for item in self.data:
            request = scrapy.Request(item['gm_url'], callback=self.parse)
            request.meta['item'] = item
            yield request

    def parse(self, response):
        item = response.meta['item']
        item['results'] = []

        for caritem in response.css("div.car-item-border"):
            data = AuctionItem()
            urllot = "https://www.website.com/img/auctions/byLot/"
            urlbase = "https://www.website.com/img/auctions/car/thumb/"
            data["marque"] = caritem.css("div.make::text").get().strip().split(" ", 2)[1]
            data["model"] = caritem.css("div.make::text").get().strip().split(" ", 2)[2]
            data["model_year"] = caritem.css("div.make::text").get().strip().split(" ", 1)[0]
            data["price_str"] = caritem.css("div.price::text").get().strip().replace(",", " ")
            if caritem.css("div.price::text").get().find("Estimate"):
                data["sold"] = True
            else:
                data["sold"] = False
            data["auction_house"] = caritem.css("div.auctionHouse::text").get().split("-", 1)[0].strip()
            data["auction_country"] = caritem.css("div.auctionHouse::text").get().rsplit(",", 1)[1].strip()
            data["auction_date"] = caritem.css("div.date::text").get().replace(",", "").strip()
            if caritem.css("div.view-auction a::attr(href)").get().find("/auction-cars/show-backup-image"):
                data["auction_url"] = caritem.css("div.view-auction a::attr(href)").get()
            else:
                data["auction_url"] = None
            data["image_urls"] = caritem.css("img.img-responsive::attr(src)").get()
            if urllot in data["image_urls"]:
                data["image_cloud"] = caritem.css("img.img-responsive::attr(src)").get().replace(urllot,"https://res.cloudinary.com/ccv/image/upload/auctions/")
                data["image_cloud"] = re.sub(r"(?<=[A-Z])/*(?=\d)", "-", data["image_cloud"])
            elif urlbase in data["image_urls"]:
                data["image_cloud"] = caritem.css("img.img-responsive::attr(src)").get().replace(urlbase, "https://res.cloudinary.com/ccv/image/upload/auctions/")
            item['results'].append(data)

        yield item

我的Python版本有什么问题吗？它可以在笔记本电脑上与Anaconda和Python 3一起使用，我不明白为什么它似乎通过“ /usr/local/lib/python2.7 ...”使用python 2.7。

此外，我的JSON输出未显示任何结果数组。

Answer 1

这全都与图书馆的版本有关。

无论get和getall方法最早由引进Parsel（Scrapy的解析库）的版本为1.2.0，如果你使用Scrapy 1.5.2或降低其不批。

您可以使用extract_first和extract作为替代品，或升级到Scrapy 1.6或更高版本。