Scrapy覆盖json文件而不是附加文件

hoo*_*ooo 5 python scrapy

有没有办法覆盖所述文件而不是附加它?

例)

scrapy crawl myspider -o "/path/to/json/my.json" -t json    
scrapy crawl myspider -o "/path/to/json/my.json" -t json
Run Code Online (Sandbox Code Playgroud)

将附加my.json文件而不是覆盖它.

eLR*_*uLL 13

scrapy crawl myspider -t json --nolog -o - > "/path/to/json/my.json"
Run Code Online (Sandbox Code Playgroud)


Sam*_*Elh 10

-O有一个标志允许覆盖输出文件,您可以通过选项而不是传递文件引用来执行-o此操作,因此您可以使用它:

scrapy crawl myspider -O /path/to/json/my.json
Run Code Online (Sandbox Code Playgroud)

更多信息:

$ scrapy crawl --help
Usage
=====
  scrapy crawl [options] <spider>

Run a spider

Options
=======
--help, -h              show this help message and exit
-a NAME=VALUE           set spider argument (may be repeated)
--output=FILE, -o FILE  append scraped items to the end of FILE (use - for
                        stdout)
--overwrite-output=FILE, -O FILE
                        dump scraped items into FILE, overwriting any existing
                        file
--output-format=FORMAT, -t FORMAT
                        format to use for dumping items

Global Options
--------------
--logfile=FILE          log file. if omitted stderr will be used
--loglevel=LEVEL, -L LEVEL
                        log level (default: DEBUG)
--nolog                 disable logging completely
--profile=FILE          write python cProfile stats to FILE
--pidfile=FILE          write process ID to FILE
--set=NAME=VALUE, -s NAME=VALUE
                        set/override setting (may be repeated)
--pdb                   enable pdb on failure
Run Code Online (Sandbox Code Playgroud)

  • 确实如此,但是“-O”选项是非常新的。不确定,但我认为它是在 2.4.0 版本中引入的,所以可能还不是每个人都可以使用。 (4认同)

小智 7

为了克服这个问题,我scrapy.extensions.feedexport.FileFeedStorage在myproject目录中创建了一个子类.

这是我的customexport.py:

"""Custom Feed Exports extension."""
import os

from scrapy.extensions.feedexport import FileFeedStorage


class CustomFileFeedStorage(FileFeedStorage):
    """
    A File Feed Storage extension that overwrites existing files.

    See: https://github.com/scrapy/scrapy/blob/master/scrapy/extensions/feedexport.py#L79
    """

    def open(self, spider):
        """Return the opened file."""
        dirname = os.path.dirname(self.path)
        if dirname and not os.path.exists(dirname):
            os.makedirs(dirname)
        # changed from 'ab' to 'wb' to truncate file when it exists
        return open(self.path, 'wb')
Run Code Online (Sandbox Code Playgroud)

然后我将以下内容添加到我的settings.py(参见:https://doc.scrapy.org/en/1.2/topics/feed-exports.html#feed-storages-base):

FEED_STORAGES_BASE = {
    '': 'myproject.customexport.CustomFileFeedStorage',
    'file': 'myproject.customexport.CustomFileFeedStorage',
}
Run Code Online (Sandbox Code Playgroud)

现在,每当我写入文件时,它都会被覆盖.