有没有办法覆盖所述文件而不是附加它?
例)
scrapy crawl myspider -o "/path/to/json/my.json" -t json
scrapy crawl myspider -o "/path/to/json/my.json" -t json
Run Code Online (Sandbox Code Playgroud)
将附加my.json文件而不是覆盖它.
eLR*_*uLL 13
scrapy crawl myspider -t json --nolog -o - > "/path/to/json/my.json"
Run Code Online (Sandbox Code Playgroud)
Sam*_*Elh 10
-O有一个标志允许覆盖输出文件,您可以通过选项而不是传递文件引用来执行-o此操作,因此您可以使用它:
scrapy crawl myspider -O /path/to/json/my.json
Run Code Online (Sandbox Code Playgroud)
更多信息:
$ scrapy crawl --help
Usage
=====
scrapy crawl [options] <spider>
Run a spider
Options
=======
--help, -h show this help message and exit
-a NAME=VALUE set spider argument (may be repeated)
--output=FILE, -o FILE append scraped items to the end of FILE (use - for
stdout)
--overwrite-output=FILE, -O FILE
dump scraped items into FILE, overwriting any existing
file
--output-format=FORMAT, -t FORMAT
format to use for dumping items
Global Options
--------------
--logfile=FILE log file. if omitted stderr will be used
--loglevel=LEVEL, -L LEVEL
log level (default: DEBUG)
--nolog disable logging completely
--profile=FILE write python cProfile stats to FILE
--pidfile=FILE write process ID to FILE
--set=NAME=VALUE, -s NAME=VALUE
set/override setting (may be repeated)
--pdb enable pdb on failure
Run Code Online (Sandbox Code Playgroud)
小智 7
为了克服这个问题,我scrapy.extensions.feedexport.FileFeedStorage在myproject目录中创建了一个子类.
这是我的customexport.py:
"""Custom Feed Exports extension."""
import os
from scrapy.extensions.feedexport import FileFeedStorage
class CustomFileFeedStorage(FileFeedStorage):
"""
A File Feed Storage extension that overwrites existing files.
See: https://github.com/scrapy/scrapy/blob/master/scrapy/extensions/feedexport.py#L79
"""
def open(self, spider):
"""Return the opened file."""
dirname = os.path.dirname(self.path)
if dirname and not os.path.exists(dirname):
os.makedirs(dirname)
# changed from 'ab' to 'wb' to truncate file when it exists
return open(self.path, 'wb')
Run Code Online (Sandbox Code Playgroud)
然后我将以下内容添加到我的settings.py(参见:https://doc.scrapy.org/en/1.2/topics/feed-exports.html#feed-storages-base):
FEED_STORAGES_BASE = {
'': 'myproject.customexport.CustomFileFeedStorage',
'file': 'myproject.customexport.CustomFileFeedStorage',
}
Run Code Online (Sandbox Code Playgroud)
现在,每当我写入文件时,它都会被覆盖.