小编Kun*_*war的帖子

我一直在尝试抓取一个网站，该网站似乎已识别并阻止了我的 IP 并抛出429 Too many requests响应。

我从这个链接安装了scrapy-proxies：https : //github.com/aivarsk/scrapy-proxies并按照给定的说明进行操作。我从这里得到了一个代理列表：http : //www.gatherproxy.com/现在这里是我的 settings.py 和 proxylist.txt 的样子：

设置.py

BOT_NAME = 'project'
SPIDER_MODULES = ['project.spiders']
NEWSPIDER_MODULE = 'project.spiders'
# Retry many times since proxies often fail
RETRY_TIMES = 10
# Retry on most error codes since proxies fail for different reasons
RETRY_HTTP_CODES = [429, 500, 503, 504, 400, 403, 404, 408]

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': 90,
    'scrapy_proxies.RandomProxy': 100,
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}

PROXY_LIST = "filepath\proxylist.txt"
USER_AGENT = 'Mozilla/5.0 …