标签: scrapy-middleware

我使用 Scrapy FakeUserAgent 并在我的 Linux 服务器上不断收到此错误。

Traceback (most recent call last):
  File "/usr/local/lib64/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 37, in process_request
    response = yield method(request=request, spider=spider)
  File "/usr/local/lib/python2.7/site-packages/scrapy_fake_useragent/middleware.py", line 27, in process_request
    request.headers.setdefault('User-Agent', self.ua.random)
  File "/usr/local/lib/python2.7/site-packages/fake_useragent/fake.py", line 98, in __getattr__
    raise FakeUserAgentError('Error occurred during getting browser')  # noqa
FakeUserAgentError: Error occurred during getting browser

当我同时运行多个蜘蛛时，我在 Linux 服务器上不断收到此错误。这个错误在我自己的笔记本电脑上很少发生。我应该怎么做才能避免这种情况？我需要提高内存还是其他什么？服务器的规格为 512MB RAM 和 1 个 vCPU。