我有一个抓取工具(基于 Python 3.4.2 和 asyncio/aiohttp 库)和一堆链接(> 10K)来检索一些少量数据。部分爬虫代码:
@asyncio.coroutine
def prepare(self, links):
semaphore = asyncio.Semaphore(self.limit_concurrent)
tasks = []
result = []
tasks = [self.request_data(link, semaphore) for link in links]
for task in asyncio.as_completed(tasks):
response = yield from task
if response:
result.append(response)
task.close()
return result
@asyncio.coroutine
def request_data(self, link, semaphore):
...
with (yield from semaphore):
while True:
counter += 1
if counter >= self.retry:
break
with aiohttp.Timeout(self.timeout):
try:
response = yield from self.session.get(url, headers=self.headers)
body = yield from response.read()
break
except asyncio.TimeoutError …Run Code Online (Sandbox Code Playgroud) 我在 django 2.2 中运行命令时遇到问题。该应用程序在INSTALLED_APPS. 我收到一条消息:
Unknown command: 'my_command'
Type 'manage.py help' for usage.
Run Code Online (Sandbox Code Playgroud)
下面是项目的结构:
project/
app/
managment/
commands/
my_command.py
models.py
views.py
...
Run Code Online (Sandbox Code Playgroud)
my_command.py
from django.core.management.base import BaseCommand
class Command(BaseCommand):
help = 'My custom command'
def handle(self, *args, **options):
code here
Run Code Online (Sandbox Code Playgroud)