我有两个脚本,scraper.py和db_control.py.在scraper.py我有这样的事情:
...
def scrap(category, field, pages, search, use_proxy, proxy_file):
...
loop = asyncio.get_event_loop()
to_do = [ get_pages(url, params, conngen) for url in urls ]
wait_coro = asyncio.wait(to_do)
res, _ = loop.run_until_complete(wait_coro)
...
loop.close()
return [ x.result() for x in res ]
...
在db_control.py中:
from scraper import scrap
...
while new < 15:
data = scrap(category, field, pages, search, use_proxy, proxy_file)
...
...
从理论上讲,刮板应该在未知时间开始,直到获得足够的数据.但是当new不是imidiatelly > 15然后这个错误发生:
File "/usr/lib/python3.4/asyncio/base_events.py", line 293, in run_until_complete
self._check_closed()
File "/usr/lib/python3.4/asyncio/base_events.py", line 265, in …我跟进了这个教程:https://pawelmhm.github.io/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html,当我做50 000个请求时,一切正常.但我需要进行1百万个API调用,然后我对此代码有问题:
url = "http://some_url.com/?id={}"
tasks = set()
sem = asyncio.Semaphore(MAX_SIM_CONNS)
for i in range(1, LAST_ID + 1):
task = asyncio.ensure_future(bound_fetch(sem, url.format(i)))
tasks.add(task)
responses = asyncio.gather(*tasks)
return await responses
因为Python需要创建100万个任务,它基本上只是滞后然后Killed在终端中打印消息.是否有任何方法可以使用预先制作的(或列表)网址的发生器？谢谢.
我正在使用Django开发一个简单的网页,我需要实现搜索功能.我目前正在使用这样的东西:
search_box = request.GET['search_box']
X = Foo.objects.filter(Q(title__contains=search_box) | Q(info__contains=search_box)).values()
如果指定的列包含搜索的字符串,它会检查我的数据库,但如果我搜索"kočík"但我的数据库包含"kocik",该怎么办？我如何在Python 3中从字符串中删除diacritis,或者实现它的最佳方法是什么？谢谢
在我的views.py中,我有以下代码:
# Exceptions for GET requests
try:
page = int(request.GET["page"])
except Exception:
page = 1
try:
price_from = request.GET["price_from"]
except Exception:
price_from = -5
try:
price_to = request.GET["price_to"]
except Exception:
price_to = "all"
# ...
# Another 10+ try/except statements for now, but more will come
我需要从GET请求获取变量,这可以但不必在链接中声明.有没有更清洁/更好的方法来做到这一点,或者在代码中有大量的尝试/除外是正常的吗？谢谢.