我有几个芹菜任务.crawl_all_sites_tasks提交多个crawl_each_batch任务.
第一个任务处理第一个for循环 - crawl_each_batch但是一旦完成,我得到这个错误 -
tasks.py
from utils import crawl_all_sites
@app.task(name='webcrawler.crawl_all_sites')
def crawl_all_sites_task():
print 'crawling tasks'
crawl_all_sites()
from utils import crawl_each_batch
@app.task(name='webcrawler.crawl_each_sites')
def crawl_each_batch_task(filename):
crawl_each_batch(filename)
Run Code Online (Sandbox Code Playgroud)
utils.py
from p_webcrawler_lib.p_crawler import main
files = ['splitfilesaa'
,'splitfilesab'
,'splitfilesac'
]
def crawl_each_batch(filename):
print 'utils_crawl_each_batch'
main('./apps/webcrawler/batch_files/'+filename)
def crawl_all_sites():
print 'utils_crawl_all_sites'
from tasks import crawl_each_batch_task
for each in files:
#crawl_each_batch(each).delay()
crawl_each_batch_task(each).delay()
Run Code Online (Sandbox Code Playgroud)
错误:-
[2015-11-18 03:36:52,013: ERROR/MainProcess] Task webcrawler.crawl_all_sites[31c84a68-0171-45ee-93ce-ab3a879dd8a7] raised unexpected: AttributeError("'NoneType' object has no attribute 'delay'",)
Traceback (most recent call last):
File "/Users/prem/state-dept/refactor/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/Users/prem/state-dept/refactor/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/Users/prem/state-dept/analytics_refactor/apps/webcrawler/tasks.py", line 21, in crawl_all_sites_task
crawl_all_sites()
File "/Users/prem/state-dept/analytics_refactor/apps/webcrawler/utils.py", line 21, in crawl_all_sites
crawl_each_batch_task(each).delay()
AttributeError: 'NoneType' object has no attribute 'delay'
Run Code Online (Sandbox Code Playgroud)
你应该打电话crawl_each_batch_task.delay(each).这将调用代表您的任务delay的Task实例的方法.
你正在这样做,你正在调用任务,好像它是任何常规的Python函数.所以你调用.delay()它的返回值,即None.因此你得到的例外.
| 归档时间: |
|
| 查看次数: |
4990 次 |
| 最近记录: |