小编Max*_*Max的帖子

Scrapy 框架有 RobotsTxtMiddleware。它需要确保 Scrapy 尊重 robots.txt。它需要在设置ROBOTSTXT_OBEY = True中设置，然后 Scrapy 会尊重 robots.txt 策略。我做到了并运行了蜘蛛。在调试中，我看到了对http://site_url/robot.txt 的请求。

1. 这是什么意思，它是如何工作的？
2. 我如何处理响应？
3. 如何查看和理解robot.txt 中的规则？

我创建了视图：

def forgot_password(request):
"""
Actions when user forgot password.
:param request: object
:return: redirect to views.home() or 'loginsys/forgot.html' with form, message
"""
message = None
if request.method == 'POST':
    form = ForgotPasswordForm(request.POST)
    if form.is_valid():
        email = form.cleaned_data['email']
        user = User.objects.filter(email=email)
        salt = random_salt(len(user[0].username))
        code = signing.dumps([user[0].id, user[0].email, user[0].username],
                             key=settings.SECRET_KEY, salt=salt)
        url = settings.SITE_URL + reverse('loginsys:reset', args=[code])
        send_email.apply_async(('Welcome', '<p>Hello</p><p><a href="{0}">Go to this link</a></p>'.
                                format(url), [email]))
        store = ForgotPasswordLink()
        store.random_salt = salt
        store.user_link_id = user[0].id
        store.code_value = code
        store.save()
        forgot_password_salt_life.apply_async((user[0].id, salt), countdown=180) …