Googlebot在Rails 4.1上导致无效的跨源请求(COR)

Question

如何在抓取网站时阻止Google导致此错误？我不想关闭"protect_from_forgery",除非这样做是安全的.

[fyi] method=GET path=/users format=*/* controller=users action=show status=200 duration=690.32 view=428.25 db=253.06 time=  host= user= user_agent=Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) session= params={""} ()
[hmm] Security warning: an embedded <script> tag on another site requested protected JavaScript. If you know what you're doing, go ahead and disable forgery protection on this action to permit cross-origin JavaScript embedding. (pid:)
[fyi] method=GET path=/users/123/flag format=*/* controller=users action=flag status=500 error='ActionController::InvalidCrossOriginRequest:Security warning: an embedded <script> tag on another site requested protected JavaScript. If you know what you're doing, go ahead and disable forgery protection on this action to permit cross-origin JavaScript embedding.' duration=26.50 time= host= user= user_agent= session= params= (pid)
[omg] ActionController::InvalidCrossOriginRequest (Security warning: an embedded <script> tag on another site requested protected JavaScript. If you know what you're doing, go ahead and disable forgery protection on this action to permit cross-origin JavaScript embedding.):
actionpack (4.1.4) lib/action_controller/metal/request_forgery_protection.rb:217:in `verify_same_origin_request'

控制器以此响应

respond_to do |format|
    format.js { render template: 'users/flag', layout: "some_layout" }
end

我无法重新创建错误,当我通过浏览器执行此操作时似乎工作正常

到目前为止,我已经查看了以下资源,但大多数似乎只是建议盲目转向CSRF或没有答案.

• 在comfy中使用特定于布局的javascript会导致InvalidCrossOriginRequest
• 升级到Rails 4.1后无效的跨源请求
• 如何避免ActionController :: InvalidCrossOriginRequest异常？
• Googlebot请求png然后我的整个Heroku网站崩溃.到底是怎么回事？
• https://github.com/rails/rails/pull/13345
• http://myownpirateradio.com/tag/rails-authentication-token/
• https://gist.github.com/aishek/8535082
• 为什么Google会在(1)之前提前; 他们的JSON回复？
• http://www.tsheffler.com/blog/?p=428
• http://edgeapi.rubyonrails.org/classes/ActionController/RequestForgeryProtection.html

---

澄清:应该保护该操作免受CSRF的影响,但我想阻止Google抓取它或者通过抓取页面生成错误.即.)我希望虚假的安全警告消失而不会实际损害我的安全功能.

Answer 1

Googlebot使用格式"*/*"(http://apidock.com/rails/Mime),应用程序呈现js,因为它是唯一可用的东西.由于它是远程的,它正确地导致无效的COR.

这可以使用以下方法重现:

curl -H "Accept: */*" https://www.example.com/users/123/flag

解决方法是为蜘蛛抓取一个html后备资源:

respond_to do |format|
  format.html { render template: 'users/flag' }
  format.js { render template: 'users/flag', layout: "some_layout" }
end