找不到404,但可以从Web浏览器正常访问

Question

我在这上面尝试过很多网址,看起来很好,直到我遇到这个特定的网址:

require 'rubygems'
require 'nokogiri'
require 'open-uri'

doc = Nokogiri::HTML(open("http://www.moxyst.com/fashion/men-clothing/underwear.html"))
puts doc

这是结果:

/Users/macbookair/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/open-uri.rb:353:in `open_http': 404 Not Found (OpenURI::HTTPError)
    from /Users/macbookair/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/open-uri.rb:709:in `buffer_open'
    from /Users/macbookair/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/open-uri.rb:210:in `block in open_loop'
    from /Users/macbookair/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/open-uri.rb:208:in `catch'
    from /Users/macbookair/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/open-uri.rb:208:in `open_loop'
    from /Users/macbookair/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/open-uri.rb:149:in `open_uri'
    from /Users/macbookair/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/open-uri.rb:689:in `open'
    from /Users/macbookair/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/open-uri.rb:34:in `open'
    from test.rb:5:in `<main>'  

我可以从网络浏览器访问它,我根本就没有得到它.

发生了什么,我该如何处理这种错误？我可以忽略它,让其余的工作吗？

Answer 1

你得到了404 Not Found (OpenURI::HTTPError),所以,如果你想让你的代码继续下去,请抢救那个例外.这样的事情应该有效:

require 'nokogiri'
require 'open-uri'

URLS = %w[
  http://www.moxyst.com/fashion/men-clothing/underwear.html
]

URLs.each do |url|
  begin
    doc = Nokogiri::HTML(open(url))
  rescue OpenURI::HTTPError => e
    puts "Can't access #{ url }"
    puts e.message
    puts
    next
  end
  puts doc.to_html
end

您可以使用更多通用异常,但是在遇到奇怪的输出时遇到问题或者可能以导致更多问题的方式处理不相关的问题,因此您需要确定所需的粒度.

您甚至可以嗅探HTTPd标头,响应状态,或者如果您想要更多控制并想要为401或404执行不同的操作,请查看异常消息.

我可以从网络浏览器访问它,我根本就没有得到它.

那么,这可能是服务器端发生的事情:也许他们不喜欢你发送的UserAgent字符串？该OpenURI文档演示了如何更改标题:

可以通过可选的哈希参数指定其他头字段.

open("http://www.ruby-lang.org/en/",
  "User-Agent" => "Ruby/#{RUBY_VERSION}",
  "From" => "foo@bar.invalid",
  "Referer" => "http://www.ruby-lang.org/") {|f|
  # ...
}

Answer 2

您可能需要将"User-Agent"作为参数传递给open方法.某些站点需要有效的用户代理,否则他们只是不响应或显示404未找到错误.

doc = Nokogiri::HTML(open("http://www.moxyst.com/fashion/men-clothing/underwear.html", "User-Agent" => "MyCrawlerName (http://mycrawler-url.com)"))