使用 ruby​​ 2.5.3 在特定站点上发出获取请求时出现超时错误

Question

任何人都可以从 www.coupang.com 网站得到正确的回复吗？我不断向“ https://www.coupang.com/ ”提出请求，但 10 次中有 9 次出现错误。（有时它有效！令人惊讶。）

Traceback (most recent call last):
        14: from lib/add_sup/test.rb:7:in `<main>'
        13: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:485:in `get_response'
        12: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:609:in `start'
        11: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:910:in `start'
        10: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:487:in `block in get_response'
         9: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:1365:in `request_get'
         8: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:1464:in `request'
         7: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:1491:in `transport_request'
         6: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:1491:in `catch'
         5: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:1494:in `block in transport_request'
         4: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http/response.rb:29:in `read_new'
         3: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http/response.rb:40:in `read_status_line'
         2: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/protocol.rb:167:in `readline'
         1: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/protocol.rb:157:in `readuntil'
/Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/protocol.rb:181:in `rbuf_fill': Net::ReadTimeout (Net::ReadTimeout)

我也尝试过使用 python3 发出请求，它工作正常。我认为我的 ruby​​ 配置或 ruby​​ 本身有问题。

Traceback (most recent call last):
        14: from lib/add_sup/test.rb:7:in `<main>'
        13: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:485:in `get_response'
        12: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:609:in `start'
        11: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:910:in `start'
        10: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:487:in `block in get_response'
         9: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:1365:in `request_get'
         8: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:1464:in `request'
         7: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:1491:in `transport_request'
         6: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:1491:in `catch'
         5: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:1494:in `block in transport_request'
         4: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http/response.rb:29:in `read_new'
         3: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http/response.rb:40:in `read_status_line'
         2: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/protocol.rb:167:in `readline'
         1: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/protocol.rb:157:in `readuntil'
/Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/protocol.rb:181:in `rbuf_fill': Net::ReadTimeout (Net::ReadTimeout)

我很感激你对这个问题的好主意。谢谢！

Answer 1

他们使用 akamai，所以首先 - 他们期望 HTTP/2（为此你需要一个 http2 gem）并且他们有一些相当严格的用户代理嗅探。

这是使用net-http2 的示例

client = NetHttp2::Client.new "https://www.coupang.com/"
res = client.call :get, '/', headers: { "User-Agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6)" }
puts res.body if res.ok?