从 9gag Api 获取帖子

Question

我想通过 9gag.com Api 下载帖子，该 API 显然存在，但没有在任何地方记录。如果您在浏览器中检查此链接，您将收到 json 响应：https://9gag.com/v1/tag-posts/tag/wtf/type/fresh

所以使用httr2这个应该很容易。然而：

library(httr2)
req <- request("https://9gag.com/v1/tag-posts/tag/samantha/type/fresh")

req_dry_run(req)
#> GET /v1/tag-posts/tag/wtf/type/fresh HTTP/1.1
#> Host: 9gag.com
#> User-Agent: httr2/0.1.1 r-curl/4.3.2 libcurl/7.82.0
#> Accept: */*
#> Accept-Encoding: deflate, gzip, br, zstd

result <- req_perform(req) 
#> Error: HTTP 403 Forbidden.

我检查了 Firefox Inspect 中的网络选项卡，并复制了输入 URL 时发送的 GET 请求：

curl 'https://9gag.com/v1/tag-posts/tag/wtf/type/fresh' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Connection: keep-alive' -H 'Cookie: ts1=456f0f77f4669644e64d425b5f7ad509ba4f2385; ____ri=1792; ____lo=DE' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: none' -H 'Sec-Fetch-User: ?1' -H 'If-Modified-Since: Fri, 18 Mar 2022 11:04:32 GMT' -H 'Cache-Control: max-age=0' -H 'TE: trailers'

除了标题之外，看起来很相似。那么让我们尝试一下所有这些标头：

req <- request("https://9gag.com/v1/tag-posts/tag/samantha/type/fresh") %>% 
  req_headers(
    'User-Agent' = 'Mozilla/5.0 (X11; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0',
    'Accept' = 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
    'Accept-Language' = 'en-US,en;q=0.5',
    'Accept-Encoding' = 'gzip, deflate, br',
    'Connection' = 'keep-alive',
    'Cookie' = 'ts1=456f0f77f4669644e64d425b5f7ad509ba4f2385; ____ri=1792; ____lo=DE',
    'Upgrade-Insecure-Requests' = '1',
    'Sec-Fetch-Dest' = 'document',
    'Sec-Fetch-Mode' = 'navigate',
    'Sec-Fetch-Site' = 'none',
    'Sec-Fetch-User' = '?1',
    'If-Modified-Since' = 'Fri, 18 Mar 2022 03:03:44 GMT',
    'Cache-Control' = 'max-age=0',
    'TE' = 'trailers'
  )

result <- req_perform(req)
#> Error: HTTP 403 Forbidden.

^{由reprex 包于 2022 年 3 月 18 日创建(v2.0.1)}

不幸的是，这一请求仍然被拒绝。有人知道为什么吗？