通过 Selenium Python 在普通/无头模式下使用 ChromeDriver/Chrome 访问 Cloudflare 网站有什么区别

Question

我有一个问题 --headlessPython Selenium for Chrome 模式。

代码

 from selenium import webdriver
 from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

 CHROME_DRIVER_DIR = "selenium/chromedriver"

 chrome_options = webdriver.ChromeOptions()
 caps = DesiredCapabilities().CHROME
 chrome_options.add_argument("--disable-dev-shm-usage")
 chrome_options.add_argument("--remote-debugging-port=9222")
 chrome_options.add_argument("--headless")  # Runs Chrome in headless mode.
 chrome_options.add_argument('--no-sandbox')  # # Bypass OS security model
 chrome_options.add_argument("--disable-extensions")
 chrome_options.add_argument("--disable-gpu")

 browser = webdriver.Chrome(desired_capabilities=caps, executable_path=CHROME_DRIVER_DIR, options=chrome_options)

 browser.get("https://www.manta.com/c/mm2956g/mashuda-contractors")
 print(browser.page_source)
 browser.quit()

当我删除chrome_options.add_argument("--headless")所有工作正常时，但是--headless*有了下一个问题

Please enable cookies.

Error 1020 Ray ID: 53fd62b4087d8116 • 2019-12-04 11:19:28 UTC

Access denied

What happened?
This website is using a security service to protect itself from online attacks.

Cloudflare Ray ID: 53fd62b4087d8116 • Your IP: 168.81.117.111 • Performance & security by Cloudflare

普通模式和 有--headless什么区别？

Answer 1

我使用这个服务器端脚本进行了测试：

<?php
echo "<pre><code>";
var_dump($_SERVER);
echo "</code></pre>";
?>
<script>
    var el = document.getElementsByTagName('code')[0];
    for(var prop in window.navigator){
        var str = JSON.stringify(window.navigator[prop])
        el.innerHTML = el.innerHTML + "window.navigator." + prop + " = " + str + "\n";
    }
    var skip_props = ['parent', 'top', 'frames', 'self', 'window'];
    for(var prop in window){
        if (skip_props.indexOf(prop) > -1) { continue; }
        el.innerHTML = el.innerHTML + "window." + prop + " = ";
        var str = JSON.stringify(window[prop])
        el.innerHTML = el.innerHTML + str + "\n";
    }
</script>

我使用 ChromeDriver 加载此页面，无论是否使用--headless，并使用 打印输出print(driver.find_element_by_tag_name('code').text)。然后我对两个输出进行了比较。
这是我发现的差异：

• HTTPAccept-Language标头：en-US,en;q=0.9对比en-US
• HTTPUser-Agent标头：Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36vs Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/83.0.4103.61 Safari/537.36（请注意HeadlessChrome第二个字符串中的提及。）
• JavaScript window.navigator.plugins：{"0":{"0":{}},"1":{"0":{}},"2":{"0":{},"1":{}}}与{}
• JavaScript window.navigator.mimeTypes：{"0":{},"1":{},"2":{},"3":{}}与{}
• JavaScript window.outerWidth：1367与0
• JavaScript window.outerHeight：641与0

值得注意的是：在您发布的 Python 脚本中，您缺少几行来删除该window.webdriver属性（如果没有这一行，服务器检测您正在使用 WebDriver 是微不足道的）[ ref ]：

driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
    "source": """
    Object.defineProperty(navigator, 'webdriver', {
      get: () => undefined
    })
  """
})

Answer 2

User-AgentCloudflare 不喜欢HTTP标头。

要解决此问题，只需更改您的user-agentchrome 选项（以下代码适用于 Python 中的 Selenium）：

option.add_argument('--headless')
option.add_argument("user-agent=Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36")

Answer 3

我获取了您的代码，删除了可选参数并添加了一些参数来执行测试，如下所示：

\n\n

\n

• 代码块：

  \n\n

  from selenium import webdriver\nfrom selenium.webdriver.common.by import By\nfrom selenium.webdriver.support.ui import WebDriverWait\nfrom selenium.webdriver.support import expected_conditions as EC\n\noptions = webdriver.ChromeOptions() \noptions.add_argument("start-maximized")\noptions.add_argument("--headless")\noptions.add_experimental_option("excludeSwitches", ["enable-automation"])\noptions.add_experimental_option(\'useAutomationExtension\', False)\ndriver = webdriver.Chrome(options=options, executable_path=r\'C:\\Utility\\BrowserDrivers\\chromedriver.exe\')\ndriver.get("https://www.manta.com/c/mm2956g/mashuda-contractors")\nprint(driver.page_source)\ndriver.quit()\n

\n

• 控制台输出：

  \n\n

  <html class="js" lang="en-US" style="opacity: 1; visibility: visible;"><!--<![endif]--><head>\n<title>Access denied | www.manta.com used Cloudflare to restrict access</title>\n<meta charset="UTF-8">\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">\n<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1">\n<meta name="robots" content="noindex, nofollow">\n<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1">\n<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection">\n<!--[if lt IE 9]><link rel="stylesheet" id=\'cf_styles-ie-css\' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->\n<style type="text/css">body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!--><script type="text/javascript" src="/cdn-cgi/scripts/zepto.min.js"></script><!--<![endif]-->\n<!--[if gte IE 10]><!--><script type="text/javascript" src="/cdn-cgi/scripts/cf.common.js"></script><!--<![endif]-->\n\n\n\n</head>\n<body>\n  <div id="cf-wrapper">\n    <div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>\n    <div id="cf-error-details" class="cf-error-details-wrapper">\n      <div class="cf-wrapper cf-header cf-error-overview">\n    <h1>\n      <span class="cf-error-type" data-translate="error">Error</span>\n      <span class="cf-error-code">1020</span>\n      <small class="heading-ray-id">Ray ID: 53fd7c2fca12d5fc \xe2\x80\xa2 2019-12-04 11:36:52 UTC</small>\n    </h1>\n    <h2 class="cf-subheadline">Access denied</h2>\n      </div><!-- /.header -->\n\n      <section></section><!-- spacer -->\n\n      <div class="cf-section cf-wrapper">\n    <div class="cf-columns two">\n      <div class="cf-column">\n        <h2 data-translate="what_happened">What happened?</h2>\n        <p>This website is using a security service to protect itself from online attacks.</p>\n      </div>\n\n\n    </div>\n      </div><!-- /.section -->\n\n      <div class="cf-error-footer cf-wrapper">\n  <p>\n    <span class="cf-footer-item">Cloudflare Ray ID: <strong>53fd7c2fca12d5fc</strong></span>\n    <span class="cf-footer-separator">\xe2\x80\xa2</span>\n    <span class="cf-footer-item"><span>Your IP</span>: 123.201.54.43</span>\n    <span class="cf-footer-separator">\xe2\x80\xa2</span>\n    <span class="cf-footer-item"><span>Performance &amp; security by</span> <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>\n\n  </p>\n</div><!-- /.error-footer -->\n\n\n    </div><!-- /#cf-error-details -->\n  </div><!-- /#cf-wrapper -->\n\n  <script type="text/javascript">\n  window._cf_translation = {};\n\n\n</script>\n\n\n\n</body></html>\n

\n

\n\n\n\n

分析

\n\n

从提取的页面源代码中可以清楚地看到，使用--headless参数您将到达一个页面：

\n\n

\n
• 标题为：访问被拒绝 | www.manta.com 使用 Cloudflare 来限制访问。
\n
• 一些信息：发生了什么？：该网站正在使用安全服务来保护自己免受在线攻击。
\n

\n\n\n\n

结论

\n\n

浏览上下文（即Chrome 浏览器会话）被检测为BOT并且导航被阻止。

\n\n\n\n

尾奏

\n\n

您可以在以下位置找到一些相关讨论：

\n\n

\n
• 是否存在无法检测到的硒版本？硒真的无法检测到吗？
\n
• 检测到通过 ChromeDriver 启动的 Chrome 浏览器
\n
• 网页正在使用 Chromedriver 作为机器人检测 Selenium Webdriver
\n

\n