仅针对最新浏览器版本进行端到端测试有什么好处

Question

当使用 Playwright、Puppeteer 和 Cypress 等工具执行端到端测试时，我相信所有这些工具（除了后者，如果我错了请原谅）只允许您使用每个浏览器的最新版本，即与每个版本捆绑在一起。

在这种情况下，我想知道运行端到端验收测试的真正好处是什么？这些测试只会针对更有限的场景验证功能，而不是确保您的应用程序仍然可以在 Chrome 70 上运行？

任何关于为什么只考虑最新的浏览器版本而忘记其余部分的见解都是一个好主意，通过使用 Playwright 而不是允许针对特定浏览器二进制文件进行定向测试的工具，例如，并使您的 E2E 测试结果如下：

PASS. Shopping-Cart (Modern)
FAIL. Shopping-Cart (chrome 70) 
// Refactor

代替

PASS. Shopping-Cart
// Yay! Release

Answer 1

Puppeteer, Playwright, Cypress with specific executable

First, let's clarify that it is possible to launch Puppeteer and Playwright with other executables than the bundled one in the case of Chrome/Chromium, so it is not specific to Cypress.

• Puppeteer: puppeteer.launch([options])

const browser = await puppeteer.launch({ executablePath: '/path/to/Chrome70' });

• Playwright: browserType.launch([options])

const browser = await chromium.launch({ executablePath: '/path/to/Chrome70' });

Second, even if 'testing' is there in the list of features in the case of Puppeteer and Playwright, they are not testing frameworks while Cypress is a testing tool primarily. Update (2023): Since I wrote this answer I'd argue with my statement that Playwright is not primarily a testing framework. It is indeed primarily a testing framework, see @playwright/test is more popular than the regular playwright library itself.

What's the benefit of E2E testing only against the latest browser versions?

Some years ago it was crucial to test across all browsers and as many versions as used by a mass of users.
Over time it has changed.

What's new in 2021?

• Only Google Chrome and Safari (Webkit) browsers have 10%+ market share. Statcounter.com, Wikpedia
• For the vast majority not even Firefox support is required (with its 3.68% market share worldwide)
• "Google Chrome on Windows and Mac auto-updates itself on a regular basis. The auto-updating procedure is performed by Google Update, which is based on the open-source Omaha project. Auto-updated provides fixes to sometimes critical issues, limiting exposure." chromium.org
• Due to the auto-updates the number of users with not the latest Chrome is very marginal. The previous versions (e.g. the last-but-one) go out of circulation in 1-2 months and remains under 1% share for a short time before vanishing. Desktop Browser Version Market Share Worldwide, Statcounter.com

Conclusion(s)

I.) It means for most products it is fairly enough and the best option to test against the latest browser versions.

You've mentioned Chrome 70 which is a relatively old version, released in October 2018, for most products it is not required to support it, they are in the same bucket as Internet Explorer 11 or legacy Edge.

II.) For some products, you may have to support specific (older) browser versions, in that case, the usage of specific executables can help.

By the way, if you decide to run tests against multiple browser versions it will be still possible with Puppeteer or Playwright as well, you just need to provide the right executables while the test suites are iterated over (e.g. in case of Jest: describe.each() can run the same tests or test suites with different configs and test data).