Executive Summary

What technologies (Playwright, Cypress, Selenium, etc.) should we use to orchestrate tests?

Recommend using Playwright as end-to-end testing framework, because it 

1. Utilizes Chrome DevTools Protocol (CDP) for direct control of the browser via websockets, which reduces test flakiness, increases test speed, and allows enhanced control of the browser with associated new features, as compared to tools implementing the WebDriver Protocol, which require an intermediary browser-driver.

2. Has cross-browser compatibility (Chrome, Firefox, Safari, Edge).

3. Has increasing popularity and community support and is backed by a large company (Microsoft).

4. Includes numerous feature differentiators that sets it apart from competitors, including out-of-the-box test parallelization locally and in CI, support for multiple users, multiple tabs, multiple browsers, and multiple origins, ability to write tests in multiple languages (TypeScript, Java, Python, .NET), superior developer tools, and locator and promise handling strategies that most closely align with existing WebEng Team products.

Associated trade-offs:

1. Not supported in much older browser versions and IE11.

2. Newer framework, with growing, but smaller community than other frameworks.

When should the end-to-end suite run? (e.g. per code change, cron schedule, before release)

Against what backend instance should the end-to-end suite run?

Recommend running end-to-end suite against backend dev stack locally during development and per code change in CI, so there is continual feedback about the impact of changes on functionality. Tests will need to clean-up any side-effects (e.g. creating a new project), so that tests can continue to pass against the same stack. 

Consider creating a set of smoke tests that run against staging stack before release to exercise functionality that isn’t implemented on the dev stack (e.g. sending emails for account verification) and to verify key workflows against a copy of production data.

Test parallelization and sharding will hopefully allow the test suite to scale without increasingly slow run times, locally or in CI.

Can this work be generalized to also run against Portals?

If we wanted to use the same framework, certainly. If we wanted to re-use the same tests, potentially. SRC components are used in both SWC and Portals, so as long as flows are similar (or could be conditionally adjusted based on the current environment), the same tests could be run against both apps. There are configuration options (e.g. baseURL) that would support selecting the tested application at runtime, see here.

However, if the user flows are sufficiently different, then consider writing separate end-to-end tests or integration tests per application instead.

What scenarios should be tested?

Recommend testing the most critical features and important user workflows as well as features that are difficult to test in other formats, such as validating that data is persisted and displayed across multiple screens. 

Recommend brainstorming as a team (and consider gathering input from other stakeholders, such as Governance, PMs, and Design) to create and prioritize a comprehensive list of critical features.

An initial list of important user workflows based on the Getting Started docs:

• Log in (email, OAuth2 provider)

• Create an account

• Agree to code of conduct / terms of use

• Request access to an entity

• Create a project

• Upload data, download data

• Create a table, add data to a table, query a table

Other critical features to consider: 

• ACT / Governance, e.g. user cannot download file without meeting ACLs and ARs

• Access workflows: e.g. a regular user requests access, an ACT user grants access, then the regular user has access

Action Items

1. Create ticket to introduce proof-of-concept to SWC – SWC-6514

   1. Run against backend dev stack

   2. Configure GitHub Actions with environment variables

   3. Design test workflow so that can run tests with side effects in parallel locally (e.g. each dev has their own credentials) and potentially run CI (e.g. register new user for each run)

2. Publish doc to Confluence

Framework Selection

General Approach

I started by identifying a variety of browser automation tools (Cypress, Playwright, Puppeteer, Selenium WebDriver, TestCafe), and then narrowed down the list by evaluating their underlying protocols, cross-browser compatibility, relative popularity, and differentiating features. Once I had narrowed down to two frameworks, I set up a proof of concept with each to evaluate developer tools and to create a variety of small tests.

Background

End-to-end testing aims to simulate real user scenarios by testing the entire application flow. Browser automation tools are often used to automate the execution of these tests. Headless browsers, i.e. browsers that can run without displaying the UI, are useful for running these tests in CI. In the past, headless browsers weren’t native to the browsers that people actually used, but were developed separately (e.g. PhantomJS, based on WebKit). Browser automation tools used the WebDriver protocol, which uses a browser-driver to translate test script commands into commands the browser could execute.

However, since 2017, Chrome and Firefox have added native support for headless browsers, so automated tests can be run in the officially supported browsers. In addition, the Chrome DevTools Protocol was released, which allows direct communication between the test script and the browser via web sockets. New browser automation tools were released that took advantage of this protocol or the paradigm of direct communication with the browser. 

A next generation W3C WebDriver protocol called WebDriver BiDi aims to standardize browser automation protocols and combine the best of both WebDriver “Classic” and CDP, but is currently under development.

I will start by evaluating the pros/cons of the protocols underlying the browser automation tools.

1) Underlying Protocols: Intermediary vs Direct Communication

*Selenium WebDriver offers a “Devtools Service”, which allows running CDP commands in tests. The service uses Puppeteer under the hood.

**Cypress currently uses its own implementation to control the browser in process, but per this comment, they are working on using CDP to communicate with the browser.

Summary

While the WebDriver protocol is a more established protocol with a large community, the lack of direct communication between the test script and browser execution leads to test flakiness. Additionally, there are certain features that can’t be readily implemented, such as manipulating network requests. While W3C is working to implement a next generation WebDriver protocol that uses web sockets for direct communication (WebDriver BiDi), the protocol is still in early stages and not fully implemented in any testing frameworks.

Both Chrome DevTools Protocol and Native Scripting approaches offer direct communication between the testing tool and browser, so the script can perform actionability checks before performing actions, such as waiting for network calls to finish or elements to appear before proceeding, which reduces test flakiness. New features, based on the enhanced control of the browser, can also be implemented.

Will proceed with tools that use protocols based on direct communication to decrease test flakiness and leverage features based on enhanced browser control: Cypress, Playwright, Puppeteer, Selenium WebDriver Devtools, TestCafe.

2) Cross-Browser Compatibility

*Selenium WebDriver Devtools Service uses Puppeteer under the hood.

Summary

Among the browser automation tools that directly communicate with the browser, either via CDP or via native scripting, Puppeteer has the least cross-browser compatibility without support for Safari or Edge. 

Will proceed with tools that at least have experimental support for all major browsers: Cypress, Playwright, TestCafe.

3) Popularity / Community / Maintenance 

Ideally, the selected testing framework will continue to be maintained and improved for many years into the future. Selecting an increasingly popular framework will also increase the likelihood that WebEng developers, both existing and new hires, are already familiar with the framework, which will decrease spin-up time and make it easier to maintain long term. 

However, predicting which frameworks will make the cut is difficult. Here, popularity, a large community, and active maintenance of the framework are used as indirect measures of the framework’s potential longevity.

TestCafe is the oldest framework –

• TestCafe: initially released Nov 7, 2016; 6.7 years old

• Cypress: initially released Sep 10, 2017; 5.8 years old

• Playwright: initially released Jan 31, 2020; 3.4 years old

However, Cypress has largest community, based on tags on stackoverflow (July 20, 2023):

• [cypress]: 9,680 questions

• [playwright]: 2,234 questions

• [testcafe]: 1,845 questions

And the most downloads (July 20, 2023):  

Open

But Playwright popularity seems to be rapidly increasing, based on GitHub repository star history (July 20, 2023):

Open

And more active management of their GitHub open issues (July 20, 2023): 

• Cypress: 1,336 open issues, 11,697 closed issues

• Playwright: 616 open issues, 9,883 closed issues

• TestCafe: 132 open issues, 4,409 closed issues

Summary

TestCafe is the oldest framework, but appears to be the least popular. Cypress is the most popular based on number of downloads and stackoverflow questions. However, Playwright appears to be increasing quickly in popularity since its release and has more active management of open issues.

Will proceed with the most popular or increasingly popular tools: Cypress, Playwright.

4) Cypress vs Playwright

Basics

Features

Developer Tools