Executive Summary

What technologies (Playwright, Cypress, Selenium, etc.) should we use to orchestrate tests?

Recommend using Playwright as end-to-end testing framework, because it 

1. Utilizes Chrome DevTools Protocol (CDP) for direct control of the browser via websockets, which reduces test flakiness, increases test speed, and allows enhanced control of the browser with associated new features, as compared to tools implementing the WebDriver Protocol, which require an intermediary browser-driver.

2. Has cross-browser compatibility (Chrome, Firefox, Safari, Edge).

3. Has increasing popularity and community support and is backed by a large company (Microsoft).

4. Includes numerous feature differentiators that sets it apart from competitors, including out-of-the-box test parallelization locally and in CI, support for multiple users, multiple tabs, multiple browsers, and multiple origins, ability to write tests in multiple languages (TypeScript, Java, Python, .NET), superior developer tools, and locator and promise handling strategies that most closely align with existing WebEng Team products.

Associated trade-offs:

1. Not supported in much older browser versions and IE11.

2. Newer framework, with growing, but smaller community than other frameworks.

When should the end-to-end suite run? (e.g. per code change, cron schedule, before release)

Against what backend instance should the end-to-end suite run?

Recommend running end-to-end suite against backend dev stack locally during development and per code change in CI, so there is continual feedback about the impact of changes on functionality. Tests will need to clean-up any side-effects (e.g. creating a new project), so that tests can continue to pass against the same stack. 

Consider creating a set of smoke tests that run against staging stack before release to exercise functionality that isn’t implemented on the dev stack (e.g. sending emails for account verification) and to verify key workflows against a copy of production data.

Test parallelization and sharding will hopefully allow the test suite to scale without increasingly slow run times, locally or in CI.

Can this work be generalized to also run against Portals?

If we wanted to use the same framework, certainly. If we wanted to re-use the same tests, potentially. SRC components are used in both SWC and Portals, so as long as flows are similar (or could be conditionally adjusted based on the current environment), the same tests could be run against both apps. There are configuration options (e.g. baseURL) that would support selecting the tested application at runtime, see here.

However, if the user flows are sufficiently different, then consider writing separate end-to-end tests or integration tests per application instead.

What scenarios should be tested?

Recommend testing the most critical features and important user workflows as well as features that are difficult to test in other formats, such as validating that data is persisted and displayed across multiple screens. 

Recommend brainstorming as a team (and consider gathering input from other stakeholders, such as Governance, PMs, and Design) to create and prioritize a comprehensive list of critical features.

An initial list of important user workflows based on the Getting Started docs:

• Log in (email, OAuth2 provider)

• Create an account

• Agree to code of conduct / terms of use

• Request access to an entity

• Create a project

• Upload data, download data

• Create a table, add data to a table, query a table

Other critical features to consider: 

• ACT / Governance, e.g. user cannot download file without meeting ACLs and ARs

• Access workflows: e.g. a regular user requests access, an ACT user grants access, then the regular user has access

Action Items

1. Create ticket to introduce proof-of-concept to SWC – SWC-6514

   1. Run against backend dev stack

   2. Configure GitHub Actions with environment variables

   3. Design test workflow so that can run tests with side effects in parallel locally (e.g. each dev has their own credentials) and potentially run CI (e.g. register new user for each run)

2. Publish doc to Confluence

Framework Selection

General Approach

I started by identifying a variety of browser automation tools (Cypress, Playwright, Puppeteer, Selenium WebDriver, TestCafe), and then narrowed down the list by evaluating their underlying protocols, cross-browser compatibility, relative popularity, and differentiating features. Once I had narrowed down to two frameworks, I set up a proof of concept with each to evaluate developer tools and to create a variety of small tests.

Background

End-to-end testing aims to simulate real user scenarios by testing the entire application flow. Browser automation tools are often used to automate the execution of these tests. Headless browsers, i.e. browsers that can run without displaying the UI, are useful for running these tests in CI. In the past, headless browsers weren’t native to the browsers that people actually used, but were developed separately (e.g. PhantomJS, based on WebKit). Browser automation tools used the WebDriver protocol, which uses a browser-driver to translate test script commands into commands the browser could execute.

However, since 2017, Chrome and Firefox have added native support for headless browsers, so automated tests can be run in the officially supported browsers. In addition, the Chrome DevTools Protocol was released, which allows direct communication between the test script and the browser via web sockets. New browser automation tools were released that took advantage of this protocol or the paradigm of direct communication with the browser. 

A next generation W3C WebDriver protocol called WebDriver BiDi aims to standardize browser automation protocols and combine the best of both WebDriver “Classic” and CDP, but is currently under development.

I will start by evaluating the pros/cons of the protocols underlying the browser automation tools.

1) Underlying Protocols: Intermediary vs Direct Communication

*Selenium WebDriver offers a “Devtools Service”, which allows running CDP commands in tests. The service uses Puppeteer under the hood.

**Cypress currently uses its own implementation to control the browser in process, but per this comment, they are working on using CDP to communicate with the browser.

Summary

While the WebDriver protocol is a more established protocol with a large community, the lack of direct communication between the test script and browser execution leads to test flakiness. Additionally, there are certain features that can’t be readily implemented, such as manipulating network requests. While W3C is working to implement a next generation WebDriver protocol that uses web sockets for direct communication (WebDriver BiDi), the protocol is still in early stages and not fully implemented in any testing frameworks.

Both Chrome DevTools Protocol and Native Scripting approaches offer direct communication between the testing tool and browser, so the script can perform actionability checks before performing actions, such as waiting for network calls to finish or elements to appear before proceeding, which reduces test flakiness. New features, based on the enhanced control of the browser, can also be implemented.

Will proceed with tools that use protocols based on direct communication to decrease test flakiness and leverage features based on enhanced browser control: Cypress, Playwright, Puppeteer, Selenium WebDriver Devtools, TestCafe.

2) Cross-Browser Compatibility

*Selenium WebDriver Devtools Service uses Puppeteer under the hood.

Summary

Among the browser automation tools that directly communicate with the browser, either via CDP or via native scripting, Puppeteer has the least cross-browser compatibility without support for Safari or Edge. 

Will proceed with tools that at least have experimental support for all major browsers: Cypress, Playwright, TestCafe.

3) Popularity / Community / Maintenance 

Ideally, the selected testing framework will continue to be maintained and improved for many years into the future. Selecting an increasingly popular framework will also increase the likelihood that WebEng developers, both existing and new hires, are already familiar with the framework, which will decrease spin-up time and make it easier to maintain long term. 

However, predicting which frameworks will make the cut is difficult. Here, popularity, a large community, and active maintenance of the framework are used as indirect measures of the framework’s potential longevity.

TestCafe is the oldest framework –

• TestCafe: initially released Nov 7, 2016; 6.7 years old

• Cypress: initially released Sep 10, 2017; 5.8 years old

• Playwright: initially released Jan 31, 2020; 3.4 years old

However, Cypress has largest community, based on tags on stackoverflow (July 20, 2023):

• [cypress]: 9,680 questions

• [playwright]: 2,234 questions

• [testcafe]: 1,845 questions

And the most downloads (July 20, 2023):  

Open

But Playwright popularity seems to be rapidly increasing, based on GitHub repository star history (July 20, 2023):

Open

And more active management of their GitHub open issues (July 20, 2023): 

• Cypress: 1,336 open issues, 11,697 closed issues

• Playwright: 616 open issues, 9,883 closed issues

• TestCafe: 132 open issues, 4,409 closed issues

Summary

TestCafe is the oldest framework, but appears to be the least popular. Cypress is the most popular based on number of downloads and stackoverflow questions. However, Playwright appears to be increasing quickly in popularity since its release and has more active management of open issues.

Will proceed with the most popular or increasingly popular tools: Cypress, Playwright.

4) Cypress vs Playwright

Basics

Features

Developer Tools

Locator Strategies

The Web Engineering team currently uses the Testing Library in the synapse-web-monorepo and in MTB. Below is a table of the recommended locator strategies in Cypress vs Playwright.

*Cypress Testing Library can be used with Cypress, so could use the same findByRole, findByLabelText, etc…Cypress docs state that their testing philosophy aligns closely with Testing Library’s ethos and approach to writing tests. So, could use Cypress without depending on test ids.

**Playwright doesn’t have ‘findBy…’ or ‘queryBy…’ queries – locators always auto-wait and retry when needed, so don’t have to worry about choosing the right method. List operations (e.g. ‘getAllBy…’) will also be handled by the `getBy` method – playwright will return a list if assertions imply a list is expected.  https://playwright.dev/docs/testing-library#migrating-queries

Speed

According to this article, Cypress “exhibits a longer startup time” and “seems to be approximating Selenium speed in longer suites”, whereas for real-world scenarios, “Playwright tops the ranking” and shows “consistently faster execution times” in their tested scenarios.

Proof of Concept Evaluation

1. Set up each tool locally and run against the Synapse dev stack.

2. Use developer tools to generate a test and review test runs.

3. Write tests to 

   1. Show an alert when trying to create an account with an invalid email address

   2. Login with username/password from environment variables

   3. Re-use authenticated session across other tests to:

      1. Navigate to “My Favorites” page

      2. Show an alert when trying to create a new Project with an existing name

Cypress: Proof of Concept

https://github.com/Sage-Bionetworks/SynapseWebClient/compare/develop...hallieswan:SWC-6477-cypress

Setup

Running Tests

Cypress doesn’t offer a built-in way to run SWC for us before running Cypress tests. Instead, serve SWC in one terminal window, and then run Cypress in another terminal window. Each time the tests are run, SWC will quickly re-compile, since the majority of the tasks associated with mvn gwt:run have already been completed. 

NOTE: the tests will fail the first time they are run after starting up SWC, because SWC fully compiles after navigating to the Portal for the first time. However, each time that the tests are subsequently run, SWC compilation will be skipped because no input files changed. This could probably be adjusted so that SWC compiles before tests are run (e.g. see options in Cypress docs), but I didn’t dig into this further.

Dev Experience

Installation / Set-up

Installation includes Cypress GUI. CI workflow must be installed separately. Cypress GUI walks through creating and running tests, but requires shifting to/from the IDE to edit tests (at least in VS Code).

Relatively straightforward to set up authentication to be re-used across tests, but setting environment variables was a bit of a hurdle since Cypress “environment variables” are not the same as OS-level environment variables, see here.

Test Generation

Offers Cypress Studio as an experimental feature within its GUI for generating tests, which worked moderately well. However, the locator strategy used in Cypress Studio followed the general Cypress strategy rather than the Testing Library strategy, i.e. test ids and class names over accessibility attributes or elements visible to the user. Generated tests would require more editing to match the locator strategy used elsewhere on the WebEng team.

Writing Tests

Cypress uses “promise-like” commands, but await does not work as expected, so nested `.then` blocks are needed to use dynamic values in tests, which is a slightly different strategy than what is used elsewhere on WebEng.

Selecting elements in Cypress was difficult at first, since the strategy is different from what is used elsewhere on WebEng (e.g. react-testing-library). However, Cypress Testing Library can be used with Cypress, which makes writing tests much more similar.

Playwright: Proof of Concept

https://github.com/Sage-Bionetworks/SynapseWebClient/compare/develop...hallieswan:SWC-6477-playwright

Setup

Running Tests

During development, we can configure Playwright to run SWC for us before running any tests by setting webServer in playwright.config.ts, as described here. However, this means that every time we re-run the tests, all tasks associated with mvn gwt:run (or whatever we configure) will also be re-run. 

For quick feedback when initially writing a test, we can comment out the webServer block in the playwright.config.ts, serve SWC in one terminal window, and then easily re-run the tests in another terminal window. Each time the tests are run, SWC will quickly re-compile, since the majority of the tasks associated with mvn gwt:run have already been completed. This greatly shortens the time to run tests. NOTE: this set-up works out of the box with SWC. Unlike the proof of concept Cypress set-up, the tests don't fail on the first run due to SWC compilation.

Configuration

Projects can be used to run sets of tests against different browsers or with different conditions. The filter testIgnore can also be used to skip tests in particular projects, which can be useful for separating "logged out" vs "logged in" tests. For an example of this set up, see this guide. 

Each project can also have different baseURLs, which could be used to run a subset of tests on Portals as well as SWC, while storing all of the tests in the same location. A single project can be run by using the --project command line option. See more info and options in the Playwright docs.

Dev Experience

Installation / Set-up

Installation generates a GitHub Actions workflow, which can be adapted to our needs.

Straightforward to set up authentication and then re-use the authenticated user throughout, see here.

VS Code Extension and Integration

There are many features that integrate directly with VS Code, including identifying locators, debugging, and generating tests. See full list here. 

Test Generation

Tests can also be generated via a GUI interface with Playwright Inspector, which can be launched with yarn playwright codegen. Tests generally needed minimal tweaking for maintainability and alignment with general WebEng locator strategy.

Writing Tests

Playwright uses a similar locator strategy as the react-testing-library and standard promise handling, so writing tests felt familiar.

Summary

While Cypress has more extensive documentation and a larger community, the set up and dev experience using Playwright was superior. Its locator and promise handling strategies more closely aligned with existing Web Engineering products, which made for faster spin up and easier test writing. It was also much easier to get the tests running locally with SWC - Cypress doesn’t offer an option to automatically launch the app when running tests and failed on the first test run with SWC, since the portal hadn’t been compiled yet. Finally, the VS Code integration worked well to run tests from the IDE and the Playwright Inspector tools did a better job generating tests that aligned with our locator strategy out-of-the-box.

Additionally, Playwright’s feature differentiators are appealing for scalability (e.g. parallelization locally and in CI, configurable test sharding) and handling of more complex tests (e.g. handling hover events, iFrames, multiple tabs, browser contexts, users, etc).

Recommend Playwright.

Resources 

Underlying protocols

• Comparison of WebDriver Protocol and Chrome DevTools Protocol, with specific reference to Playwright features:  https://medium.com/slalom-build/playwright-vs-webdriver-the-future-of-browser-automation-854a7ae63218

• Comparison of architecture for common browser automation approaches: https://dev.to/jankaritech/different-approaches-protocols-to-automate-the-browser-39f1

• WebDriver BiDi: https://www.selenium.dev/documentation/webdriver/bidirectional/chrome_devtools/

• Headless browser description: https://en.wikipedia.org/wiki/Headless_browser

• Headless browser support in official browsers: https://hacks.mozilla.org/2017/12/using-headless-mode-in-firefox/

• WebDriver BiDi roadmap: https://developer.chrome.com/articles/webdriver-bidi/

Cypress vs Playwright

• Playwright vs testing-library: https://playwright.dev/docs/testing-library#migrating-queries

• Popularity and architecture: https://www.qawolf.com/blog/why-qa-wolf-chose-playwright-over-cypress

• General features: https://cathalmacdonnacha.com/cypress-vs-playwright-which-is-best-for-e2e-testing

• Comparison of writing tests for specific features: https://tsh.io/blog/cypress-vs-playwright/

• Thorough comparison: https://mtlynch.io/notes/cypress-vs-playwright/

Appendix

Underlying Protocols: Architecture Diagrams

Architecture diagrams from: https://dev.to/jankaritech/different-approaches-protocols-to-automate-the-browser-39f1

WebDriver Protocol: 

Chrome DevTools Protocol:

Native Scripting: