Visual Regression Testing
Please refer to this branch for the work done around visual regression testing using Playwright’s visual comparisons: https://github.com/kianamcc/SynapseWebClient/tree/visual-regression-testing
Visual Regression testing captures screenshots of the UI and compares them against “baseline” images to detect unintended visual changes. A use case for this is the migration to load SWC styles through Vite. This migration can potentially cause breaking changes in the UI because the order in which styles are loaded may change. Instead of manually looking through the entirety of synapse.org to verify every page, these tests automatically detect these differences for us.
After research and discussion, we decided not to integrate this into SWC. One of the reasons being that it is high maintenance. Authenticated pages often contained dynamic content such as project names and timestamps that differed between runs. We attempted to hide dynamic elements in the code but managing these overrides across the entire site is not sustainable. Another reason is that to capture specific parts of the UI or wait for pages to load we used unstable selectors. A simple refactor the HTML structure or change in CSS classes would break the test. Another issue was the complexity of generated styles. Our environment uses styles that are generated at runtime (CSS-In-JS), often resulting in hashed names. This means a feature branch might generate a different hash than the baseline branch even if the UI looks identical, this would result in a failing test.
Resources: