End-to-End Tests With Playwright: Frontend, Backend and Database · Part 14 of 14 in the series
Part 14: The Tests in Your Pipeline
Let GitHub or GitLab run all four sections on every push, in the same container that runs them on your machine, with a database beside it and the report kept for every run, red ones included.
End-to-End Tests With Playwright: Frontend, Backend and DatabasePart 14 of 14
On this page
Quest objectives
- Prepare the Playwright config for a pipeline: no forgotten test.only, and retries that are reported
- Run the suite in the Playwright image with a PostgreSQL service beside it
- Keep the HTML report of every run, also when tests fail
- Know how to add more browsers and how to split a run across machines
- Know where to start in a project of your own
The suite is complete: four sections, 81 tests, about nine seconds. So far it runs when you think of running it. This last part hands that job to a machine. Every push runs all sections, a red test shows up before anything is merged, and the report is kept, so you can look into a failure that happened on a machine you never see.
Most of the work for this is done. Since the last part the whole suite runs in a container, and a pipeline is little more than that container, started by a push.
Two settings for the pipeline
The config gets two lines that only matter in a pipeline. GitHub and GitLab both set the environment variable CI for everything they run, and the config asks for it. Add the lines to playwright.config.ts, below fullyParallel:
// In a pipeline: a forgotten test.only is an error, and a failed test gets two more tries.
forbidOnly: !!process.env.CI,
retries: process.env.CI ? 2 : 0,forbidOnly. While you work on one test, you can writetest.only, and Playwright runs that test and no other. Forget to take it out again, and a pipeline would run one test, skip the rest without a word and report green. WithforbidOnly, atest.onlyanywhere is an error.retries. In the pipeline a failed test gets up to two more tries. If it passes on one of them, the run is green, and the report lists the test as flaky, so it does not go unnoticed. Treat such an entry as a bug that is still to be found. On your own machine there are no retries: there you want to see a test fail the first time it does.
That was the last change to the config. Here it is in full, as it stands at the end of the series:
import { existsSync } from 'node:fs';
import { defineConfig, devices } from '@playwright/test';
import { MEMBER_STATE } from './tests/support/users.ts';
// The tests read the same settings as the app. Where there is no .env, such as in a
// pipeline, the variables are set from outside.
if (existsSync('.env')) process.loadEnvFile();
// The built frontend, served by `vite preview`.
const preview = 'http://localhost:4173';
export default defineConfig({
fullyParallel: true,
// In a pipeline: a forgotten test.only is an error, and a failed test gets two more tries.
forbidOnly: !!process.env.CI,
retries: process.env.CI ? 2 : 0,
reporter: [['list'], ['html']],
use: {
// Record every test, and keep the recording of those that fail.
trace: 'retain-on-failure',
screenshot: 'only-on-failure',
},
projects: [
{
name: 'setup',
testDir: './tests/setup',
testMatch: 'database.setup.ts',
},
{
name: 'login',
testDir: './tests/setup',
testMatch: 'login.setup.ts',
dependencies: ['setup'],
use: { ...devices['Desktop Chrome'] },
},
{
name: 'database',
testDir: './tests/database',
dependencies: ['setup'],
},
{
name: 'backend',
testDir: './tests/backend',
dependencies: ['login'],
use: { storageState: MEMBER_STATE },
},
{
name: 'frontend',
testDir: './tests/frontend',
use: { ...devices['Desktop Chrome'], baseURL: preview },
},
{
name: 'e2e',
testDir: './tests/e2e',
dependencies: ['login'],
use: { ...devices['Desktop Chrome'], storageState: MEMBER_STATE },
},
],
// Before the tests, Playwright builds the frontend and serves the result. The frontend
// tests open it there; the servers of the workers hand out the same files from web/dist.
webServer: {
command: 'npm run build && npm run preview',
url: preview,
reuseExistingServer: false,
},
});Two setup projects, the four sections and one web server for the built frontend. Apart from the two lines that ask for CI, nothing in it depends on where it runs.
The job
What the pipeline has to do is the same on both platforms:
Run in the Playwright image, the same one that made the reference pictures. The browsers are in it, and the visual tests compare pictures from the same environment.
Start a PostgreSQL beside it as a service. The job reaches it under the host name
postgres.Set two variables:
DATABASE_URL, which points at that service, andVISUAL, so the three visual tests run instead of being skipped. There is no.envin a pipeline, and the config does not need one.Install the packages with
npm ci, which installs exactly whatpackage-lock.jsonlists, and runnpx playwright test. Everything else happens inside the suite, as on your machine: the config builds the frontend, the setup project builds the template database, and every worker copies it and starts its own server.Keep the folder
playwright-report/, above all when tests have failed.
The file that says so looks different on the two platforms:
GitHub
On GitHub a pipeline is called a workflow, and workflows live in the folder .github/workflows. Create .github/workflows/tests.yml:
name: Tests
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
timeout-minutes: 15
# The same image that made the screenshots, with the browsers already in it.
container:
image: mcr.microsoft.com/playwright:v1.65.1-noble
options: --user 1001
services:
postgres:
image: postgres:18-alpine
env:
POSTGRES_USER: questboard
POSTGRES_PASSWORD: questboard
POSTGRES_DB: questboard
options: >-
--health-cmd "pg_isready -U questboard"
--health-interval 5s
--health-timeout 3s
--health-retries 10
env:
DATABASE_URL: postgres://questboard:questboard@postgres:5432/questboard
VISUAL: '1'
steps:
- uses: actions/checkout@v7
- run: npm ci
- run: npx playwright test --reporter=list,html,github
- uses: actions/upload-artifact@v7
if: ${{ !cancelled() }}
with:
name: playwright-report
path: playwright-report/
retention-days: 14From the top:
onsays when the workflow runs: on every push tomain, and for every pull request.The job gets a fresh machine (
runs-on) and at most 15 minutes (timeout-minutes), so a run that hangs is stopped.containermakes the steps run inside the Playwright image instead of directly on that machine.--user 1001is the user GitHub's runner works as, so the steps in the container work on the checked-out files as their owner. Playwright's documentation sets the same option for jobs in its image.services.postgresstarts the database in a container beside the job's, reachable under the name of the service,postgres. ThePOSTGRES_*values are those of yourcompose.yml, and the--health-…options are its health check in another spelling: the steps wait until the database reports that it is ready.envsets the two variables for all steps.The steps fetch the code, install and test.
--reporter=list,html,githubreplaces the reporters of the config for this run:listfor the log,htmlfor the report, andgithub, which turns every failed test into an annotation that GitHub shows with the run.The last step uploads the report. Normally a step does not run when an earlier one has failed, and that is exactly when you want the report.
if: ${{ !cancelled() }}lets it run after green and red runs alike, and leaves it out only when the run was cancelled.
You find the report on the run's Summary page, under Artifacts, for 14 days. Download playwright-report and unpack it, and you have the report you know from your machine.
GitLab
On GitLab the pipeline is one file in the top folder of the repository. Create .gitlab-ci.yml, with the dot in front:
tests:
# The same image that made the screenshots, with the browsers already in it.
image: mcr.microsoft.com/playwright:v1.65.1-noble
services:
- postgres:18-alpine
variables:
POSTGRES_USER: questboard
POSTGRES_PASSWORD: questboard
POSTGRES_DB: questboard
DATABASE_URL: postgres://questboard:questboard@postgres:5432/questboard
VISUAL: '1'
PLAYWRIGHT_JUNIT_OUTPUT_FILE: results.xml
script:
- npm ci
- npx playwright test --reporter=list,html,junit
artifacts:
when: always
paths:
- playwright-report/
reports:
junit: results.xml
expire_in: 2 weeksFrom the top:
testsis the name of the job. It has no conditions, so it runs for every push, on every branch.imagesays in which container the job runs. It is the Playwright image.servicesstarts a second container beside it for as long as the job runs. The job reaches it under the namepostgres.variableshas two kinds of entries. The threePOSTGRES_*ones configure the database, with the same user, password and database name as yourcompose.yml. The others are for Playwright:DATABASE_URLandVISUALfor the tests, and one for a reporter, which comes next.scriptinstalls and tests.--reporter=list,html,junitreplaces the reporters of the config for this run:listfor the job's log,htmlfor the report, andjunit, which writes the results into an XML file.PLAYWRIGHT_JUNIT_OUTPUT_FILEsays where.artifactsare the files GitLab keeps from a job.when: alwaysis the line that matters: the report is kept after a failed job too, and that is exactly when you want it. The line underreportshands the XML file to GitLab, which then shows the tests in the merge request and in the pipeline's tab Tests.
You find the HTML report on the job's page, under the job's artifacts (Browse), for two weeks. It is the report you know from your machine.
Push it
The pipeline starts with the push that brings its file to the platform. That takes a repository on GitHub or GitLab that your project pushes to. If it has none yet, create one there and connect your project folder to it first.
git add . takes everything along that is not committed yet. That has to include the reference pictures from the last part, because the pipeline compares against them:
$ git add .$ git commit --message "Run the tests in a pipeline"$ git push
Open the pipeline on your platform and watch the job. It prints the lines you know from your terminal, and at the end it should say what the container said in the last part: 81 passed. From now on that happens with every push.
A red pipeline is shown on the pull request or merge request. Whether it also blocks the merge is a setting of the repository on both platforms, not a part of the file. It is worth switching on.
If you need more
Two things the quest board can do without, and your project maybe not.
More browsers. The same tests in another browser are one more project each. Add two to projects in playwright.config.ts:
{
name: 'e2e-firefox',
testDir: './tests/e2e',
dependencies: ['login'],
use: { ...devices['Desktop Firefox'], storageState: MEMBER_STATE },
},
{
name: 'e2e-webkit',
testDir: './tests/e2e',
dependencies: ['login'],
use: { ...devices['Desktop Safari'], storageState: MEMBER_STATE },
},Both are copies of e2e with another device: Desktop Firefox runs the tests in Firefox, and Desktop Safari in WebKit, the engine of Safari. Run for this series, all end-to-end tests passed in both, 8 in each. The Playwright image already contains the two browsers, so the pipeline needs no change. On your machine, install them once:
$ npx playwright install firefox webkitWhy only the end-to-end tests? The sections database and backend have no browser at all. The mocked frontend tests could get two copies in the same way, if you want the details of the page checked in every browser too.
Sharding. When a suite has grown until one machine takes too long, Playwright can split the run. Each of these commands runs one of two parts of the suite:
$ npx playwright test --shard=1/2$ npx playwright test --shard=2/2
One after the other on one machine, that gains nothing. It pays off when every part runs on a machine of its own at the same time, and both platforms can start the parts as parallel jobs. At nine seconds, the quest board is a long way from needing it.
What you have now
You started with an app of three layers and not a single test. Now it has a suite in four sections that each have a job: the rules of the database, the answers of the API, the behaviour of the page against a mocked API, and a few flows through all of it. Every worker has a database and a server of its own, so the tests run in parallel and in any order. They log in once, check accessibility and looks, and run on every push in the same container as on your machine. When one of them turns red, the section tells you where to look, and the trace shows you what happened.
In a project of your own, do not start with all of it. Start with the isolation from part 4: a template database, and a copy of it for every worker. Everything else rests on tests that cannot see each other. Then add one section at a time.
The finished project, as it stands after this part, is here to download and take apart: questboard-finished.zip.
Comments
The comments are GitHub Discussions: sign in with GitHub to write one.
The comments come from GitHub, through giscus. Loading them sends your IP address to giscus.app and GitHub.