Check Performance Data - Playwright E2E suite

End-to-end tests for the Check Performance Data web app. Drives a real Chromium browser against a deployed instance — no in-process server, no mocked data layer. Lives at tests/DfE.CheckPerformanceData.E2ETests/.

This doc explains the shape of the suite: what it tests, the design choices behind it, and the gotchas that drove those choices. The test project's own README is the reference for how to run it (commands, env vars, snapshot update workflow). Read this first to orient, then the README when you need the dial settings.

Versions

All E2E-facing versions in one place. Source of truth for NuGet pins is the test project's own .csproj: src/Directory.Packages.props (central package management) only reaches the projects under src/.

Required on the host (for `make test-e2e-fast` / native runs)

Tool	Version	Notes
.NET SDK	10.0.x	Test project targets `net10.0`.
PowerShell (pwsh or Windows PowerShell)	5.1+ / Core 7+	Only needed once, to run `playwright.ps1 install chromium`.
Docker	any recent	Required for `make test-e2e` (canonical container path) and for the compose stack the host run hits.

Required for the container path (`make test-e2e`)

Tool	Version	Notes
Docker + Compose v2	any recent	Compose profile `e2e` builds and runs the test image.
Image: `mcr.microsoft.com/playwright/dotnet`	`v1.64.1-noble`	Pinned, with its digest, in `tests/DfE.CheckPerformanceData.E2ETests/Dockerfile` (dependabot.md). Must move in lockstep with the `Microsoft.Playwright` NuGet version. Drift detection: `bash scripts/check-playwright-pin.sh`, which the Build workflow runs on every push and pull request. Dependabot raises the NuGet package only, so its pull request fails until this tag is changed to match (dependabot.md).
.NET SDK (inside image)	10.0.x	Comes with the Playwright image, which is built on `mcr.microsoft.com/dotnet/sdk:10.0` (v1.64.1 has SDK 10.0.402). The image digest fixes the version. The Dockerfile fails the build if the image does not have a .NET 10 SDK.

NuGet packages

Pinned in tests/DfE.CheckPerformanceData.E2ETests/DfE.CheckPerformanceData.E2ETests.csproj.

Package	Version	Purpose
`Microsoft.Playwright`	1.64.1	Browser automation. Must match the Docker image tag `v1.64.1-noble` — Chromium binary protocol mismatch otherwise.
`Microsoft.Playwright.Xunit`	1.64.1	xUnit `PageTest` base class + browser/page lifecycle.
`xunit`	2.9.3	Test framework.
`Xunit.SkippableFact`	1.5.85	`Skip.If(...)` / `Skip.IfNot(...)` for tests that need a precondition.
`xunit.runner.visualstudio`	4.0.0	VS / `dotnet test` runner.
`Microsoft.NET.Test.Sdk`	18.10.1	Test SDK.
`xRetry`	1.9.0	`[RetryFact]` for the rare flaky-by-design test.
`coverlet.collector`	10.1.0	Coverage.

Runtime stack the suite drives

What the tests connect to, not what they're built with. Pinned in docker-compose.yaml.

Component	Image	Version
Web app	`cypd_web:latest` (built locally from `src/DfE.CheckPerformanceData.Web/Dockerfile`)	matches branch
Postgres	`postgres`	`18.1-alpine`
Azure storage emulator	`mcr.microsoft.com/azure-storage/azurite`	`latest`

Container map

The compose file at the repo root wires up everything the suite needs. Profiles control which services start; the E2E flow uses the e2e profile, which is intentionally narrower than all.

┌─────────────────────────────────────────────────────────────────┐
│ docker network: cypd (bridge)                                   │
│                                                                 │
│  ┌──────────────────┐         ┌──────────────────┐              │
│  │ cypd_e2e_tests   │ HTTP    │ cypd_web         │              │
│  │ (profile: e2e)   ├────────►│ (profile: e2e)   │              │
│  │                  │  :8080  │                  │              │
│  │ playwright/      │         │ ASP.NET Core 10  │              │
│  │ dotnet:1.59.0    │         │ Razor + GDS      │              │
│  │ + .NET 10 SDK    │         └────────┬─────────┘              │
│  │                  │                  │                        │
│  │ runs:            │                  │ EF Core 10             │
│  │  dotnet test     │                  ▼                        │
│  └──────────────────┘         ┌──────────────────┐              │
│                               │ cypd_db          │              │
│                               │ (profile: e2e)   │              │
│                               │ postgres:18.1    │              │
│                               └──────────────────┘              │
│                                        ▲                        │
│                                        │ (web + rules-engine    │
│                                        │  share the DB)         │
│                               ┌────────┴─────────┐              │
│                               │ cypd_azurite     │              │
│                               │ (profile: e2e)   │              │
│                               │ azurite:latest   │              │
│                               │ blob :10000      │              │
│                               │ queue:10001      │              │
│                               └──────────────────┘              │
└─────────────────────────────────────────────────────────────────┘

Other containers in compose, NOT started by the e2e profile:
  cypd_rules_engine     profile: rules_engine, all
  cypd_pgadmin          profile: database, all   (browser DB UI on :5050)

What each container does in the E2E flow

Container	Image	Role in E2E
`cypd_web`	`cypd_web:latest` (built from `src/DfE.CheckPerformanceData.Web/Dockerfile`)	The system under test. Listens on `:8080`. The PlaywrightFixture polls `/healthcheck` here for readiness before any test runs.
`cypd_db`	`postgres:18.1-alpine`	Real Postgres on `:5432`. Web app's `ConnectionStrings__Postgres` points here. Persistent volume `postgres_data`.
`cypd_azurite`	`mcr.microsoft.com/azure-storage/azurite:latest`	Azure Blob + Queue emulator on `:10000` / `:10001`. Web app's `ConnectionStrings__AzureStorage` points here. Persistent volume `azurite_data`.
`cypd_e2e_tests`	`cypd_e2e_tests:latest` (built from `tests/DfE.CheckPerformanceData.E2ETests/Dockerfile`)	The runner. Mounts the repo at `/work`, restores into the named volume `cpd-nuget-cache`, executes `dotnet test`. Chromium ships pre-installed in the upstream Playwright base image.

Volumes

Volume	Purpose	Persists across runs
`postgres_data`	Postgres data dir	yes
`azurite_data`	Azurite blob/queue state	yes
`cpd-nuget-cache`	NuGet package cache for the test runner. First run is ~3-5 min cold; subsequent runs are seconds-to-output because this volume survives `--rm`.	yes
`pgadmin_data`	Only present when `pgadmin` profile is up. Not used by E2E.	yes

Readiness gating

depends_on in compose is the default service_started condition — that means "the container started", not "the app is up". The authoritative readiness gate is PlaywrightFixture.WaitForDeploymentReadyAsync, which polls ${CPD_E2E_BASE_URL}/healthcheck every 2s for up to 90s (configurable via CPD_E2E_READY_TIMEOUT_SECONDS). The web service has no compose-level healthcheck today; if you add one, the fixture's poll becomes redundant but harmless.

Running the tests

Three host-side flows: PowerShell (no IDE), VS Code, Visual Studio. They differ in how you trigger the run; the underlying dotnet test invocation is the same and the harness contract is identical.

Before any of them, the compose stack (web + db + azurite) must be reachable on http://localhost:8080 — that's what PlaywrightFixture polls for readiness. The two exceptions are make test-e2e (which runs inside the compose network and uses http://web:8080) and any flow where you've set CPD_E2E_BASE_URL to point at a different deployment.

Solution file note: the .NET solution is at src/DfE.CheckPerformanceData.slnx, not at repo root. Open that one in IDEs.

PowerShell

Run from the repo root (check-performance-data\). Native host SDK, no Docker for the test runner itself.

# 1. One-time: bring up the app stack (web + db + azurite).
docker compose --profile e2e up -d web db azurite

# 2. One-time per machine: install Chromium for Playwright.
#    Build first so playwright.ps1 exists.
dotnet build tests/DfE.CheckPerformanceData.E2ETests/ --configuration Release
powershell.exe -ExecutionPolicy Bypass `
  -File tests\DfE.CheckPerformanceData.E2ETests\bin\Release\net10.0\playwright.ps1 `
  install chromium

# 3. Run the suite.
dotnet test tests/DfE.CheckPerformanceData.E2ETests/ `
  --configuration Release

Useful filter variants:

# A single area, by namespace
dotnet test tests/DfE.CheckPerformanceData.E2ETests/ --filter "FullyQualifiedName~Admin"

# Single test by name
dotnet test tests/DfE.CheckPerformanceData.E2ETests/ --filter "FullyQualifiedName~SoftDeleteWikiPageTests"

To run the suite in the Linux Playwright container from PowerShell — same as make test-e2e:

docker compose --profile e2e run --rm e2e-tests

That builds the runner image (first time only), brings up dependencies, runs the full suite and tears down.

If you alternated between native and container runs and hit NETSDK1047 'project.assets.json' doesn't have a target for 'net10.0/win-x64':

Remove-Item -Recurse -Force tests\DfE.CheckPerformanceData.E2ETests\bin, tests\DfE.CheckPerformanceData.E2ETests\obj

VS Code

Requires the C# Dev Kit extension (ms-dotnettools.csdevkit) — it bundles Test Explorer integration for dotnet test. The base C# extension alone (ms-dotnettools.csharp) gives you OmniSharp + IntelliSense but no Test Explorer.

Open the repo folder in VS Code. The Solution Explorer panel (added by C# Dev Kit) will detect src/DfE.CheckPerformanceData.slnx.

Bring the stack up in the integrated terminal:

docker compose --profile e2e up -d web db azurite

One-time Chromium install (same as the PowerShell flow above).
Open the Testing view (flask icon in the sidebar). The DfE.CheckPerformanceData.E2ETests tree appears. Click ▶ next to a test, class, or the whole project to run.
To filter by category, run from the integrated terminal instead — Test Explorer doesn't expose [Trait] filtering directly:
```
dotnet test tests/DfE.CheckPerformanceData.E2ETests/ --filter "Category=Smoke"
```

Debug a single test: right-click the test in Test Explorer → "Debug Test". Set breakpoints in the test method or in production code; the debugger will attach to the test runner process. Note: it does not attach to the web app under test — that's still the cypd_web container. To debug into the web app code, attach a separate debugger to the container or run the web app directly from VS (see below).

Visual Studio (2022 17.12+ or 2026)

VS speaks .slnx natively in 17.12+ (LTSC) / 2026.

Open src/DfE.CheckPerformanceData.slnx. VS restores packages.
Bring the stack up. Easiest from the Developer PowerShell inside VS (View → Terminal):
```
docker compose --profile e2e up -d web db azurite
```
Or use the Containers tool window (View → Other Windows → Containers) to start individual services.
Test Explorer (Test menu → Test Explorer or Ctrl+E, T) lists every test grouped by project / namespace / class. Right-click a node → "Run" or "Debug".
To run only the smoke set, use the search box at the top of Test Explorer:
```
Trait:"Smoke"
```
(this is a VS Test Explorer filter syntax, not a dotnet test filter.)
One-time Chromium install is still required — VS doesn't run playwright.ps1 install for you. Build the test project once (Build → Build Selection on the E2ETests project), then in Developer PowerShell:
```
powershell.exe -ExecutionPolicy Bypass `
  -File tests\DfE.CheckPerformanceData.E2ETests\bin\Release\net10.0\playwright.ps1 `
  install chromium
```
(Or bin\Debug\net10.0\ if you've only built Debug.)

A solution build tears the stack down. docker-compose.dcproj is part of the solution and builds with it, and it sets DockerComposeProjectName to check_performance_data — the same project name as name: in docker-compose.yaml. VS therefore manages the same containers the CLI started, and its build-time compose lifecycle removes them. Test Explorer builds before it runs, so the containers can disappear underneath the very run you just started.

The symptom is the whole suite failing at once with Microsoft.Playwright.PlaywrightException : net::ERR_CONNECTION_REFUSED at http://localhost:8080/…. That is a missing app, not a broken test — and not CPD_E2E_BASE_URL either, which already defaults to http://localhost:8080. Confirm with docker compose ps: an empty list means the stack went away.

Recover with docker compose up -d, then re-run the tests without an intervening solution build. To avoid it entirely, either unload the docker-compose project in VS (right-click → Unload Project) and drive the stack from the CLI, or run the suite in its own container with make test-e2e, which manages its web dependency itself.

Debug a single test: right-click in Test Explorer → "Debug". VS attaches to the test process. As with VS Code, this debugs the test code, not the web app — the web app under test is still the container. To step through Razor + controller code while a test drives it, set the Web project as startup, F5 to launch it locally on :8080 (Postgres + Azurite still need to be running), then run the test against that local instance instead of the container.

Choosing which flow

You want to…	Use
Tight inner loop, functional tests only	PowerShell or VS/VSC Test Explorer
Step through a single failing test in a debugger	VS Code or VS, "Debug Test"
Reproduce a CI failure exactly	`make test-e2e` — same image, same Linux Chromium

What it tests

Two layers, both behind the same fixture:

Browser-driven UI tests (Wiki/, Web/) — Playwright loads pages, clicks things, asserts visible state. Covers the help CMS read path, search sidebar, soft-delete flow, warning-text rendering, GOV.UK assets, the wiki/content-block CRUD round trips, the 404 surface, and the sign-in nav cluster round-trip (anonymous → impersonate → sign-out → anonymous, in Web/SignInNavTests.cs).
HTTP-only tests (Web/) — HttpClient without a browser. Faster, used where the assertion is "controller redirected to X" or "endpoint returned status Y" and the rendered page isn't the point.

Smoke tests (HarnessSmokeTests.cs) prove the harness itself can reach the deployed app and scrape an antiforgery token before any real test runs.

Why a black-box harness, not WebApplicationFactory

The suite drives the deployed app over HTTP — docker compose up brings up Web + Postgres + Azurite, and the tests hit http://localhost:8080. No WebApplicationFactory, no in-process Postgres, no TestAuthenticationHandler. That choice is deliberate.

Realistic auth + cookies + CSRF. Antiforgery tokens come from the real Razor-rendered form (scraped via PlaywrightFixture.ScrapeAntiforgeryTokenAsync); cookies flow through the real ASP.NET pipeline. An in-process harness would short-circuit half the middleware and miss configuration drift.
Stack matches production. Postgres is the real postgres:18.1-alpine image, not a SQLite shim. EF query translation, Postgres-specific operators (FTS, etc.), and migrations are exercised end to end.
Same artefact across local + CI. The container path (make test-e2e) builds the test runner against the same compose stack CI uses. No "works on my machine but breaks in CI" failure mode for harness setup.

The trade-off is speed: cold start (compose up + readiness probe + first Playwright launch) is ~30-40s. We accept that. Tight inner-loop runs use make test-e2e-fast natively on the host.

Test categories

Tests are grouped by folder — Wiki/, Web/, Admin/ — and scoped with --filter "FullyQualifiedName~...". Category traits describe behaviour, not grouping:

Trait	Scope
`Smoke`	The pull request check. Shows that a deployed app is up and that each area and journey still works.
`FullRegression`	Marks a class whose tests are not all smoke tests.
`Slow`	Tests that wait on real timeouts or polling.

Page stabilisation

Helpers/PageStabilisationExtensions.cs exposes IPage.StabiliseAsync() — three things, in order:

Inject CSS that disables every animation and transition (* { animation: none !important; transition: none !important; }). GOV.UK Frontend has hover and focus transitions, so screenshots differ from run to run without this.
await document.fonts.ready — fonts load asynchronously, and a screenshot taken before fonts are ready captures fallback metrics that diverge once the real fonts arrive.
WaitForLoadStateAsync(NetworkIdle) — last-line defence for late-arriving images or background fetches.

Tests call this before a screenshot, or when assertions depend on a settled DOM.

The container test runner

make test-e2e is the canonical path. It runs the suite inside a thin overlay on the official mcr.microsoft.com/playwright/dotnet image (tag and digest in the Dockerfile), against the compose stack:

The base image already has Chromium, its system dependencies and the .NET 10 SDK. The overlay (tests/DfE.CheckPerformanceData.E2ETests/Dockerfile) only sets environment variables and checks the SDK is .NET 10. No pwsh playwright.ps1 install step is needed at runtime.
A named NuGet cache volume keeps dotnet restore warm across runs. Cold first run is ~3-5 min (image pull + cache warm); subsequent runs are seconds-to-output.
Compose profile e2e makes the runner opt-in — docker compose up won't start it, only docker compose --profile e2e run --rm e2e-tests does (which is what the Make target does).

make test-e2e-fast is the host path: same dotnet test command, RID-native. Use it when you want the inner loop tight.

Switching between the two flows can leave RID-mismatched bin/obj artefacts and produce NETSDK1047 'project.assets.json' doesn't have a target for 'net10.0/win-x64' (or linux-x64) on the next run. make clean-test-bin clears them; the targets are idempotent.

Test data isolation

The suite seeds wiki pages and content blocks via direct HTTP POST against the controller surface (SeedHelpers.SeedWikiPageAsync / SeedContentBlockAsync). No fixture seeds are committed; every test owns its data.

Each seeded entity uses an e2e-{Guid:N}- prefix on its slug or key. UUID prefix means tests across runs and across parallel CI shards never collide.
Cleanup is IAsyncLifetime.DisposeAsync per test class — wiki pages get soft-deleted; content blocks leak (no DELETE route exists for them, by design). The leak is harmless: the UUID prefix prevents test-run cross-contamination, and content-block volume is small.

The admin rules-editor tests are the one exception to per-test seeding. The rules they render come from the rules-config blobs (rules.json / country-languages.json), not from an HTTP seed. The web app self-seeds these from its image-bundled seed copies during startup seeding (it runs as Development in the harness), so bringing up web is enough — no per-test or extra compose step is needed. (When the rules-engine worker is also run locally, the azurite_init one-shot seeds the same blobs for it; the E2E stack doesn't run the worker.)

Auth state in tests

Real DfE Sign-In isn't in the harness — the OIDC handshake against the external test IdP is too slow and flaky to run on every pipeline. Instead, Helpers/AuthHelpers.cs drives the dev-only impersonation endpoints in DevImpersonationController, which set a cypd-dev-impersonation cookie that DevImpersonationClaimsTransformer reads on every request to overlay the editor role onto the principal.

Three helpers, three endpoints, three cookie states:

Helper	Endpoint	Cookie after	Test scenario it serves
`ImpersonateAsEditorAsync`	`/dev/impersonate/editor`	`editor`	Tests that need the CMS admin role (edit/delete/move).
`ImpersonateAsUnprivilegedUserAsync`	`/dev/impersonate/user`	`user`	Tests that need an authenticated principal without the editor role (authorisation guards, role-conditional UI).
`ClearImpersonationAsync`	`/dev/impersonate/clear`	(deleted)	Cleanup, or tests that need true-anonymous after an impersonated step.

The helpers thread the cookie value through TestHttpClients.ImpersonationCookieHeader so the shared no-redirect HttpClient can send it manually (Playwright's browser context manages its own cookies).

The editor / user distinction matters at the auth layer even though they look similar — editor triggers the claims transformer to add ClaimTypes.Role = WikiConstants.EditorRole; user is a synthetic authenticated principal with no roles. clear removes the cookie entirely, leaving the next request genuinely anonymous (no synthetic principal at all).

The endpoints 404 in production. They're guarded by IHostEnvironment.IsProduction() in the controller and by the DI registration in Program.cs so they can't be reached even via direct request.

Web/SignInNavTests.SignInCluster_RoundTrips_* is the end-to-end coverage for the sign-in cluster in _Layout.cshtml: anonymous user clicks the caret dropdown → "As CMS admin" → page flips to "Sign out (impersonating CMS admin)" → click → returns to anonymous and the cookie is genuinely deleted (not flipped to user). Unit-level Razor-source assertions for the same branches live in LayoutRenderTests in the UnitTests project — those catch a missing branch; the E2E test catches integration-level breakages (JS doesn't toggle, claims transformer doesn't apply the role, the cookie isn't actually being deleted server-side).

What's not covered

Real DfE Sign-In OIDC handshake. Dev impersonation covers authenticated paths in the harness; the real OIDC round-trip isn't exercised. That's a deliberate boundary — the IdP is external, the test slot is shared, and the value of running through it on every PR is low compared to the flake cost.
Cross-browser. Chromium only. Firefox/WebKit aren't part of the budget.
Mobile gesture interactions. Viewport sizes are set, but no touch/swipe simulation.
Trace replay in CI. Playwright's Tracing.StartAsync isn't wired into the harness — failure debugging in CI is by log + snapshot artefact. To enable trace replay locally, hook Context.Tracing.StartAsync/StopAsync around the test body and inspect the resulting .zip with playwright.ps1 show-trace.

Five-minute runtime budget

Hard cap: the full suite must complete within 5 min wall-clock on ubuntu-latest (2-4 vCPU). If a single test exceeds 30s, investigate before merging — almost always a WaitForLoadState.NetworkIdle waiting on an unrelated background fetch, fixable by switching to DOMContentLoaded plus an explicit Expect(...).ToBeVisibleAsync() on the element you actually care about.