How to check customer journeys after every deployment

A release goes out, the pipeline is green, the health checks pass and the team moves on. Forty minutes later the contact centre reports that nobody can buy a policy online. The change that caused it was in the release, but nothing tested the journey in production.

This guide covers why that happens, how to verify customer journeys straight after each deployment, and how to connect a failure to the release that caused it.

How post-deployment breakage shows up

The failures that escape a pipeline are usually about production being different from where the tests ran:

• Configuration and secrets. A missing environment variable, a feature flag set differently, or a third-party key that exists in staging but not in production.
• Caches and assets. A CDN serves old JavaScript against a new API, or a new bundle references a file that did not publish.
• Real data and real services. A pricing rule, a product code or a payment provider behaves differently with production data.
• Changed markup. A button renamed or moved breaks a downstream script, a tag manager rule or an assistive-technology path.
• Partial exposure. The break affects one region, one device size or one A/B test group, so most internal checks pass.

It tends to surface slowly: a dip in conversion, then a handful of complaints, then an incident call where the first twenty minutes go on working out what changed.

Why pipeline tests and uptime checks miss it

• Pipeline tests are meant to be isolated. They run against staging, fixtures and mocks, which is what makes them fast and repeatable. It also means they are not production.
• Smoke tests are often shallow. A post-deploy smoke test that calls /health or loads the home page proves the release started, not that a customer can finish a task.
• Uptime monitors poll URLs. They see 200 OK from every page of a journey whose final step no longer works.
• Nobody links the timeline. When the failure is found, the release history sits in one tool and the monitoring in another.

How to do it yourself

1. Pick the journeys that carry the business. Two to five is enough to start: get a quote, buy a policy, sign in, make a claim.
2. Script them against production. Use a real browser and agreed test data: a test customer, a test card, a product the business is happy to have test orders against. Assert on the outcome at each step.
3. Trigger them from the pipeline. Add a job after the deployment completes. Allow a short wait for CDN caches to update, then run the journeys and report the result back to the pipeline.
4. Decide what a failure means. For some teams it blocks a promotion or triggers a rollback; for others it pages the on-call engineer. Write the rule down before the first failure.
5. Record the release. Store the version, the commit and the time next to each result. Reading “the first failure was nine minutes after release 42” beats searching chat history.
6. Keep checking on a schedule. Certificates, DNS, third-party scripts, payment providers and content changes break journeys without a deployment of yours. A schedule every few minutes catches those.
7. Compare with the last good run. Keep screenshots and network logs from the last passing run before the release, so the difference is visible at once.

If you run Playwright already, the same journey test can run in both places: in the pipeline straight after deploy, and on a schedule from a location outside your network.

What good evidence looks like

• The release and the failure on one timeline. When the release was recorded, when the last run passed, when the first run failed.
• The failing step and its assertion. Which step, what it expected and what it saw.
• Before and after. A screenshot from the last passing run and one from the first failing run, side by side.
• The network view. Failed requests, new status codes or a missing script in the network log of the failing run.
• Scope. Which locations and environments are affected, and whether the earlier steps of the journey still work.

A change that did not break the outcome is worth knowing about too. If a button is renamed but the journey still completes, that is a behaviour change to review, not an outage, and it helps to see the two separately.

How ScoutSentinel does it

• Run now. Every journey has a Run now button, and the API can trigger the same run from your pipeline with an API key (API access is included from Team).
• Release markers. Connect GitHub and each successful deployment becomes a marker, or post one from your pipeline through the API. Markers appear on the event timeline next to the runs. Release markers are included from Team.
• Scheduled checks keep running from Cloudflare edge locations between releases, as often as your plan allows.
• Outcome and behaviour are kept apart. Failed runs at the same step open an event. A change in how the journey behaves while it still passes is recorded as a behaviour change, separate from the outcome, and can open its own change event if you choose.
• Before and after evidence. The event pairs the last passing run with the first failing one: screenshots, and the HTML, network log and trace your check captures.

The free trial runs on Starter limits, so Run now and the scheduled checks are there from the first day; release markers and API-triggered runs need Team.