Availability is not task success: can customers complete what they came to do?

The status page is green. Every service met its availability target last month. And yet the contact centre logged a morning of calls from customers who could not buy a policy online. Both statements are true, because they measure different things.

This guide explains the difference between availability and task success, why the gap exists, and how to measure the thing customers and regulators care about: whether the task can be completed.

How the gap shows up

• An error inside a working page. The quote engine times out, and the site shows a polite message in a page that loads perfectly.
• A third party fails in the browser. Payment fields, address lookup or identity checks fail for customers while your servers see no errors.
• One step in a long chain breaks. Sign-in works, the quote works, but the step that issues the policy documents has stopped.
• It fails for some customers. One region, one browser, one product variant or one time window.
• The follow-up does not arrive. The confirmation page appears, but the email or the call-back the customer was promised does not.

In each case the availability numbers look fine. The customer’s experience is that the service did not work.

Why availability metrics miss it

Availability answers “did the component respond?”. It is measured per component, usually from inside the network, and usually as a status code and a response time. That is exactly right for running infrastructure, and it leaves three gaps:

• A task spans many components. Suppose a quote-and-buy journey depends on eight services, each 99.9% available, failing independently. The journey as a whole works about 99.2% of the time, which leaves roughly six hours of a 30-day month when it does not, against about 43 minutes for any one component.
• Errors are often served successfully. Many failures are rendered as normal pages with a 200 status, so they do not count against availability.
• Averages hide the moment. A month at 99.95% can contain a two-hour window in which nobody could buy.

How to measure task success yourself

1. Name the task and its success condition. “A customer can get a quote and buy a travel policy online, and sees a policy reference.” Be specific about the end state.
2. Check it end to end from outside. Script the task in a real browser with test data and run it on a schedule from the public internet. Each run gives a clear answer: completed, failed at step N, or could not be checked.
3. Use real-user data alongside. Funnel completion rates from analytics show whether real customers are finishing, though they arrive later and are affected by marketing and seasonality.
4. Report time, not averages. For each period, show how long the task could be completed, how long it could not, and the longest single disruption.
5. Keep “unknown” separate. A run that could not execute (a network problem between the checker and the site, for example) is neither a pass nor a fail. Report it as a coverage gap.
6. Say what was not tested. A check of the web journey says nothing about the mobile app or the phone line. Listing the gaps makes the numbers credible.

Where impact tolerances come in

UK banks, insurers and many other financial firms work under the FCA and PRA operational resilience rules. They identify their important business services, such as “customers can buy motor insurance online”, and set an impact tolerance for each: the most disruption the firm can tolerate before customers or markets are harmed. Since March 2025 firms have been expected to be able to stay within those tolerances in severe but plausible scenarios.

An impact tolerance is expressed in customer terms, usually as a length of time. Task success measured from outside is expressed the same way, which makes it a natural source of evidence when a firm assesses whether a disruption stayed within tolerance. It supports that self-assessment; it does not replace mapping, scenario testing or the firm’s own judgement.

What good evidence looks like

• Per task, over time. When it could be completed, when it could not, and for how long.
• The failure boundary. The first step that failed in each disruption, with the assertion and the screenshot.
• Coverage. How much of the period was actually checked, from where, and how many runs were unknown.
• What was out of scope. Channels and paths the checks did not cover.
• Integrity. Screenshots and responses hashed when they were captured, so they can be relied on later.

How ScoutSentinel does it

• Journeys are customer tasks. Each journey is checked from outside your systems in a real browser or through your API, and its status comes from its checks.
• Outcome and behaviour are separate. Whether the task completed is one signal; whether the site behaved differently while completing it is another. They are kept apart, not merged into one status.
• Unknown is a state. Runs that could not complete for reasons on our side are recorded as unknown, with the reason.
• Important business services and impact tolerances. From Team, group journeys under an important business service with its impact tolerance. ScoutSentinel measures each disruption against it and alerts at 50% and 80% of the tolerance and when it is breached.
• Reports. From Business, a report per important business service shows whether it stayed within tolerance, the longest disruption, how much of the period was checked and what was not tested.

See the example on the home page: our demo insurance site’s quote step fails while its uptime checks read healthy.