Seeing ScoutSentinelBot in your logs?

What it is

Our customers monitor the domains their business depends on. For each of those domains, ScoutSentinelBot keeps a short inventory of public facts: which pages exist, which third-party scripts and frames they load, where forms send their data, which security headers are present, and which links answer with an error. Customers use it to notice when a script, a form destination or a page changes.

The principle: we only follow what a site publishes, and never go looking for what it does not. The crawler does not guess paths, scan ports, try logins, submit forms or test for vulnerabilities.

ScoutSentinelBot is also the identity of every other request discovery sends, not only the crawl: see our two identities for the full list, and requests that are not the crawl for what they fetch.

How to identify it

Discovery requests no longer carry the X-ScoutSentinel-Probe header; that header belongs to journey checks. Anyone can copy a user agent, so to identify us reliably, verify the signature; the allowlisting options below say what each approach proves.

Our two identities

All of our traffic carries one of exactly two user agents. Which one depends on who asked for the request.

Allowing both on Cloudflare

• Scan bot: allow verified bots once Cloudflare lists ScoutSentinelBot as a signed agent (registration is pending), or verify its Web Bot Auth signature yourself. Until then, a custom rule on the user agent (ScoutSentinelBot/1.0) is the stop-gap.
• Journey bot: it is not signed and never will be, so "verified bots" does not cover it. Use the customer's marker where you can, or a custom rule on the user agent (ScoutSentinel/1.0).

One custom rule covers both. Create it under Security, WAF, Custom rules, with your own host name, the action Skip (Super Bot Fight Mode and whichever managed or rate limiting rules challenge us), and place it first:

(http.host eq "www.example.com") and (cf.client.bot or http.user_agent contains "ScoutSentinelBot/1.0" or http.user_agent contains "ScoutSentinel/1.0")

The user-agent parts of the rule let through anything that claims to be us, because anyone can send those strings. Keep the rule to your own host names, and drop the ScoutSentinelBot/1.0 part once Cloudflare verifies our signature for you.

Verifying ScoutSentinelBot

Because discovery runs on Cloudflare's shared network, its IP addresses and reverse DNS say nothing about who sent a request. Instead, every request it makes, including each redirect hop and each request a rendered page makes, is signed with Web Bot Auth: HTTP Message Signatures (RFC 9421) with an Ed25519 key only we hold. Each request carries three headers:

• Signature-Agent: "https://api.scoutsentinel.com", where our public keys are published.
• Signature-Input, which covers @authority (your host name) and signature-agent, with tag="web-bot-auth", alg="ed25519", a random nonce, and a created and expires one minute apart. Its keyid is the key's JWK thumbprint (RFC 7638).
• Signature, the signature itself.

Our key directory is at https://api.scoutsentinel.com/.well-known/http-message-signatures-directory. It lists our current public key (its kid is the keyid our signatures carry) and, during a key rotation, the next or previous key too. The directory response is itself signed with each key it lists, so it cannot be swapped in transit. A request whose signature verifies against a key from that directory, for your host name and within its validity window, comes from us.

We have applied to register ScoutSentinelBot with Cloudflare as a verified signed agent. Registration is pending: until it completes, Cloudflare does not recognise our crawler as a verified bot, so Cloudflare's verified-bot settings do not yet apply to it.

What it fetches

• /robots.txt first, every time. Nothing else is requested if robots.txt answers with a server error or does not answer.
• The home page of hosts our customer already knows under their domain.
• For a domain whose name imitates one our customer monitors (a possible phishing site), at most one request for /robots.txt and one for the home page per lookalike check, without following redirects, to read its title and whether it asks for a password. Nothing else on that site is requested.
• Links on those pages to the same site, and the sitemaps your robots.txt lists. Links marked rel="nofollow", and every link on a page whose robots meta tag says nofollow, are not followed.
• HTML pages only, with GET requests only. Documents such as PDFs are listed by address and never downloaded. Tracking parameters are removed and at most one address per path is fetched.
• The web app manifest a crawled page links to (at most 32 KB, no redirects), to see which mobile apps it names.
• We read public app store listings for apps linked from your site: Apple's public lookup API, and the public Google Play page of the app, whose robots.txt we honour and of which we read only the meta tags and structured data. When a listing is read we also download the icon it names from Apple's or Google's image server (no redirects, at most 256 KiB), so that our customers see our copy rather than loading it from the store. Only the app's public id is sent, each listing is read at most once a week, and it is shared between our customers. Both are signed like every other discovery request.

It never sends cookies or credentials, never submits a form, and never follows a link to another site.

Requests that are not the crawl

These also identify as ScoutSentinelBot and are signed, but they are not part of the crawl above:

• The HTTP check of host names our customer already knows under its domain: one GET of the home page (following at most five redirects), the /favicon.ico it serves, and one GET of http:// (and one more when that redirects to plain HTTP on the same host) to see whether plain HTTP redirects to HTTPS. For a domain its owner has not verified with us, it is a single GET of the home page with no redirect followed.
• Published files meant to be read by machines: /.well-known/security.txt (and /security.txt), /.well-known/change-password, /.well-known/openid-configuration, /.well-known/apple-app-site-association, /.well-known/assetlinks.json, and the MTA-STS policy at mta-sts.<domain>/.well-known/mta-sts.txt for a domain that publishes an MTA-STS record. At most a few requests per host per run, and fewer for unverified domains.
• Files published for AI agents, at most once a day per host: /.well-known/agent-card.json (and /.well-known/agent.json), /.well-known/ai-catalog.json (or /.well-known/mcp/server-card.json), /.well-known/oauth-protected-resource and the authorization server metadata it names when that server is on the same domain, /.well-known/api-catalog, /llms.txt, /auth.md, /robots.txt, and, for a verified domain only, the home page asked for Markdown (headers only, the body is not read).
• MCP servers: for a verified domain only, an MCP server those files name on the same domain (at most two per host, once a day) is asked to initialize and to list its tools, and the session is then closed. We keep the server's name, version and tool names. We never call a tool, and a server that asks for credentials is not asked anything more.
• The origin exposure check: for a host behind a CDN, one of our regional runners asks the origin's own address for the home page (and, to compare, the CDN), following only redirects on the same host. The runner does not hold our key: our API signs a small set of requests for that one host when it hands out the check, valid only until the check's deadline, and the runner sends nothing with a signature that has expired.
• Historical URL checks: for a verified domain only, one HEAD request for a path the public Common Crawl index lists, after reading robots.txt.

The HTTP check and the published-file reads do not consult robots.txt: they read what a host publishes at its front door and at the addresses standards reserve for machines. To stop them, contact us (below) or block the user agent.

What is kept

Page contents are never stored. For each page the crawl keeps facts only:

• its address, the page that linked to it, its status, title (up to 200 characters) and whether it was rendered;
• the origins of scripts, frames and other requests it makes, which of them are third parties, detected technologies, and whether scripts lack integrity checks;
• for each form: where it sends data, its method and whether it has a password field, never any field values;
• whether security headers are present (HSTS, Content Security Policy, frame protection, content type options, referrer policy) and whether the page loads mixed content;
• addresses of same-site links (up to 100) and short link texts (up to 50 per page, 80 characters each, with email addresses, phone numbers and long digit runs removed), host names of other sites it links to, and addresses of linked documents;
• business phone numbers published on the page (up to 10), with a label only from a fixed list of department words such as "Claims" or "Sales"; numbers marked up as belonging to a person are skipped;
• role mailbox addresses such as info@ or support@; for every other email address, only its domain;
• mobile apps the page links to;
• the host names of other sites the page presents as related: language editions (hreflang) and the canonical copy, links in its navigation, header and footer or in a section about the owner's brands, and domain names written on privacy, terms and "about us" pages (never the text around them);
• to explain a failed or blocked request: redirects (without query strings), timings, an allow-list of response headers (such as server, cf-ray, content-type and retry-after), the request headers we sent (user-agent, accept, accept-language, content-type) and whether the request was signed.

From robots.txt we keep a summary (whether it allowed the crawl, which group applied, the crawl delay and your Content Signals), never the file. The results are not linked to any of our customers. Only each site's latest crawl is kept; the next crawl replaces it, and it is deleted once it is more than 90 days old and none of our customers has named the site in its own crawl record for 90 days.

Some pages are rendered in a headless browser

Tag managers add third-party scripts only when a page runs in a browser, and some sites build their pages with JavaScript. So the home page, and pages that look client-rendered (an empty application root, very little text, a notice asking for JavaScript), are also loaded in a headless browser. While rendering:

• Analytics and advertising beacons are blocked (for example Google Analytics and Google tag collection calls, the Meta pixel, DoubleClick and LinkedIn Insight), as are tracking pixels, media, fonts, long-lived connections and any request that is not a GET. A crawl should not appear in your analytics or spend your advertising budget.
• Scripts still load, so we can see which tags a tag manager injects; the requests they try to send are recorded and blocked as above.
• Nothing is clicked, typed, scrolled or submitted. Each page gets a fresh browser context, at most 15 seconds to load and up to 20 more seconds for its network requests to finish. No screenshots are taken.

How often and how much

• At most one request per second to a host, or slower when your robots.txt sets a Crawl-delay (honoured up to 10 seconds).
• For a domain whose owner has not verified it with us: at most 50 pages, a home page render plus up to five more, once a week.
• For a domain its owner has verified with us: at most 200 pages and 30 renders, once a day.
• Results are shared between our customers: however many of them watch your site, it is crawled no more often than this.

Allowing ScoutSentinelBot

If your bot protection blocks discovery and you are happy for us to read your public pages, use the first option that works for you. They are in order of how strongly they identify us. To allow our journey checks as well, one rule can cover both identities.

1. Recommended: allow requests whose signature verifies

Verify the Web Bot Auth signature as described above, at your origin or in code that runs in front of it (for example a Cloudflare Worker), and let a request through only when it verifies against a key from our directory for your host name. This is the only method that proves a request came from us, and it keeps working when we rotate keys.

2. When our Cloudflare registration completes: allow verified bots

Once Cloudflare lists ScoutSentinelBot as a verified signed agent, Cloudflare checks the signature for you, and a zone that allows verified bots lets the crawler through without a custom rule. Registration is pending, so this does not work yet. We will update this page when it does.

3. Temporary exception on Cloudflare: a user-agent rule

Before you use it: a rule that matches the user agent lets through anything that claims to be us, because anyone can send that string. Use it only until one of the options above is available to you, keep it as narrow as you can (one host name, the bot rules only), and remove it when our registration completes.

1. In the Cloudflare dashboard, open your zone, then Security and WAF, and create a custom rule.
2. Use the expression (http.user_agent contains "ScoutSentinelBot" and http.host eq "www.example.com"), with your own host name.
3. Choose the action Skip and select only what challenges the crawler: the Super Bot Fight Mode rules, and any managed or rate limiting rule that blocks it. The options offered depend on your plan.
4. Place the rule first and save it. The next crawl will use it.

If you block by country: the crawler runs from Cloudflare locations worldwide, so a country block can stop it. Add and not http.user_agent contains "ScoutSentinelBot" to that rule, with the same limitation as above, or allow verified bots once we are verified.

On some plans, Bot Fight Mode cannot be skipped by a custom rule; there, the options are to turn Bot Fight Mode off or to move to Super Bot Fight Mode, whose settings can be adjusted. The "Block AI bots" setting and Cloudflare's AI crawler controls may also stop the crawler, so check them if the rule alone does not help. If the crawler meets a Cloudflare Access login, that is expected for private hosts and nothing needs to change.

Keeping the crawler out instead

You do not need any of this to control the crawler: robots.txt, below, is honoured on every crawl, and Content Signals say how facts from your site may be used.

How to block it

Add a group for our token to your robots.txt. To block it everywhere:

User-agent: ScoutSentinelBot
Disallow: /

To keep it out of part of the site, or to slow it down:

User-agent: ScoutSentinelBot
Disallow: /account/
Crawl-delay: 5

Changes take effect on the next crawl: robots.txt is read at the start of every crawl and never cached between runs. Allow, Disallow, * and $ follow the robots exclusion standard (RFC 9309).

Content Signals

We honour Content Signals. The crawler reads Content-Signal lines in robots.txt, from the group for ScoutSentinelBot or otherwise the * group:

• ai-input: some customers ask an AI model to suggest which customer journeys to monitor, using facts from the crawl. If your robots.txt says ai-input=no, facts from your site are left out of that model input. The only exception is the verified owner of the domain, using facts about its own site.
• ai-train: we never train models on crawled content.
• search: we do not build a search index.

User-agent: *
Content-Signal: search=yes, ai-input=no, ai-train=no

Contact

If the crawler causes you a problem, or you want us to stop crawling a site you operate, email security@scoutsentinel.com with the hostname and, if you have them, the times of a few requests from your logs. General questions: hello@scoutsentinel.com. Our policies: trust centre, privacy, terms.