Crawl-delay is a minimum
If robots.txt asks us to wait longer, we wait longer. A workflow setting cannot make that shorter.
CRAWL POLICY
Measured traffic is part of the product.
Every public request goes through the same queue and site-level controls. “Run now” is not a bypass.
Before we make a request
- Check scope.
The target has to be inside the customer-authorised test. - Read site policy.
We load the relevant robots.txt, Crawl-delay and Content Signals. - Wait for the site-level traffic slot.
Runs aimed at the same site share the same pressure and pacing limits. - Attach the right identity.
Eligible direct requests are signed before they leave NBT. - Make the request and record the outcome.
Rate limits, blocks and Retry-After feed back into later work.
How we react
Retry-After wins
If a site tells us when to come back, we follow it. Without one, we use a cautious cooldown rather than repeatedly checking.
Blocks slow or stop related work
Responses such as 403, 429, 430 and 503, plus challenge pages, can pause later requests to that site.
Safe duplicate reads are reduced
Where it is safe, we reuse fresh public reads or revalidate them instead of fetching the same thing again for separate internal consumers.
We do not respond to a block by changing User-Agent, proxy or browser transport just to force the same request through.
Paid or metered access
An HTTP 402 Payment Required or pay-per-crawl requirement is treated as a stop, not another retry. NBT does not buy access automatically. Any paid-access feature would need explicit customer approval, a fixed spend limit and clear evidence of what was paid for.
External links
Links to other sites are references by default, not new crawl targets. If a customer turns on external link checking, those destinations get their own policy checks and a very small request budget.