---
title: NBT Bot crawl policy
canonical: https://nbt.bot/crawl-policy/
---

# NBT Bot crawl policy

Every public request goes through the same queue and site-level controls. “Run now” is not a bypass.

## Before a request

1. Check that the target is inside the customer-authorised scope.
2. Load robots.txt, Crawl-delay and Content Signals.
3. Wait for the shared site-level traffic slot.
4. Attach the right identity.
5. Make the request and record the result, including rate limits or Retry-After.

## Behaviour

- Crawl-delay is a minimum.
- Retry-After is followed when present.
- 403, 429, 430, 503 and challenge pages can slow or stop related work.
- NBT does not change identity, proxy or browser transport just to force a blocked request through.
- Safe duplicate public reads may be reused or revalidated to reduce repeat traffic.
- External links are references by default, not automatic crawl targets.

## Paid or metered access

HTTP 402 or pay-per-crawl is treated as a stop, not another retry. NBT does not buy access automatically. Any paid-access feature would need explicit customer approval, a fixed spend limit and clear evidence.