ROBOTS.TXT

Tell NBT how you want it to crawl.

NBT checks robots.txt before normal crawling and caches it for a sensible period. We do not fetch it again before every page.

Block NBT completely

User-agent: NBTBot
Disallow: /

Allow selected areas and slow us down

User-agent: NBTBot
Allow: /products/
Allow: /collections/
Disallow: /account/
Disallow: /checkout/
Crawl-delay: 5

If there is no more specific NBT group, we also respect the matching wildcard group.

Content Signals

NBT does not use customer content to train models. If a site publishes Content Signals, we use them to decide what later processing is allowed. For example, ai-input=no means we do not send that content through an optional model-based analysis step.

User-agent: *
Content-Signal: search=yes, ai-input=no, ai-train=no, use=reference
Allow: /