---
title: NBT Bot robots.txt and Content Signals
canonical: https://nbt.bot/robots/
---

# robots.txt and Content Signals

NBT checks robots.txt before normal crawling and caches it for a sensible period.

## Block NBT

```text
User-agent: NBTBot
Disallow: /
```

## Slow NBT down

```text
User-agent: NBTBot
Allow: /
Crawl-delay: 5
```

Crawl-delay is a minimum. Other rules can make NBT slower, not faster.

## Content Signals

NBT does not use customer content to train models. If a site publishes Content Signals, we use them to decide what later processing is allowed.

```text
User-agent: *
Content-Signal: search=yes, ai-input=no, ai-train=no, use=reference
Allow: /
```

If robots.txt is temporarily unavailable or rate-limited, NBT waits. We do not treat that as permission.