VERIFY NBT
Check the request, not just the name.
Anyone can copy a User-Agent. A valid Web Bot Auth signature is stronger proof that a direct crawler request came from NBT.
Expected direct crawler identity
User-Agent: NBTBot/1.0 (+https://nbt.bot/)
NBT-Scan-Id: <opaque run UUID>
Signature-Agent: "https://nbt.bot"
Signature-Input: nbt=(...)
Signature: nbt=:...:The scan ID is just a trace ID. It does not contain a customer name, email address or secret.
Public signing keys
Direct Web Bot Auth verification keys are published here. Private signing keys are never published.
https://nbt.bot/.well-known/http-message-signatures-directoryKey rotation
During a planned key change, the directory may contain both the old and new public keys for a short overlap. Each published key has its own valid directory signature. We switch request signing only after the new key is live and checked.
What to check
- Check the User-Agent.
NBT’s production crawler uses the exact identity shown above. - Check Web Bot Auth.
Validate the signature against the public key directory instead of trusting the name alone. - Check what the request is doing.
It should relate to a customer-authorised test of that site, not unrelated internet scanning. - Send us the trace ID if something looks wrong.
IncludeNBT-Scan-Id, timestamp, hostname/path and any Cloudflare Ray ID.