AI agent directory · Anthropic
ClaudeBot
ClaudeBot is Anthropic's crawler for collecting pages that may be used to train AI models. Here's how to recognize it and check a request really is it.
At a glance
ClaudeBot facts
- Operator: Anthropic
- Type: Crawler
- Why it fetches: Collects training data
- User agent contains: `ClaudeBot`
- Verified by: published IP ranges
- Operator's documentation: support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
Is it really?
How to tell a real ClaudeBot from an impostor
Anyone can put "ClaudeBot" in a user agent. What can't be faked is below.
- Check the address. Anthropic publishes the IP ranges ClaudeBot uses: claude.com/crawling/bots.json. A request whose user agent says ClaudeBot from an address outside them is an impostor.
In Punchline
How Punchline counts ClaudeBot
Punchline labels a request that names ClaudeBot with how sure it is: verified or declared when the evidence checks out, a lower tier when it can't be checked, and impostor when the evidence contradicts it. Each visit is counted under Anthropic, with its purpose, so you can see whether ClaudeBot is reading your site and which pages it reads. How it works · Known agents