ClaudeBot: what it is and how to block or allow it

By Michael Nefedov, noetio · Updated 21 October 2026

ClaudeBot is the web crawler Anthropic uses to collect public web content that could contribute to training its Claude models. You control it with one rule in robots.txt.

Short answer

  • Anthropic runs three bots: ClaudeBot (training), Claude-User (user questions), Claude-SearchBot (search).
  • To keep your content out of training, add User-agent: ClaudeBot and Disallow: /.
  • Blocking Claude-User or Claude-SearchBot may reduce your visibility. Blocking ClaudeBot is only about training.

The three Anthropic bots

Anthropic runs three bots, and only ClaudeBot is about training. Source: its support page about crawling and blocking(dated 7 April 2026 on the page), read on 21 October 2026. This page covers Anthropic's crawler only, not other tools that share the name.

User-agentWhat Anthropic says it doesWhat blocking does
ClaudeBotCollects web content that could contribute to training of its modelsSignals that your site's future material should be excluded from training datasets
Claude-UserSupports Claude users: it fetches a page when a user asks something that needs itStops retrieval of your content for a user query. May reduce your visibility in user-directed search
Claude-SearchBotAnalyzes content to improve the relevance and accuracy of search resultsStops indexing for search. May reduce your visibility in search results
Table from Anthropic's support article: ClaudeBot collects web content that could contribute to model training; Claude-User fetches pages when users ask questions and disabling it may reduce visibility for user-directed web search; Claude-SearchBot improves search results and disabling it may reduce visibility in search results
The three bots in Anthropic's own words. Source: Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”, page dated 7 April 2026, captured 21 October 2026. Open full size.

The other two bots are about showing your pages in answers. OpenAI splits its bots the same way. See GPTBot vs OAI-SearchBot vs ChatGPT-User.

ClaudeBot robots.txt: copy and paste

Put these lines in the robots.txt file in the top-level folder of your site. Anthropic says to repeat them on every subdomain.

Block ClaudeBot, allow the rest

This keeps ClaudeBot out and leaves the two search and user-question bots open. Anthropic describes the effect of blocking each bot separately. Under RFC 9309 section 2.2.1 a crawler reads the group that names its own token, so these are three separate groups.

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

Allow all three

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

Block all three

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

Slow ClaudeBot down instead

Anthropic supports the non-standard Crawl-delay extension. This is its own example, from the same support article:

User-agent: ClaudeBot
Crawl-delay: 1

Block only one folder

User-agent: ClaudeBot
Disallow: /members/
Disallow: /drafts/

One trap: RFC 9309, section 2.2.1 says: “If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present.” So a crawler obeys the User-agent: * group only if no group matches its own token. If you add a ClaudeBot group, repeat any Disallow lines from your * group inside it.

RFC 9309 section 2.2.1: if no matching group exists crawlers must obey the group with a user-agent line with the * value; Figure 3 shows a * group with disallow /foo and /bar being the applicable group for ExampleBot when only a BazBot group also exists
The rule and its example. Source: RFC 9309, Robots Exclusion Protocol, section 2.2.1 (IETF, September 2022), captured 21 October 2026. Open full size.

Should you block ClaudeBot?

Block ClaudeBot if you want to opt out of training; Anthropic does not tie it to your visibility in answers. It does not say that allowing ClaudeBot gets you named in Claude answers, or that blocking it removes you. What it does say is narrow:

  • Blocking ClaudeBot is a request to keep your future content out of training sets.
  • Blocking Claude-User or Claude-SearchBot may lower how often Claude shows or uses your pages.
  • robots.txt is the method Anthropic recommends. IP blocking “may not work correctly or persistently guarantee an opt-out”.
Anthropic support article: Crawl-delay extension example for ClaudeBot, the Disallow: / example, a warning that blocking IP addresses may not work correctly or persistently guarantee an opt-out, and the line that a crawler with a source IP on the list comes from Anthropic
Anthropic's robots.txt examples and its warning about IP blocking. Source: same Anthropic support article, captured 21 October 2026. Open full size.
  • You sell something and want to appear in Claude answers: use the first snippet. It blocks training and leaves the two answer bots open.
  • Your content is the product: use the third snippet. It blocks all three, and Anthropic says that may reduce your visibility.

See the full picture in our AI crawlers guide.

Robots.txt works for future crawls. Anthropic's page talks about excluding future material; it does not describe removing data it already collected.

Common robots.txt mistakes

Most mistakes come from a typo, the wrong host or a hidden CMS rule.

  • Wrong spelling. The token is ClaudeBot. A typo means no rule applies and the bot sees an open site.
  • One file for many hosts. A robots.txt covers one host: RFC 9309, section 2.3 defines its address as scheme://authority/robots.txt, and Anthropic says to add the rule on every subdomain. www.example.com, example.com and blog.example.com each need their own file or a redirect that leads to one.
  • A broken * group. Once a group names ClaudeBot, the * group no longer applies to that bot (RFC 9309). Copy over any private folders you listed there.
  • A rule that is only in your CMS. Some site builders generate robots.txt for you. Open the live file at /robots.txt and read it. That is the only version a bot sees.
  • Expecting instant effect. Anthropic gives no timing for a robots.txt change, so check your logs after a few days.

How to see if ClaudeBot visits your site

Search your server access log for the three user-agent names. Look for ClaudeBot, Claude-User and Claude-SearchBot. The counting command is in the AI crawlers guide; add these three names to it.

A name in a user-agent string is easy to fake. Anthropic says that if a crawler has a source IP on its list, the crawler comes from Anthropic. The list is a JSON file at claude.com/crawling/bots.json. On 21 October 2026 it returned HTTP 200 and showed a creation time of 2026-10-20 and a list of 14 IPv4 prefixes. The creation time changes, so read the file itself for the current list. It is linked from Anthropic's page.

What we ran: on 21 October 2026, curl -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" against getcompassseo.com returned HTTP 200, and so did plain curl.

Checklist

Six steps, in this order.

  1. 1Decide: training yes or no, answers yes or no.
  2. 2Paste the matching snippet into robots.txt.
  3. 3Repeat it on each subdomain.
  4. 4Open yoursite.com/robots.txt and read it back.
  5. 5Search your logs to see which of the three bots visit (command above).
  6. 6Check that your CDN or firewall does not block the bots you allowed.

Related: GPTBot, OAI-SearchBot and ChatGPT-User, the AI crawler table, and whether llms.txt matters.

Sources

All read on 21 October 2026.

  1. Anthropic Privacy Center, Does Anthropic crawl data from the web, and how can site owners block the crawler?, privacy.claude.com, page dated 7 April 2026. Used for the three bots, what blocking does, robots.txt examples, Crawl-delay, the IP-blocking warning.
  2. Anthropic, claude.com/crawling/bots.json, the IP list linked from the article. Fetched 21 October 2026: HTTP 200, creation time 2026-10-20, 14 IPv4 prefixes.
  3. IETF, RFC 9309, Robots Exclusion Protocol (September 2022), sections 2.2.1 and 2.3. Used for group matching and the per-host file address.
  4. Our own curl tests of getcompassseo.com, run 21 October 2026, described in the section above.

Questions

No. ClaudeBot is for training. Claude-User fetches a page for a user question. Claude-SearchBot is for search. The table above has the detail.

Anthropic does not say so. For ClaudeBot it only mentions training. The visibility warnings are for Claude-User and Claude-SearchBot.

Anthropic warns that blocking by IP "may not work correctly or persistently guarantee an opt-out". It recommends robots.txt. Anthropic links an IP list (a JSON file) from the same page, so you can check whether a request came from an address on it.

Anthropic says it supports the non-standard Crawl-delay extension to robots.txt. Its example is Crawl-delay: 1 for ClaudeBot. Its page does not say what the number means or how fast a change takes effect.

Anthropic says so itself: "Anthropic's Bots respect 'do not crawl' signals by honoring industry standard directives in robots.txt."

More guides

Turn your next topic into a useful page

Compass reads your website, helps you plan relevant topics and creates sourced article drafts. Review and approve the work, then send it to a connected CMS or export it.

Written by Michael Nefedov, founder of noetio. Updated 21 October 2026.