Anthropic updated its crawler documentation this week, clarifying how its Claude bots access websites and how you can block them.

  • Anthropic’s document explains what each bot does, how it affects AI training and search visibility, and how to opt out through robots.txt.

Why we care. If you publish or own content, you want control over how AI systems use it. Anthropic separates training crawlers, user-triggered fetches, and search indexing. Blocking one bot doesn’t block the others. Each choice carries different visibility and training trade-offs.

The robots. Anthropic uses three separate user agents:

  • ClaudeBot collects public web content that may be used to train and improve Anthropic’s generative AI models. If you block ClaudeBot in robots.txt, Anthropic said it will exclude your site’s future content from AI training datasets.
  • Claude-User retrieves content when a user asks Claude a question that requires access to a webpage. If you block Claude-User, Anthropic can’t fetch your pages in response to user queries. The company says this may reduce your visibility in user-directed search responses.
  • Claude-SearchBot crawls content to improve the quality and relevance of Claude’s search results. If you block Claude-SearchBot, Anthropic won’t index your content for search optimization, which may reduce visibility and accuracy in Claude-powered search answers.

How to block them. The bots respect standard robots.txt directives, including “Disallow” rules and the non-standard “Crawl-delay” extension, Anthropic said. To block a bot across your entire site:

User-agent: ClaudeBot
Disallow: /

  • You must add directives for each bot and each subdomain you want to restrict.
  • IP blocking may not work reliably because its bots use public cloud provider IP addresses, Anthropic said. Blocking those ranges could prevent the bot from accessing robots.txt. The company doesn’t publish IP ranges.

The document. Does Anthropic crawl data from the web, and how can site owners block the crawler?

Search Engine Land is owned by Semrush. We remain committed to providing high-quality coverage of marketing topics. Unless otherwise noted, this page’s content was written by either an employee or a paid contractor of Semrush Inc.

Danny GoodwinDanny Goodwin

Danny Goodwin is Editorial Director of Search Engine Land & Search Marketing Expo – SMX. He joined Search Engine Land in 2022 as Senior Editor. In addition to reporting on the latest search marketing news, he manages Search Engine Land’s SME (Subject Matter Expert) program. He also helps program U.S. SMX events.

Goodwin has been editing and writing about the latest developments and trends in search and digital marketing since 2007. He previously was Executive Editor of Search Engine Journal (from 2017 to 2022), managing editor of Momentology (from 2014-2016) and editor of Search Engine Watch (from 2007 to 2014). He has spoken at many major search conferences and virtual events, and has been sourced for his expertise by a wide range of publications and podcasts.