ClaudeBot, GPTBot and PerplexityBot Explained
ClaudeBot is Anthropic's web crawler for collecting content that may be used to train its Claude models, GPTBot does the same job for OpenAI, and PerplexityBot crawls the web to show and link sites in Perplexity's search results. Each AI company also runs separate search and user-request bots, so the real choice isn't "block AI or not": it's whether to allow training crawlers, and B2B suppliers who want to be found should almost always allow the search ones.
This guide lists the main AI crawlers, what each one does according to its owner's documentation as of November 2026, and how to decide which to block.
Why AI crawlers come in three kinds
Most of the AI companies now separate their bots by purpose:
- Training crawlers collect public web pages that may be used to train future models.
- Search crawlers build the index the assistant searches when it looks things up for a user.
- User-request fetchers visit a specific page because a user asked the assistant about it, or the assistant decided to read it while answering.
Blocking a training crawler and blocking a search crawler have very different effects, so it's worth knowing which is which.
ClaudeBot and Anthropic's other crawlers
Anthropic's help article on how it crawls the web describes three bots:
- ClaudeBot collects web content that could contribute to training Anthropic's models. Anthropic says blocking it signals that your site's future content should be excluded from training data.
- Claude-SearchBot crawls to improve the quality of Claude's search results. Anthropic says blocking it may reduce your site's visibility in Claude's search answers.
- Claude-User fetches pages when people ask Claude questions. Blocking it stops Claude retrieving your pages for those requests, which Anthropic says may reduce your visibility.
Anthropic says its bots honour robots.txt, and supports the non-standard Crawl-delay rule if you want ClaudeBot to slow down. It also says blocking its IP addresses may not work reliably, because that stops it reading your robots.txt file, so use robots.txt instead. You need a robots.txt rule on each subdomain you want to opt out.
One newer detail: Anthropic says that from 30 October 2026 it has begun recognising a machine-readable "no AI training" preference signal, based on draft work in the IETF's aipref working group. Where your robots.txt names ClaudeBot directly, that rule takes priority.
GPTBot and OpenAI's other crawlers
OpenAI's overview of its crawlers lists:
- GPTBot, which crawls content that may be used to train OpenAI's foundation models. Disallowing it says your content shouldn't be used for training.
- OAI-SearchBot, which is used to show websites in ChatGPT's search features. OpenAI says sites that opt out of it won't be shown in ChatGPT search answers, though they can still appear as navigational links. It recommends allowing it, and allowing its published IP ranges, if you want to appear.
- ChatGPT-User, which visits pages when a ChatGPT user or a custom GPT asks for them. OpenAI says that because these visits are started by a user, robots.txt rules may not apply, and that it isn't used to decide what appears in search.
- OAI-AdsBot, which checks the landing pages of ChatGPT ads. It only visits pages submitted as ads. Our post on ChatGPT ads covers the ads themselves.
OpenAI says the settings are independent: you can allow OAI-SearchBot to appear in search while disallowing GPTBot to opt out of training. It also says changes to robots.txt can take about 24 hours to affect search results.
PerplexityBot and Perplexity-User
Perplexity's crawler documentation describes two agents:
- PerplexityBot surfaces and links websites in Perplexity's search results. Perplexity says it isn't used to crawl content for training AI foundation models, and recommends allowing it in robots.txt.
- Perplexity-User visits pages to answer a user's question. Perplexity says that because the user requested the fetch, it generally ignores robots.txt rules, and that it isn't used for training.
Google-Extended
Google doesn't run a separate AI crawler. Instead, Google-Extended is a robots.txt token that lets you say whether content Google crawls may be used to train future Gemini models and for grounding in some Gemini products. Google says it doesn't affect inclusion in Google Search and isn't a ranking signal. Google's guidance on AI features says that what appears in AI Overviews and AI Mode is controlled through the normal Search controls, such as Googlebot's robots.txt rules and the nosnippet and noindex directives.
Should you block AI crawlers?
For a B2B supplier, distributor or manufacturer that wants buyers to find it, our view is:
- Allow the search crawlers: OAI-SearchBot, Claude-SearchBot and PerplexityBot. Blocking them is the most direct way to drop out of those assistants' search answers.
- Allow the user-request fetchers: ChatGPT-User, Claude-User and Perplexity-User. These are the bots reading your page because a buyer asked about you.
- Decide on training crawlers as a business choice: ClaudeBot, GPTBot and Google-Extended. Some companies block training on principle or to protect content they sell. For a supplier whose pages are mostly marketing, being in training data means a model may know about you even when it answers without searching. Nobody outside these companies knows how much that matters for recommendations, so we wouldn't claim either way.
One more thing to know: allowing a crawler doesn't get you recommended. It only makes you eligible. Whether an assistant names you depends on what it finds about you, on your site and elsewhere. Our guide on how to get cited by ChatGPT covers that part.
An example robots.txt
This example allows search and user-request bots and opts out of training by OpenAI, Anthropic and Google. Adjust it to the choice you make.
# Opt out of AI training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Allow AI search crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Everyone else
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
If you'd rather allow everything, a single User-agent: * group with Allow: / does that.
How to tell a real AI crawler from a fake one
Anyone can put "GPTBot" or "ClaudeBot" in a user agent string, and some scrapers do. If you're making decisions from your logs, check that the visits come from the company's own addresses. OpenAI publishes the IP ranges for OAI-SearchBot, GPTBot and ChatGPT-User on its crawler overview. Anthropic's help article links to a list of the IP addresses its bots use, and says a crawler coming from one of those addresses is from Anthropic.
The reverse also applies. If your firewall blocks traffic by IP address and you haven't added these ranges, you may be turning away the real crawlers while your robots.txt says they're welcome. OpenAI specifically recommends allowing its published IP ranges if you want to appear in ChatGPT search.
Check you aren't blocking them by accident
Robots.txt isn't the only thing that can keep AI crawlers out. Firewalls, bot protection and content delivery networks can block them too. Cloudflare, for example, announced in July 2025 that new domains are asked at sign-up whether to allow AI crawlers. If someone chose for you, or a security setting was switched on later, your robots.txt may say one thing while your CDN does another.
Three quick checks:
- Read your robots.txt at
yourdomain.com/robots.txtand look for the names above, and for a blanketDisallow: /. - Check your CDN or firewall's bot settings for an "AI crawlers" or "AI bots" rule.
- Look in your server logs for the user agents above, to see whether they visit and get pages rather than errors.
Our free AI crawler checker reads your robots.txt and tells you which of these crawlers it allows or blocks. To see whether assistants actually name you once they can read your site, try the free AI visibility checker, and if you want help with what it finds, see our AI visibility service or book a free call.
For a related file that's often mentioned alongside robots.txt, read llms.txt: what it is and whether it works.