Skip to content

Free tool

Robots.txt checker for AI crawlers

This robots.txt checker reads your site's robots.txt the way the crawlers do and shows, crawler by crawler, whether each AI company's bots may read your site: their search crawlers, their training crawlers, and the fetchers that act for a person.

Check your robots.txt

We read robots.txt by the rules in RFC 9309, as Google documents them: a crawler follows the group that names it, or the * group if none does, and the longest matching rule wins.

A public website, such as example.com. We read its robots.txt, llms.txt, and home page.

Enter a website above to see its checklist. We keep nothing we read, and results are a sample of what the site served at that moment.

Search crawlers and training crawlers are different

Most AI companies run more than one crawler. Search crawlers, such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot, fetch pages for the search that assistants run when they answer. Training crawlers, such as GPTBot and ClaudeBot, collect content that may be used to train future models.

Blocking a training crawler doesn't block the same company's search crawler: each has its own name in robots.txt. Google-Extended and Applebot-Extended crawl nothing themselves; they only say how Google and Apple may use pages their search crawlers already fetched.

Fetchers that act for a person, such as ChatGPT-User and Perplexity-User, open a page when someone asks the assistant to. OpenAI and Perplexity say these fetches may not follow robots.txt.

Questions

What is GPTBot?
GPTBot is OpenAI's training crawler. OpenAI says disallowing it signals that a site's content shouldn't be used to train its models. It's separate from OAI-SearchBot, which finds pages for ChatGPT's search answers.
What is ClaudeBot?
ClaudeBot is Anthropic's training crawler. Anthropic also runs Claude-SearchBot, for Claude's search results, and Claude-User, which opens a page when a Claude user's request needs it. Anthropic says all three follow robots.txt.
What is PerplexityBot?
PerplexityBot finds and links pages in Perplexity's search results. Perplexity says it isn't used to train models. Perplexity-User, which fetches a page when a user's question needs it, generally ignores robots.txt, Perplexity says.
Should I block AI training crawlers?
That's a business choice. Blocking training crawlers affects what future models may learn from your pages. It doesn't affect the search crawlers, so blocking GPTBot while allowing OAI-SearchBot is common.
Is this an AI bot checker for my server logs?
No. It reads your robots.txt rules, which say what crawlers may do. It can't see which bots actually visited. For that, look at your server or CDN logs, and verify bots by the IP ranges their vendors publish.
Where does the list of AI crawlers come from?
From each vendor's own documentation: OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, and Meta. We read them on 14 November 2026. Vendors add and rename crawlers, so we re-check the list.

Where to go next

Talk it through first

30 minutes, booked online. Free, with no obligation.

Book a call