The Complete List of AI Crawlers (and Whether to Block or Allow Each One)
Every AI crawler that visits your site does one of three jobs. Here's what each one is — and whether blocking it is a privacy decision or a visibility mistake.
On this page9 sections
If you've ever opened your website's server logs, you've probably seen names like GPTBot, ClaudeBot, or PerplexityBot showing up again and again. These are AI crawlers — bots that visit your site on behalf of AI companies. Some are new. All of them matter more than most site owners realize.
Here's the problem: most guides just list these bots without explaining what each one actually does. That leaves you guessing whether blocking one helps you or hurts you. This guide fixes that. For every crawler below, you'll see what it does, and whether blocking it helps or hurts your visibility in AI answers.
First, the one thing that matters more than any single bot name
Every AI crawler does one of three jobs. Once you know the job, deciding whether to block it becomes simple — you don't need to memorize every name.
1. Training crawlers. These collect pages to train future AI models. Blocking them is a choice about training consent. It has almost no effect on whether ChatGPT or Claude mentions your business today — that's a separate job, done by a different crawler.
2. Search/retrieval crawlers. These build the live index an AI assistant actually searches when answering a question. This is your real citation pipeline. Block one of these, and your pages simply can't be pulled into an answer — you become invisible to that specific AI tool.
3. User-triggered fetchers. These visit a page because a real person, inside an AI chat, asked the assistant to look at it right now. Closest thing to a real visitor.
If you only remember one thing from this article, remember this: blocking a training crawler is a privacy decision. Blocking a search/retrieval crawler is a visibility decision. Mixing them up is the single most common reason a business quietly disappears from AI answers without anyone noticing why.
OpenAI (ChatGPT)
Crawler | Job | Block or allow? |
|---|---|---|
| Training | Your call — no visibility cost either way |
| Search indexing (feeds ChatGPT Search) | Allow — blocking removes you from ChatGPT's live answers |
| User-triggered fetch | Allow — this fires when a real user asks ChatGPT to check your page |
Anthropic (Claude)
Crawler | Job | Block or allow? |
|---|---|---|
| Training | Your call |
| Search indexing | Allow |
| User-triggered fetch | Allow |
Perplexity
Crawler | Job | Block or allow? |
|---|---|---|
| Search indexing | Allow |
| User-triggered fetch | Allow |
Crawler | Job | Block or allow? |
|---|---|---|
| Classic search (also feeds AI Overviews and AI Mode) | Never block — this removes you from Google Search entirely, not just AI features |
| Not a separate crawler — a robots.txt setting that controls whether Gemini/Vertex AI can use your content for training | Your call |
The trap here is real: you cannot block Google's AI Overviews without blocking Google Search itself, because both run through the same Googlebot. Google-Extended only opts you out of training data — it does nothing to keep you out of AI Overviews.
Others worth knowing
Crawler | Job | Block or allow? |
|---|---|---|
| Training (Apple Intelligence) | Your call |
| Training/search (Amazon's AI products) | Your call |
| Training | Your call |
| Training (ByteDance) | Your call — mixed compliance record reported, worth watching |
| Training (Common Crawl — feeds datasets used by GPT, Llama, Mistral, and most major models) | Your call — this one has an outsized effect since so many models train on its data |
A ready-to-use robots.txt template
# Allows every retrieval/search crawler.
# Permits training crawlers by default — uncomment the Disallow lines
# under each one if you want to opt out of training specifically.
# OpenAI
User-agent: GPTBot
Allow: /
# Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
# Anthropic
User-agent: ClaudeBot
Allow: /
# Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
# Perplexity
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
# Google (training opt-out only — Googlebot itself must stay allowed)
User-agent: Google-Extended
Allow: /
# Disallow: /An honest opinion
Most site owners who block AI crawlers do it out of a general unease about AI, without realizing they're blocking the exact crawler that would have gotten them cited in a ChatGPT answer. I understand the instinct — but if being found by AI matters to your business at all, blocking OAI-SearchBot, Claude-SearchBot, or PerplexityBot specifically is almost always working against your own interest. Training crawlers are a genuinely separate decision, and a reasonable business can go either way on those.
Check your own site in two minutes
Open your site's robots.txt file (yoursite.com/robots.txt) and look for any of the names above next to a Disallow: / line. If you find one on a search/retrieval crawler, that's likely why AI tools aren't mentioning you.
Here's what ours looks like, as a real example:
User-Agent: OAI-SearchBot
Allow: /
User-Agent: ChatGPT-User
Allow: /
User-Agent: Claude-SearchBot
Allow: /
User-Agent: Claude-User
Allow: /
User-Agent: PerplexityBot
Allow: /
User-Agent: Google-Extended
Allow: /
User-Agent: GPTBot
Allow: /
User-Agent: ClaudeBot
Allow: /
User-Agent: Googlebot
Allow: /
User-Agent: Bingbot
Allow: /
User-Agent: *
Allow: /
Disallow: /admin
Disallow: /api
Sitemap: https://alrebro.com/sitemap.xmlWe built a free tool that checks this automatically, including the newer crawlers most tools don't check: Robots.txt Tester
We've also written about a real case where this exact thing was happening silently on a client's site: Is Your Site Blocking AI Crawlers?
Founder, Alrebro
Fazal Ur Rehman is the founder of Alrebro and an AI SEO strategist focused on search visibility, technical SEO, and digital growth. He shares practical insights, industry trends, and actionable strategies to help businesses succeed online.
More from Fazal Ur RehmanRelated articles
How Does ChatGPT Decide Which Businesses to Recommend?
For anything current or local, ChatGPT almost never answers from memory — it runs a live search, pulls a handful of pages, and pieces together an answer from what those pages say.
Schema Markup for AI Search: What It Actually Does (2026 Guide)
Dozens of guides claim schema gives a "50% lift" in AI visibility with no source behind it. Here's what structured data genuinely does for AI citation, the six schema types worth your time, and how to add them free in about 20 minutes.
GEO vs SEO: The Difference, and Which One Your Business Actually Needs
GEO and SEO share about 70% of the same work, and most guides get the other 30% wrong. The real differences, the confusion worth clearing up, and a straight answer on where your budget should go — by business type.
Get new articles in your inbox
No spam — just SEO strategy and updates, occasionally.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Find out what's actually wrong
39 checks, your off-page signals, and a plan you can act on. Free, about a minute, no account needed.