Should you block AI crawlers?
If you want to be discoverable in AI answers, do not block AI search crawlers. Whether to allow training crawlers is a business choice, not an SEO requirement. Control both with named user-agents in robots.txt.
What is the difference between search and training bots?
Practice AI companies publish separate crawlers. For example OpenAI documents GPTBot for training and OAI-SearchBot for search. Anthropic documents ClaudeBot and Claude-SearchBot. Perplexity documents PerplexityBot. Blocking one does not block the others. Check each vendor’s current documentation before editing rules, because names change.
Does blocking Google-Extended hide me from AI Overviews?
Proven No. Google states that Google-Extended does not affect Google Search, including AI Overviews and AI Mode. To limit what appears in Search results, Google points to controls such as nosnippet, max-snippet and noindex, which also reduce visibility.
What do other sites do?
On 22 October 2026 we read the live robots.txt of 15 sites, including SEO publishers and tool vendors and AI and developer companies. Of the 14 we could read, none blocked AI crawlers site-wide. This shows common behavior, not what is best for you.
Is robots.txt enough?
Practice robots.txt is a voluntary convention. Well-behaved crawlers follow it, but it is not a security control. Also check that your CDN or firewall does not block crawlers you intend to allow.
What does AEOing do?
We allow both search and training crawlers, because our goal is to be read and cited. You can view our robots.txt.