The short answer: open yoursite.com/robots.txt. If you see a block that starts with Cloudflare's "Content Signals Policy" comment, or Disallow: / under GPTBot, ClaudeBot or Google-Extended that you didn't add, Cloudflare is managing your AI crawler rules. Whether that's a problem depends on which bots are blocked.
Step 1: check your robots.txt (1 minute)
Visit https://yoursite.com/robots.txt in your browser. Look for lines like these:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Cloudflare's managed robots.txt adds a block like this when AI training is set to "disallow". According to Cloudflare's documentation, the list can include Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent, along with a Content-signal: search=yes, ai-train=no line.
Step 2: check Cloudflare's firewall settings (1 minute)
Robots.txt is only half the story. Cloudflare can also block AI bots at the firewall, before they ever read robots.txt. In your Cloudflare dashboard, open your domain, go to Security → Settings, filter by Bot traffic, and look at the AI crawler and agent access options and AI Crawl Control. If AI bots are set to "block", crawlers get an error page no matter what robots.txt says.
What each AI bot actually does
| Bot | Company | Used for | Block it and you lose… |
|---|---|---|---|
| GPTBot | OpenAI | Training data | Future model knowledge, not ChatGPT search |
| OAI-SearchBot | OpenAI | ChatGPT search results | Being shown and linked in ChatGPT search |
| ChatGPT-User | OpenAI | Fetching a page a user asked about | ChatGPT reading your page live |
| ClaudeBot | Anthropic | Training data | Future model knowledge |
| PerplexityBot | Perplexity | Perplexity search results | Being cited in Perplexity |
| Google-Extended | Gemini and other AI uses | Not AI Overviews: those use Googlebot |
So the common Cloudflare setup, which blocks training bots, mostly protects your content from training while leaving AI search open. That's a reasonable choice. The real problem is when a search bot like OAI-SearchBot or PerplexityBot is also blocked, or when firewall rules block everything.
Step 3: decide what you want
If you want maximum AI visibility (most small businesses, shops and new products), allow everything. Turn off Cloudflare's managed AI rules and add this to robots.txt:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
If you want AI search but not training (publishers, paid content), keep GPTBot, ClaudeBot and Google-Extended blocked, but make sure OAI-SearchBot and PerplexityBot are allowed and not blocked at the firewall.
Step 4: confirm it worked
Reload your robots.txt to check the change is live. Then, after a few weeks, test what AI tools say about you with the free monthly ChatGPT test. Access is only the first step: the rest of the work is in getting your business recommended by ChatGPT.
Curious how common this is? See the full numbers in our study of 1,323 new websites.
Questions people ask
Does blocking GPTBot remove me from ChatGPT?
Not directly. GPTBot gathers training data. ChatGPT search uses OAI-SearchBot and live browsing uses ChatGPT-User. If you want to appear in ChatGPT search, make sure OAI-SearchBot isn't blocked.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says AI Overviews and AI Mode use the normal Search index built by Googlebot. Google-Extended controls use of your content in some of Google's other AI systems.
Is robots.txt enough to block AI bots?
Robots.txt is a request that well-behaved crawlers follow. Cloudflare's firewall-level bot blocking actually stops the request, which is why you should check both places.
