Home / Guides / Is Cloudflare blocking ChatGPT from your website? How to check and fix it

AI search guide

Is Cloudflare blocking ChatGPT from your website? How to check and fix it

If your site runs through Cloudflare, your robots.txt may contain rules you never wrote. In my study of 1,323 new websites, 9% blocked AI crawlers, and the pattern matched Cloudflare's managed rules almost exactly. Here's how to check yours and decide what to allow.

By Cornelius Oni · Updated 2026-11-13

The short answer: open yoursite.com/robots.txt. If you see a block that starts with Cloudflare's "Content Signals Policy" comment, or Disallow: / under GPTBot, ClaudeBot or Google-Extended that you didn't add, Cloudflare is managing your AI crawler rules. Whether that's a problem depends on which bots are blocked.

Step 1: check your robots.txt (1 minute)

Visit https://yoursite.com/robots.txt in your browser. Look for lines like these:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Cloudflare's managed robots.txt adds a block like this when AI training is set to "disallow". According to Cloudflare's documentation, the list can include Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent, along with a Content-signal: search=yes, ai-train=no line.

Step 2: check Cloudflare's firewall settings (1 minute)

Robots.txt is only half the story. Cloudflare can also block AI bots at the firewall, before they ever read robots.txt. In your Cloudflare dashboard, open your domain, go to Security → Settings, filter by Bot traffic, and look at the AI crawler and agent access options and AI Crawl Control. If AI bots are set to "block", crawlers get an error page no matter what robots.txt says.

What each AI bot actually does

BotCompanyUsed forBlock it and you lose…
GPTBotOpenAITraining dataFuture model knowledge, not ChatGPT search
OAI-SearchBotOpenAIChatGPT search resultsBeing shown and linked in ChatGPT search
ChatGPT-UserOpenAIFetching a page a user asked aboutChatGPT reading your page live
ClaudeBotAnthropicTraining dataFuture model knowledge
PerplexityBotPerplexityPerplexity search resultsBeing cited in Perplexity
Google-ExtendedGoogleGemini and other AI usesNot AI Overviews: those use Googlebot

So the common Cloudflare setup, which blocks training bots, mostly protects your content from training while leaving AI search open. That's a reasonable choice. The real problem is when a search bot like OAI-SearchBot or PerplexityBot is also blocked, or when firewall rules block everything.

Step 3: decide what you want

If you want maximum AI visibility (most small businesses, shops and new products), allow everything. Turn off Cloudflare's managed AI rules and add this to robots.txt:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

If you want AI search but not training (publishers, paid content), keep GPTBot, ClaudeBot and Google-Extended blocked, but make sure OAI-SearchBot and PerplexityBot are allowed and not blocked at the firewall.

Step 4: confirm it worked

Reload your robots.txt to check the change is live. Then, after a few weeks, test what AI tools say about you with the free monthly ChatGPT test. Access is only the first step: the rest of the work is in getting your business recommended by ChatGPT.

Curious how common this is? See the full numbers in our study of 1,323 new websites.

Questions people ask

Does blocking GPTBot remove me from ChatGPT?

Not directly. GPTBot gathers training data. ChatGPT search uses OAI-SearchBot and live browsing uses ChatGPT-User. If you want to appear in ChatGPT search, make sure OAI-SearchBot isn't blocked.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says AI Overviews and AI Mode use the normal Search index built by Googlebot. Google-Extended controls use of your content in some of Google's other AI systems.

Is robots.txt enough to block AI bots?

Robots.txt is a request that well-behaved crawlers follow. Cloudflare's firewall-level bot blocking actually stops the request, which is why you should check both places.

Want ChatGPT to recommend your business?

I run AI visibility audits that test what ChatGPT, Perplexity and Google AI say about you and your competitors, then fix what's missing.