A large number of business websites return an error to AI crawlers without their owners knowing. If ChatGPT, Perplexity or Claude cannot fetch your pages, your business cannot appear in their answers — regardless of how good your content or your Google rankings are. Here is how to check in about five minutes.
Why this happens by accident
Two things changed. First, Cloudflare — which sits in front of a very large share of websites — moved to blocking AI crawlers by default. Site owners who never opened a settings page were opted out automatically. Second, many security plugins and CDN “bot protection” presets treat AI crawlers as unwanted scrapers and block them alongside genuinely malicious traffic.
The result is a site that looks perfectly healthy to Google, ranks normally, and is completely invisible to AI assistants.
Check 1: your robots.txt
Visit yourdomain.com/robots.txt in a browser. Look for any of these names followed by Disallow: /:
GPTBotandOAI-SearchBot— ChatGPTPerplexityBot— PerplexityClaudeBot— ClaudeGoogle-Extended— GeminiCCBot— Common Crawl, which feeds many models
Also look for a line beginning Content-Signal:. If it contains ai-input=no, you have told AI systems not to use your content for generating answers.
Check 2: the actual server response
This is the check most people miss, and it is the one that matters. robots.txt is a request; your server or CDN can block crawlers outright regardless of what robots.txt says. A site can have a perfectly permissive robots.txt and still return a hard 403 error to every AI crawler.
Testing this requires sending a request that identifies itself as an AI crawler and observing the response code. If you are comfortable with a terminal, a single curl command with the crawler’s user agent will tell you. If you are not, this is exactly the kind of thing worth asking someone technical to run — it takes seconds and the answer is unambiguous.
A 200 response means you are visible. A 403 means you are blocked.
Check 3: are you blocked selectively?
It is common to find a site that allows Googlebot and Bingbot — so Google’s AI Overviews and Microsoft Copilot work fine — while blocking GPTBot, PerplexityBot and ClaudeBot. Because Google still works, everything looks normal from the inside. Meanwhile ChatGPT, which is where a very large share of AI queries actually happen, cannot see the business at all.
Check each crawler individually rather than assuming one result covers them all.
What to do if you are blocked
- Decide deliberately. Allowing AI crawlers means your content can be read, summarised and cited. For most businesses selling a service, that is straightforward marketing — being quoted is the goal. For businesses whose content is the product, the calculation is different. Make it a decision rather than an accident.
- Separate the two questions. “May AI systems read my site to answer questions about it?” and “May AI systems train on my content?” are different, and you can answer them differently. Allowing answer crawlers while declining training crawlers is a perfectly coherent position.
- Fix it in both places. Update robots.txt and your CDN or firewall rules. Fixing only robots.txt while the firewall still returns 403 changes nothing.
- Verify afterwards. Re-run the same server-response check and confirm you now get a 200.
Being readable is the floor, not the ceiling
Unblocking crawlers makes you eligible to be cited. It does not make you likely to be cited. That still depends on whether your pages answer real questions clearly, whether your structured data tells machines what you are, and whether other credible sources corroborate what you claim.
But none of that work returns anything while the door is shut. Check the door first.
Not sure where you stand? Ask us for an AEO readiness check — we test every major AI crawler against your live site and tell you exactly what each one sees.