The short version: 16% of the new sites sent crawlers an almost empty page because all their content loads with JavaScript. 9% block at least one AI crawler in robots.txt, often without knowing it. Nearly half have no structured data. Only 41% of sites passed all six basic checks.
What I checked
For each site I fetched the homepage without running JavaScript, which is how GPTBot, ClaudeBot and PerplexityBot see pages, then checked six things:
- Readable HTML: does the page contain at least 250 characters of real text before JavaScript runs?
- robots.txt: does it block GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, anthropic-ai or OAI-SearchBot?
- Meta description and Open Graph image (the link preview on X, LinkedIn and Slack).
- sitemap.xml, either at the root or listed in robots.txt.
- Structured data (schema.org JSON-LD).
- llms.txt, the optional AI guide file.
The results
| Issue | Sites | Share |
|---|---|---|
| Empty page without JavaScript | 215 | 16.3% |
| robots.txt blocks at least one AI crawler | 121 | 9.1% |
| No meta description | 90 | 6.8% |
| No Open Graph image | 337 | 25.5% |
| No sitemap | 379 | 28.6% |
| No structured data | 632 | 47.8% |
| No llms.txt | 781 | 59.0% |
Only 541 of 1,323 sites (41%) passed the first six checks. I left llms.txt out of the pass mark because it's optional, and I explain why in does your business need an llms.txt file.
Finding 1: one in six sites is blank to AI crawlers
215 sites returned less than 250 characters of text before JavaScript ran. To a person in a browser they look fine. To ChatGPT's and Perplexity's crawlers, which mostly don't run JavaScript, the homepage is a title and an empty <div id="root">. They have nothing to quote, so they can't describe or recommend the product.
Google does render JavaScript, but later and less reliably than plain HTML. This was the single most damaging issue in the study, and it was most common on single-page apps built with React or Vue and no pre-rendering.
Fix: pre-render or server-render your public pages so the headline, description and key copy are in the HTML. Test with curl -s https://yoursite.com | grep -i "<h1". If your headline doesn't come back, crawlers can't see it either.
Finding 2: 9% block AI crawlers, mostly by accident
121 sites disallow at least one AI crawler. The most blocked were GPTBot (114 sites), ClaudeBot (110) and Google-Extended (86). That exact trio is what Cloudflare's managed robots.txt adds when AI training is set to "disallow", which suggests most founders didn't write these rules themselves. I explain how to check and change it in is Cloudflare blocking ChatGPT from your website.
It's worth being precise here. GPTBot, ClaudeBot and Google-Extended are mainly about AI training. ChatGPT search uses a separate bot, OAI-SearchBot, and Google's AI Overviews use normal Googlebot. Blocking training bots is a legitimate choice. But at least 29 sites (2.2%) also blocked a search crawler such as PerplexityBot or OAI-SearchBot, which directly removes them from those AI search tools.
Finding 3: the basics that help everyone are missing
- 48% have no structured data. A few lines of JSON-LD tell Google and AI tools what the product is, what it costs and who makes it.
- 29% have no sitemap. Search engines have to discover pages by following links.
- 26% have no link preview image. In launch week, when links get shared most, these show as bare URLs.
Finding 4: llms.txt is catching on with technical founders
41% of these sites already have an llms.txt file, far more than I see on small-business websites. It's a cheap signal, but it can't make up for the issues above: an llms.txt on a site that blocks crawlers or serves an empty page doesn't help much.
What to do with your own site
- Run the curl test above and make sure your real content is in the HTML.
- Open
yoursite.com/robots.txtand check which bots you block, and whether you meant to. - Add a meta description, an Open Graph image and a sitemap.
- Add structured data for your organisation and product.
- Then test what AI tools actually say about you with the free 15-minute ChatGPT test.
For the bigger picture, read how to get your business recommended by ChatGPT.
Method and limits
I only checked homepages and root files, so a site with a JavaScript homepage but well-rendered inner pages would still count as "blank" here. Robots.txt matching looked for a Disallow: / rule under each named user agent, and only the first three blocked bots per site were recorded, so the per-bot counts are minimums. The data covers launches on one community, so it skews toward software products and technical founders.
Questions people ask
How was this study done?
I collected every Show HN post with its own domain between 26 October and 13 November 2026, fetched each homepage the way a crawler does (no JavaScript), and checked robots.txt, sitemap.xml, llms.txt, the meta description, Open Graph image and structured data.
Does blocking GPTBot hide my site from ChatGPT search?
Not on its own. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and live browsing uses ChatGPT-User. But many sites block several bots at once, and an empty JavaScript-only page hurts every crawler.
Can I use this data?
Yes. Please link to this page as the source.
