# What we learned from 67 agent readiness audits > We checked 67 websites for agent readiness: who has an llms.txt, who blocks AI crawlers, why Shopify stores do well, and how much traffic ChatGPT sends. Source: https://pixelhop.io/writing/agent-readiness-study-2026/ --- [Back to writing](https://pixelhop.io/writing/) # What we learned from 67 agent readiness audits. Jozef / 09 Nov 2026 [AI](https://pixelhop.io/writing/tags/ai/) [AI agents](https://pixelhop.io/writing/tags/ai-agents/) [Business](https://pixelhop.io/writing/tags/business/) We checked 67 websites for agent readiness: who has an llms.txt, who blocks AI crawlers, why Shopify stores do well, and how much traffic ChatGPT sends. Agent readiness is how easily AI agents and crawlers can find, read and use a website. In July we launched a free [Agent Readiness Checker](https://chatthing.ai/tools/agent-readiness-checker) for Chat Thing that scores it from a URL. We wanted to see what the results look like across everyone who's used it, so we pulled the reports and went through them. We kept 67 websites: 28 marketing sites, 22 SaaS products, 16 online shops and one documentation site. Their owners scanned them between August and November. On 9 November we checked all 67 sites again, so the numbers below show where they are now. These are sites whose owners went looking for an agent readiness checker, so they're probably more clued up than most. It's a small sample and we're not claiming it represents the web. The short version: - Most sites have the basics (robots.txt, a sitemap, Open Graph tags), and 79% have an llms.txt, often generated for them. - All 11 Shopify stores had llms.txt, UCP and OAuth files set up by Shopify, but none of them named an AI crawler in robots.txt. - Only 5 of the 67 sites block any AI crawlers. 45 don't mention them at all. - On our own sites, ChatGPT sends far more visitors than any other AI tool, and on Chat Thing those visitors signed up four times as often as visitors from Google. ## Most sites have the basics 93% have a robots.txt, 87% point to their sitemap from it, and 79% have canonical and Open Graph tags set up properly. Only 4 of the 63 sites we could render depend on JavaScript for their content. That matters because GPTBot, ClaudeBot and PerplexityBot don't run JavaScript. The worst one had 2 words in its raw HTML and 1,410 once the page loaded, so those crawlers see a page with nothing on it. ## Most llms.txt files are auto-generated 53 of the 67 sites (79%) have an llms.txt. We expected far fewer, until we looked at where they came from. 11 are Shopify stores, which all have almost the same file because Shopify now creates one for every store. At least 3 more were made by WordPress SEO plugins (Yoast, Rank Math and All in One SEO), and a few others look generated too. That's not a bad thing, but it means having an llms.txt doesn't tell you much any more. It's become a default, like a sitemap. ## Shopify stores get llms.txt and UCP for free All 11 Shopify stores have three things most other sites don't: - an llms.txt written as instructions for AI agents on how to browse and buy from the store - a UCP (Universal Commerce Protocol) file, which tells an agent how to place an order - OAuth metadata, which tells an agent how to log in on someone's behalf Shopify set all of this up. On paper, a small clothing shop on Shopify is better set up for an AI agent to buy from than most of the SaaS products in our sample are for an agent to use. Only 4 of the 22 SaaS sites have an MCP server card, an A2A agent card or an API catalog. The shop owners haven't done much beyond what Shopify gave them, though. Only 2 of the 16 online shops mention an AI crawler in their robots.txt, only 2 serve markdown to agents, and only 2 say how AI is allowed to use their content. If your shop is on Shopify, it's worth opening yoursite.com/llms.txt to see what it says about you. ## Very few sites block AI crawlers, and most don't mention them We thought we'd find a lot of sites blocking AI crawlers. One site blocks all crawlers. Four block at least one AI crawler by name, mostly Amazonbot and training crawlers like CCBot and Bytespider. Only one blocks GPTBot and only one blocks ClaudeBot. What we found instead was that 45 of the 67 sites (67%) don't name a single AI crawler in their robots.txt, so every AI bot gets the default rules. The same number don't use [Content Signals](https://contentsignals.org/) to say whether AI can train on, search or quote their content. Two of the 18 sites in our most recent scans allowed PerplexityBot in robots.txt but blocked it at the firewall. That's usually a CDN or firewall bot setting rather than a deliberate choice. ## Very few sites support the newer agent protocols | | Sites | | ----------------------------------------------------- | ------ | | robots.txt, sitemap, canonical and OG tags | 79-93% | | llms.txt | 79% | | Names AI crawlers / Content Signals / serves markdown | 31-33% | | UCP (mostly Shopify) | 21% | | API catalog, A2A agent card or MCP server card | 16% | The 11 sites with an API catalog, A2A card or MCP server card are mostly small independent sites and developer-tool companies. The bigger brands in our sample, a couple of them household names, don't have any of them yet. ## Some sites changed after scanning We know what each site had when it was first scanned, so we can compare that with what it has now. 7 of the 18 sites that didn't have an llms.txt have added one since, including a global car brand. 12 sites were scanned more than once, most of them again on the same day. One was scanned six times in six days, and between scans its owner added markdown support and fixed its structured data. ## ChatGPT traffic on our own sites The audits show what sites have set up. They don't show whether any of it brings visitors, so we also looked at the analytics (Plausible) for two sites we run or look after: [Chat Thing](https://pixelhop.io/our-work/chat-thing/) and [Rye & Beyond](https://pixelhop.io/our-work/rye-and-beyond/), a holiday cottage company in Rye that we work with. ChatGPT is the only AI tool sending meaningful traffic. Across the two sites over the last 12 months, it sent about 840 visitors. Perplexity sent 34, Claude 20, Gemini 13 and Copilot 4. Here's October this year against October last year: | ChatGPT visitors | October 2025 | October 2026 | Compared with Google, Oct 2025 | Compared with Google, Oct 2026 | | ---------------- | ------------ | ------------ | ------------------------------ | ------------------------------ | | Rye & Beyond | 18 | 104 | 5% as many | 29% as many | | Chat Thing | 68 | 102 | 21% as many | 19% as many | ChatGPT sent Rye & Beyond almost six times as many visitors as a year ago, and it now sends nearly a third as many as Google. Chat Thing's ChatGPT traffic is up by half, but its Google traffic grew as well, so ChatGPT's share is about the same as it was last October. On Rye & Beyond, ChatGPT is now the third biggest source of visitors after Google and direct. Most of the ChatGPT visitors who don't arrive on the homepage land on a specific cottage's page, so ChatGPT is recommending individual properties rather than the business in general. On Chat Thing, the ones who don't land on the homepage mostly arrive on docs pages for specific integrations (Discord, Telegram, WhatsApp) and on our free tools. The visitors behave differently too. From June to early November, 44 of the 231 people who came to Chat Thing from ChatGPT signed up (19%), compared with 116 of 2,591 from Google (4.5%). ChatGPT visitors also bounced less (21% against 62%) and stayed longer (about 5 minutes against under 2). Rye & Beyond is more mixed. ChatGPT visitors clicked "Book now" a bit more often than Google visitors (4.3% against 3.0%), but none of them completed a booking in that period, while 21 Google visitors did. The part we didn't expect is that Rye & Beyond, the site getting the most from ChatGPT, has no llms.txt, doesn't serve markdown and doesn't mention AI crawlers in its robots.txt. Chat Thing has an llms.txt and serves markdown. Two sites can't tell you what causes AI traffic, but they do show that none of these files is required to get it. What Rye & Beyond does have is a clear page for every cottage, and those pages are what ChatGPT links to. ## What we'd do 1. Look at your /llms.txt, especially if you're on Shopify or use a WordPress SEO plugin, so you know what it says. If you don't have one, our free [llms.txt generator](https://chatthing.ai/tools/llms-txt-generator) builds one from your sitemap. 2. Add rules for AI crawlers to robots.txt. Training crawlers (GPTBot, CCBot, Google-Extended) and search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) do different jobs, so decide on each. Add Content Signals at the same time. 3. Check your firewall or CDN isn't blocking crawlers that robots.txt allows. 4. Make sure your main content is in the HTML without JavaScript. 5. Give each thing you offer its own clear page (each product, each property, each integration). On our own sites, those are the pages ChatGPT sends people to. 6. If agents could do something useful with your product, look at MCP, A2A or an API catalog. Only 11 of these 67 sites have any of them so far. You can check your own site with the free [Agent Readiness Checker](https://chatthing.ai/tools/agent-readiness-checker). If you want help with any of this, [get in touch](https://pixelhop.io/services/ai-integration). Next time we'd like to scan a fixed set of sites and re-check the same set every quarter, so we can track changes properly. If there's a sector you'd like us to include, let us know. 🐰 ### 0 Responses [Web Mentions?](#) Want to respond? [Tweet this post!](https://twitter.com/intent/tweet?via=pixelhopio&text=What we learned from 67 agent readiness audits&url=https://pixelhop.io/writing/agent-readiness-study-2026/) ### Tags [WebAR](https://pixelhop.io/writing/tags/webar/) [Hop Tips](https://pixelhop.io/writing/tags/hop-tips/) [Nuxt](https://pixelhop.io/writing/tags/nuxt/) [AI](https://pixelhop.io/writing/tags/ai/) [Typescript](https://pixelhop.io/writing/tags/typescript/) [Snipcart](https://pixelhop.io/writing/tags/snipcart/) [Be the Forest](https://pixelhop.io/writing/tags/be-the-forest/) [Gridsome](https://pixelhop.io/writing/tags/gridsome/) [TensorFlowJs](https://pixelhop.io/writing/tags/tensorflowjs/) [JAMstack](https://pixelhop.io/writing/tags/jamstack/) [Kubernetes](https://pixelhop.io/writing/tags/kubernetes/) [News](https://pixelhop.io/writing/tags/news/) [Side projects](https://pixelhop.io/writing/tags/side-projects/) [stack.earth](https://pixelhop.io/writing/tags/stack-earth/) [Strapi](https://pixelhop.io/writing/tags/strapi/) [Node.js](https://pixelhop.io/writing/tags/node-js/) [Tailwind UI](https://pixelhop.io/writing/tags/tailwind-ui/) [Vue.js](https://pixelhop.io/writing/tags/vue-js/) [Three.js](https://pixelhop.io/writing/tags/three-js/) [VuePress](https://pixelhop.io/writing/tags/vuepress/) [Spicy Tips](https://pixelhop.io/writing/tags/spicy-tips/) [Design](https://pixelhop.io/writing/tags/design/) [Business](https://pixelhop.io/writing/tags/business/) [Development](https://pixelhop.io/writing/tags/development/) [AI](https://pixelhop.io/writing/tags/ai/) ### Subscribe. Like what you’re reading? Sign up to receive our updates straight to your inbox. No spam, unsubscribe anytime! ### Related posts. [All writing](https://pixelhop.io/writing/) [AI](https://pixelhop.io/writing/tags/ai/) [AI agents](https://pixelhop.io/writing/tags/ai-agents/) [AI product development](https://pixelhop.io/writing/tags/ai-product-development/) [Development](https://pixelhop.io/writing/tags/development/) 16 Oct 2026 #### Our agentic coding workflow: how we make coding agents check their own work How visual verification works in our agentic coding workflow. Our coding agents screenshot and record every UI change, check the captures themselves and put the evidence in the pull request. [Read more](https://pixelhop.io/writing/agentic-coding-workflow-visual-verification/) [AI](https://pixelhop.io/writing/tags/ai/) [AI product development](https://pixelhop.io/writing/tags/ai-product-development/) [AI agents](https://pixelhop.io/writing/tags/ai-agents/) [Business](https://pixelhop.io/writing/tags/business/) 16 Oct 2026 #### How much does AI development cost? Our real prices for 2026 What it really costs to build and run an AI product, agent or automation in 2026, using our actual prices, real projects and today's model pricing. [Read more](https://pixelhop.io/writing/how-much-does-ai-development-cost/) [AI](https://pixelhop.io/writing/tags/ai/) [AI agents](https://pixelhop.io/writing/tags/ai-agents/) [AI Automations](https://pixelhop.io/writing/tags/ai-automations/) 09 Sep 2026 #### We built free AI tools for SEO, then realised they could generate leads too How a free SEO checker started surfacing potential leads, and the human-reviewed Glitch workflow we built around it. [Read more](https://pixelhop.io/writing/free-ai-tools-for-seo-then-realised-they-could-generate-leads-too/)