AI Crawler Statistics (2026)

A data roundup of who crawls the web for AI in 2026, how much traffic those crawlers send back, and how site owners control them with robots.txt and llms.txt.

Cloudflare says that in 2026, for the first time, more than half of Internet traffic was not human, and 52% of the crawler requests it saw in June 2026 were for AI training, up from 22% in Spring 2025. AI crawlers still send little back: Cloudflare Radar measured Anthropic’s crawl-to-refer ratio as high as 500,000:1 in 2025, while Google sends about 88% of referral traffic. On the site side, the Web Almanac finds GPTBot named in 4.5% of desktop robots.txt files and llms.txt on just 2.13% of sites. These are the numbers behind picking SEO tools and GEO tools that track AI search.

Key takeaways

• More than half of Internet traffic was not human in 2026 (Cloudflare)
• 52% of crawler requests were for AI training in June 2026, up from 22% in Spring 2025 (Cloudflare)
• Daily AI agent requests grew by more than 1,700% in a year (Cloudflare)
• 17% of sites on Cloudflare block AI training crawlers; fewer than 1% block search crawlers (Cloudflare)
• Google still sends about 88% of referral traffic (Cloudflare)
• Googlebot is 39% of all AI and search crawler traffic (Cloudflare)
• AI bots averaged 4.2% of HTML requests in 2025; Googlebot alone was 4.5% (Cloudflare Radar 2025)
• Anthropic’s crawl-to-refer ratio reached as much as 500,000:1 in 2025 (Cloudflare Radar 2025)
• GPTBot is named in 4.5% of desktop robots.txt files, up about 55% (Web Almanac SEO)
• Only 2.13% of desktop sites have a valid llms.txt file (Web Almanac SEO)

How much of the web’s traffic is bots and AI agents?

1. More than half of Internet traffic was not human in 2026, a first, per Cloudflare (Cloudflare).
2. Daily requests from AI agents on Cloudflare’s network grew by more than 1,700% over the past year (Cloudflare).
3. Cloudflare traffic rose from an average of 63 million HTTP requests a second at the end of 2024 to 115 million, with peaks above 150 million (Cloudflare).
4. Cloudflare sees more than 500 billion verified bot requests each week (Cloudflare).
5. More than 20% of the web sits behind Cloudflare’s network, which is why its bot data is a useful proxy (Cloudflare).
6. 32% of traffic across Cloudflare’s network is automated, including search crawlers and AI assistants (Cloudflare).
7. AI crawlers account for 80% of the self-identified AI bot traffic Cloudflare sees (Cloudflare).
8. AI bots averaged 4.2% of HTML requests in 2025, ranging from 2.4% in early April to 6.4% in late June (Cloudflare Radar 2025).
9. As of December 2, 2025, humans generated 47% of HTML requests and non-AI bots 44% (Cloudflare Radar 2025).
10. 40% of global bot traffic came from the United States (Cloudflare Radar 2025).

What it means: bots are now the majority audience for many sites. If your analytics only count humans, you are missing the readers that decide whether AI answers mention you.

Why are AI crawlers crawling?

11. 52% of crawler requests were for AI training as of June 2026, up from 22% in Spring 2025 (Cloudflare).
12. Mixed-use crawlers (search, agent use and training blended) represent over 36% of crawler activity (Cloudflare).
13. Over the 12 months to mid-2025, 80% of AI crawling was for training, 18% for search and 2% for user actions (Cloudflare).
14. Training’s share of AI crawling rose from 72% in July 2024 to 79% in July 2025, while search fell from 26% to 17% (Cloudflare).
15. AI “user action” crawling (fetching pages when a user asks) increased by over 15x in 2025 (Cloudflare Radar 2025).
16. ChatGPT-User peak request volumes were as much as 16x higher than at the start of 2025 (Cloudflare Radar 2025).
17. ChatGPT-User requests surged 2,825% from May 2024 to May 2025, reaching a 1.3% share (Cloudflare).

Which AI crawlers crawl the most?

18. Googlebot accounts for 39% of all AI and search crawler traffic (Cloudflare).
19. Googlebot’s share of AI and search crawling rose from 30% to 50% between May 2024 and May 2025 (Cloudflare).
20. Googlebot crawling grew 96% from May 2024 to May 2025 (Cloudflare).
21. Googlebot originated 4.5% of HTML requests in 2025, slightly more than all AI bots combined (Cloudflare Radar 2025).
22. Googlebot was responsible for over 28% of Verified Bot traffic in 2025 (Cloudflare Radar 2025).
23. GPTBot originated about 7.5% of Verified Bot traffic in 2025 (Cloudflare Radar 2025).
24. Search engine crawlers were 40% of Verified Bot traffic, AI crawlers 20% and SEO bots over 13% (Cloudflare Radar 2025).
25. GPTBot’s share of AI and search crawler traffic grew from 4.7% in July 2024 to 11.7% in July 2025 (Cloudflare).
26. Meta’s crawler jumped from 0.9% to 7.5%, and ClaudeBot rose from 6% to nearly 10% (Cloudflare).
27. ByteDance’s Bytespider dropped from 14.1% to 2.4%, and Amazonbot from 10.2% to 5.9% (Cloudflare).
28. Among AI-only crawlers, GPTBot surged from 5% to 30% share between May 2024 and May 2025, and Meta-ExternalAgent entered at 19% (Cloudflare).
29. Bytespider, the former leader, plummeted from 42% to 7% of AI crawler traffic (Cloudflare).
30. PerplexityBot recorded a 157,490% increase in raw requests, from a 0.2% share (Cloudflare).
31. AI and search crawler traffic grew 18% from May 2024 to May 2025 (Cloudflare).
32. Googlebot accounts for between 35% and 55% of crawler traffic in each world region (October 2025) (Cloudflare Radar 2025).
33. On Vercel’s network, GPTBot made 569 million requests in a month and Claude 370 million (Vercel).

How much traffic do AI crawlers send back?

34. Google remains the dominant gateway to discovery, with about 88% of referral traffic (Cloudflare).
35. Anthropic’s crawl-to-refer ratio reached as much as 500,000:1 in 2025 (Cloudflare Radar 2025).
36. For June 19-26, 2025, ratios ranged from Anthropic’s 70,900:1 down to Mistral’s 0.1:1 (Cloudflare Radar).
37. Anthropic still crawled 38,000 pages for every referred page visit in July 2025, after an 87% decline (Cloudflare).
38. Perplexity’s crawl-to-refer rose 256.7%, from 54 to 195 bots per human between January and July 2025 (Cloudflare).
39. OpenAI’s ratio dropped from 1,217 to 1,091 bots per human between January and July 2025 (-10%) (Cloudflare).
40. Google referrals to news sites were down 15% in April 2025 compared with January (Cloudflare).
41. Google has access to about 2x more information than leading AI companies because its crawler is mixed-use, per Cloudflare (Cloudflare).
42. Heavily crawled categories such as Retail, Computer Software, IT & Services and Financial Services saw human traffic decline as much as 40% in less than a year (Cloudflare).

What it means: AI platforms read far more than they refer. Being cited in an AI answer matters more than raw crawl volume, which is why AI visibility tools measure mentions, not hits.

How do sites control AI crawlers with robots.txt and llms.txt?

43. 17% of sites on Cloudflare block AI training crawlers, while fewer than 1% block search crawlers (Cloudflare).
44. About 14% of top domains with a robots.txt file (546 of 3,816) had directives aimed at AI bots in June 2025 (Cloudflare).
45. GPTBot is named in 4.5% of desktop and 4.2% of mobile robots.txt files, up from 2.9% and 2.7%, a ~55% increase (Web Almanac SEO).
46. ClaudeBot is named in 3.6% of desktop robots.txt files, nearly doubling from 1.9% (Web Almanac SEO).
47. PerplexityBot and ChatGPT-User each appear in 2.8% of desktop robots.txt files, and Google-Extended in 3.4% (Web Almanac SEO).
48. GPTBot is mentioned in the robots.txt of 20.9% of the top 1k sites but only 3.6% of the top 1 million (Web Almanac Generative AI).
49. About 94.1% of the ~12.9 million sites analyzed serve a robots.txt file with at least one directive (Web Almanac Generative AI).
50. 85% of robots.txt requests returned a valid 200 status in 2025, up from 84% (Web Almanac SEO).
51. AhrefsBot is named in 9.3% of desktop robots.txt files, a sign that SEO-tool crawlers get managed too (Web Almanac SEO).
52. Only 2.13% of desktop sites have a valid llms.txt file (324,184 valid files on mobile) (Web Almanac SEO).
53. 39.6% of llms.txt files are tied to the All in One SEO plugin and 3.6% to Yoast SEO (Web Almanac SEO).

How well do AI crawlers handle your site?

54. ChatGPT’s crawler spent 34.82% of its fetches on 404 pages and Claude’s 34.16%, versus 8.22% for Googlebot (Vercel).
55. ChatGPT and Claude crawlers fetch JavaScript files (11.50% and 23.84% of requests) but don’t execute them, so client-rendered content is invisible to them (Vercel).

What it means: AI crawlers read server-rendered HTML and waste a third of their fetches on dead URLs. Clean redirects, current sitemaps and pages that render without JavaScript are the cheapest GEO wins.

What to do with this data

1. Decide per purpose, not per bot. Cloudflare now splits Search, Agent and Training controls, and most sites block training while allowing search. Write robots.txt rules the same way.
2. Do not block the crawlers that cite you. ChatGPT-User visits pages when people ask ChatGPT questions, and OAI-SearchBot surfaces sites in ChatGPT search. Blocking them can remove you from those answers.
3. Measure citations, not crawls. Track whether ChatGPT, Perplexity and Google’s AI answers mention you with AI visibility tools or GEO tools.
4. Fix 404s and render HTML on the server. It helps Googlebot and AI crawlers alike. See SEO tools for agencies for audit options.
5. Treat llms.txt as optional. Adoption is about 2%, and the Web Almanac notes Google has often stated that it does not use the file.

How we compiled these statistics

We fetched primary sources on October 17, 2026 and recorded 64 data points in docs/content/stats-ledgers/ai-crawler-statistics.json. Every numbered figure above maps to a ledger row with a verbatim quote. Cloudflare figures describe traffic on Cloudflare’s network (more than 20% of the web), not the whole Internet, and crawl-to-refer ratios undercount referrals from native apps that send no Referer header. Web Almanac figures come from the HTTP Archive crawl of millions of sites; its SEO and Generative AI chapters use different datasets, so their GPTBot baselines differ. Vercel’s figures cover one month of its own network to December 2024. Sources: Cloudflare, The Internet has a second audience (2026), Cloudflare, Content Independence Day, one year on (2026), Cloudflare, The crawl-to-click gap, Cloudflare, From Googlebot to GPTBot, Cloudflare Radar 2025 Year in Review, Cloudflare, crawl-to-refer ratio on Radar, Cloudflare, Rethinking cache for the AI era, Web Almanac 2025 SEO chapter, Web Almanac 2025 Generative AI chapter and Vercel, The rise of the AI crawler. Related reading: AI search statistics, SEO statistics, local SEO statistics, SEO tools.