# Common Crawl Statistics 2026: 2.16B Pages a Crawl, 64% of LLMs

Common Crawl statistics: 2.16B pages in the Oct 2026 crawl, used by 64% of 47 LLMs, 584,107 llms.txt files analysed. 80 sourced stats for AI visibility.

_Source: https://tools8020.com/blog/common-crawl-statistics/_

- **Author:** [The tools8020 team](https://tools8020.com/authors/tools8020-team/)
- **Published:** 2026-11-11

_A data roundup of Common Crawl, the open web archive behind most large language models, taken only from Common Crawl’s own pages: its About page, FAQ, CCBot page, monthly crawl releases, web graph releases and research posts._

**Common Crawl is the layer under AI visibility: its October 2026 crawl holds 2.16 billion pages from 41.4 million hosts, and Common Crawl says at least 64% of 47 LLMs published from 2019 to 2023 were trained on filtered versions of its data** ([October 2026 crawl release](https://commoncrawl.org/blog/october-2026-crawl-archive-now-available), [Common Crawl About page](https://commoncrawl.org/about)). Use this page if you need numbers on how pages get into AI training data, how big each crawl is, whether llms.txt matters, or what blocking CCBot costs you. Skip it if you want live AI answer citations; this is the training-data layer, not the answer layer. To see the answer layer for your own brand, [Mentionova](https://tools8020.com/tools/mentionova/) tracks what [ChatGPT](https://tools8020.com/tools/chatgpt/), [Perplexity](https://tools8020.com/tools/perplexity/), [Claude](https://tools8020.com/tools/claude/), Gemini and Google AI say about you each week and which sources they cite, from $99/mo billed annually. Every figure below has a verbatim quote in our ledger, and third-party studies that Common Crawl quotes are labelled as such.

## Key takeaways

- **2.16 billion** pages in the October 2026 crawl (371.19 TiB uncompressed), from 41.4 million hosts ([October 2026 crawl release](https://commoncrawl.org/blog/october-2026-crawl-archive-now-available))
- **64%** of 47 LLMs published from 2019 to 2023 used filtered Common Crawl data, per the Mozilla Foundation study cited by Common Crawl ([Common Crawl, About](https://commoncrawl.org/about))
- **More than 80%** of GPT-3’s training tokens came from filtered Common Crawl data ([Common Crawl, About](https://commoncrawl.org/about))
- **830.1 million** brand-new URLs entered the October 2026 crawl ([October 2026 crawl release](https://commoncrawl.org/blog/october-2026-crawl-archive-now-available))
- **11.72%** of sampled hosts served a text llms.txt file in July 2026, and 41.34% of all files came from [Wix](https://tools8020.com/tools/wix/) ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive))
- **26.23%** of decided CCBot verdicts inside llms.txt files were denials, yet none of the 32 sites behind them blocked CCBot in robots.txt ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive))
- **About 41 percent** of the latest crawl is English ([Common Crawl, AI Visibility Audit](https://commoncrawl.org/blog/introducing-the-ai-visibility-audit))

**2.16B**: pages, October 2026 crawl

**64%**: of 47 LLMs used CC data

**\>10 PiB**: total archive size

**584,107**: llms.txt files analysed

## How big is Common Crawl?

Common Crawl is a nonprofit that has archived the public web since 2008 and gives the data away. Its own counters put the archive above 10 PiB.

1. **Common Crawl’s total archive is larger than 10 PiB** ([Common Crawl, About](https://commoncrawl.org/about)).
2. **Each crawl holds about 2 billion or more pages, and crawls are published about once a month** ([Common Crawl, About](https://commoncrawl.org/about)).
3. **Common Crawl was founded in 2007 by Gil Elbaz and has collected web data since 2008** ([Common Crawl, About](https://commoncrawl.org/about)).
4. **CCBot processes approximately three billion pages per crawl cycle** ([Common Crawl, About](https://commoncrawl.org/about)).
5. **Each month it crawls several billion new pages from a frontier of over 1 trillion pages** ([Opt-Out Registry post](https://commoncrawl.org/blog/common-crawl-foundation-opt-out-registry)).
6. **Its datasets are cited in 12,000+ research papers, up from “over 10,000” in September 2025** ([Common Crawl, About](https://commoncrawl.org/about)) ([Opt-Out Registry post](https://commoncrawl.org/blog/common-crawl-foundation-opt-out-registry)).
7. **The newest crawl listed on the data overview is CC-MAIN-2026-43, the October 2026 archive** ([Common Crawl data overview](https://commoncrawl.org/overview)).
8. **The Host Index, launched in 2025, holds one row per web host per crawl, 24 crawls at about 7 gigabytes each** ([Host Index announcement](https://commoncrawl.org/blog/introducing-the-host-index)).

## How big is each monthly crawl in 2026?

The crawl has held at roughly 2 billion pages a month through 2026, smaller than a year ago. October 2025 held 2.61 billion pages; October 2026 holds 2.16 billion.

9. **The October 2026 crawl holds 2.16 billion pages (371.19 TiB uncompressed), crawled October 5 to 18** ([October 2026 crawl release](https://commoncrawl.org/blog/october-2026-crawl-archive-now-available)).
10. **October 2026 captures came from 41.4 million hosts and 34.3 million registered domains** ([October 2026 crawl release](https://commoncrawl.org/blog/october-2026-crawl-archive-now-available)).
11. **The October 2026 crawl added 830.1 million new URLs never visited before** ([October 2026 crawl release](https://commoncrawl.org/blog/october-2026-crawl-archive-now-available)).
12. **October 2026 WARC files total 86.19 TiB compressed across 100,000 files** ([October 2026 crawl release](https://commoncrawl.org/blog/october-2026-crawl-archive-now-available)).
13. **The September 2026 crawl holds 2.17 billion pages (361.4 TiB), with 587.2 million new URLs** ([September 2026 crawl release](https://commoncrawl.org/blog/september-2026-crawl-archive-now-available)).
14. **The August 2026 crawl holds 2.14 billion pages (360 TiB) and 0.7 billion new URLs** ([August 2026 crawl release](https://commoncrawl.org/blog/august-2026-crawl-archive-now-available)).
15. **The July 2026 crawl holds 2.14 billion pages (364.01 TiB) from 40.5 million hosts** ([July 2026 crawl release](https://commoncrawl.org/blog/july-2026-crawl-archive-now-available)).
16. **The June 2026 crawl holds 2.10 billion pages (354.59 TiB), with 479 million new URLs, the fewest this year** ([June 2026 crawl release](https://commoncrawl.org/blog/june-2026-crawl-archive-now-available)).
17. **The May 2026 crawl holds 2.16 billion pages (365.56 TiB), with 668 million new URLs** ([May 2026 crawl release](https://commoncrawl.org/blog/may-2026-crawl-archive-now-available)).
18. **The April 2026 crawl holds 2.19 billion pages (379.2 TiB), with 660.5 million new URLs** ([April 2026 crawl release](https://commoncrawl.org/blog/april-2026-crawl-archive-now-available)).
19. **The March 2026 crawl holds 1.97 billion pages (344.64 TiB), the smallest of 2026** ([March 2026 crawl release](https://commoncrawl.org/blog/march-2026-crawl-archive-now-available)).
20. **The February 2026 crawl holds 2.1 billion pages (363 TiB), with 626 million new URLs** ([February 2026 crawl release](https://commoncrawl.org/blog/february-2026-crawl-archive-now-available)).
21. **The January 2026 crawl holds 2.3 billion pages (398 TiB), the largest of 2026, from 44.9 million hosts** ([January 2026 crawl release](https://commoncrawl.org/blog/january-2026-crawl-archive-now-available)).
22. **A year earlier, the October 2025 crawl held 2.61 billion pages (468 TiB) from 47 million hosts** ([October 2025 crawl release](https://commoncrawl.org/blog/october-2025-crawl-archive-now-available)).

| Crawl | Pages | New URLs |
| --- | --- | --- |
| Oct 2026 | 2.16B | 830.1M |
| Sep 2026 | 2.17B | 587.2M |
| Aug 2026 | 2.14B | 0.7B |
| Jul 2026 | 2.14B | 603M |
| Jun 2026 | 2.10B | 479M |
| May 2026 | 2.16B | 668M |
| Apr 2026 | 2.19B | 660.5M |
| Mar 2026 | 1.97B | 600M |
| Feb 2026 | 2.1B | 626M |
| Jan 2026 | 2.3B | 616M |
| Oct 2025 | 2.61B | 747M |

_Source: Common Crawl monthly release posts ([Oct 2026](https://commoncrawl.org/blog/october-2026-crawl-archive-now-available), [Sep 2026](https://commoncrawl.org/blog/september-2026-crawl-archive-now-available), [Aug 2026](https://commoncrawl.org/blog/august-2026-crawl-archive-now-available), [Jul 2026](https://commoncrawl.org/blog/july-2026-crawl-archive-now-available), [Jun 2026](https://commoncrawl.org/blog/june-2026-crawl-archive-now-available), [May 2026](https://commoncrawl.org/blog/may-2026-crawl-archive-now-available), [Apr 2026](https://commoncrawl.org/blog/april-2026-crawl-archive-now-available), [Mar 2026](https://commoncrawl.org/blog/march-2026-crawl-archive-now-available), [Feb 2026](https://commoncrawl.org/blog/february-2026-crawl-archive-now-available), [Jan 2026](https://commoncrawl.org/blog/january-2026-crawl-archive-now-available), [Oct 2025](https://commoncrawl.org/blog/october-2025-crawl-archive-now-available)), fetched November 11, 2026. New URLs are pages not visited in any prior crawl._

| Crawl | Hosts / domains | Size |
| --- | --- | --- |
| Oct 2026 | 41.4M / 34.3M | 371.19 TiB |
| Jul 2026 | 40.5M / 33.2M | 364.01 TiB |
| Apr 2026 | 43.2M / 35.4M | 379.2 TiB |
| Jan 2026 | 44.9M / 36.9M | 398 TiB |
| Oct 2025 | 47M / 38.5M | 468 TiB |

_Source: the same Common Crawl release posts. Size is uncompressed content as Common Crawl reports it; domains are registered domains._

Crawl size

Pages per monthly crawl, billions

Common Crawl release posts, Oct 2025 to Oct 2026

- Oct 2025: 2.61B
- Jan 2026: 2.3B
- Apr 2026: 2.19B
- Jul 2026: 2.14B
- Oct 2026: 2.16B

_Source: Common Crawl monthly crawl release posts listed above._

## How much AI training data comes from Common Crawl?

This is why Common Crawl matters for AI visibility: if a page is not in the crawl, it is much less likely to be in the model.

23. **Common Crawl says 64% of LLMs were trained on its data** ([Common Crawl, About](https://commoncrawl.org/about)).
24. **The Mozilla Foundation study it cites found at least 64% of 47 LLMs published between 2019 and 2023 used filtered Common Crawl data** ([Common Crawl, About](https://commoncrawl.org/about)).
25. **More than 80% of GPT-3’s training tokens came from filtered Common Crawl data (OpenAI, 2020)** ([Common Crawl, About](https://commoncrawl.org/about)).
26. **Google built C4 by filtering a single Common Crawl snapshot and used it to train T5 in 2019** ([Common Crawl, About](https://commoncrawl.org/about)).
27. **Meta’s LLaMA used Common Crawl data processed through the CCNet pipeline** ([Common Crawl, About](https://commoncrawl.org/about)).
28. **Falcon was trained on RefinedWeb, processed from Common Crawl** ([Common Crawl, About](https://commoncrawl.org/about)).
29. **A 2025 COLM paper found that honouring robots.txt opt-outs does not degrade LLM general knowledge, a close-to-zero “Data Compliance Gap”** ([Common Crawl, About](https://commoncrawl.org/about)).
30. **Common Crawl says opt-out requests are flagged to OpenAI, Meta, Amazon and researchers, and that exclusion is “essentially permanent”** ([Common Crawl, From SEO to AIO](https://commoncrawl.org/blog/from-seo-to-aio-why-your-content-needs-to-exist-in-ai-training-data)).

| Model or dataset | Common Crawl role |
| --- | --- |
| GPT-3 (OpenAI, 2020) | More than 80% of training tokens |
| C4 / T5 (Google, 2019) | One filtered snapshot |
| LLaMA (Meta, 2023) | Processed through CCNet |
| Falcon (TII) | RefinedWeb, from Common Crawl |
| 47 LLMs, 2019 to 2023 | At least 64% used it |

_Source: [Common Crawl About page](https://commoncrawl.org/about), “Role in AI Development”, fetched November 11, 2026. The 47-model figure is the Mozilla Foundation’s 2024 study as cited by Common Crawl._

## How does Common Crawl decide what to crawl?

CCBot does not try to archive whole sites. It samples, and it prioritises pages by how central a site is to the web’s link graph.

31. **CCBot prioritises URLs with Harmonic Centrality, a measure of how close a site is to the web’s structural core, alongside PageRank** ([Common Crawl, About](https://commoncrawl.org/about)).
32. **The August to October 2026 host-level web graph has 242.8 million nodes and 2.4 billion edges** ([Aug to Oct 2026 web graph release](https://commoncrawl.org/blog/host--and-domain-level-web-graphs-august-september-and-october-2026)).
33. **73.55% of host nodes are dangling, and the largest strongly connected component holds 15.91% of hosts** ([Aug to Oct 2026 web graph release](https://commoncrawl.org/blog/host--and-domain-level-web-graphs-august-september-and-october-2026)).
34. **The domain-level graph has 135.0 million nodes and 1.8 billion edges, with 24.62% of domains in the largest strongly connected component** ([Aug to Oct 2026 web graph release](https://commoncrawl.org/blog/host--and-domain-level-web-graphs-august-september-and-october-2026)).
35. **The full Web Graph holds about 607 million domain records, and each monthly release covers 94 to 163 million domains** ([Common Crawl, Web Graph data for AI ranking signals](https://commoncrawl.org/blog/how-seos-are-using-common-crawls-web-graph-data-for-ai-ranking-signals)).
36. **Common Crawl archives a randomly selected subset of each site, not entire websites** ([Common Crawl FAQ](https://commoncrawl.org/faq)).
37. **CCBot uses any sitemap announced in robots.txt** ([Common Crawl FAQ](https://commoncrawl.org/faq)).
38. **The crawler recrawls pages that rarely change “less and less often”** ([Common Crawl, Measuring crawled coverage](https://commoncrawl.org/blog/measuring-crawled-coverage-of-a-website-in-common-crawl)).
39. **CCBot follows up to four redirects (five for robots.txt) and does not execute JavaScript or use cookies** ([Common Crawl FAQ](https://commoncrawl.org/faq)).
40. **CCBot obeys the Crawl-delay directive and backs off on HTTP 429 and 5xx responses** ([Common Crawl FAQ](https://commoncrawl.org/faq)).
41. **CCBot runs on dedicated IP ranges with reverse DNS, and the ranges are published as JSON** ([CCBot page](https://commoncrawl.org/ccbot)).
42. **By default CCBot treats the absence of blocking rules as permission to crawl** ([Common Crawl, AI Optimization Is Here](https://commoncrawl.org/blog/ai-optimization-is-here-are-you-ready-for-search-2-0)).

## What does Common Crawl say about AI visibility?

Common Crawl now publishes advice for SEOs and GEOs directly. Its message: inclusion in the crawl comes before inclusion in a model.

43. **Common Crawl’s AI Visibility Audit is a five-check audit using only free tools that takes about 90 minutes** ([Common Crawl, AI Visibility Audit](https://commoncrawl.org/blog/introducing-the-ai-visibility-audit)).
44. **English is roughly 41 percent of the latest crawl** ([Common Crawl, AI Visibility Audit](https://commoncrawl.org/blog/introducing-the-ai-visibility-audit)).
45. **Common Crawl says CDN and WAF defaults now silently block AI crawlers and training-data crawlers** ([Common Crawl, AI Visibility Audit](https://commoncrawl.org/blog/introducing-the-ai-visibility-audit)).
46. **Children’s Hospital Los Angeles was missing from Gemini and ChatGPT answers because its Cloudflare defaults blocked AI crawlers, including CCBot** ([Common Crawl, From SEO to AIO](https://commoncrawl.org/blog/from-seo-to-aio-why-your-content-needs-to-exist-in-ai-training-data)).
47. **Common Crawl puts the SEO industry at $80 billion and search ads at a $300 billion market** ([Common Crawl, AI Optimization Is Here](https://commoncrawl.org/blog/ai-optimization-is-here-are-you-ready-for-search-2-0)).
48. **In a study quoted by Common Crawl (Brie Moreau with DataForSEO), a Google #1 result had a 46-48% chance of being cited by AI, falling to about 37% at position 2** ([Common Crawl, Web Graph data for AI ranking signals](https://commoncrawl.org/blog/how-seos-are-using-common-crawls-web-graph-data-for-ai-ranking-signals)).
49. **The same study puts AI citation probability at about 19-20% by position 10** ([Common Crawl, Web Graph data for AI ranking signals](https://commoncrawl.org/blog/how-seos-are-using-common-crawls-web-graph-data-for-ai-ranking-signals)).
50. **Comparative listicles made up 32.5% of AI citations across 177 million sources in that study; blogs and opinion pieces were 9.91% and store pages 4.73%** ([Common Crawl, Web Graph data for AI ranking signals](https://commoncrawl.org/blog/how-seos-are-using-common-crawls-web-graph-data-for-ai-ranking-signals)).

AI citations

Share of AI citations by content type

Brie Moreau / DataForSEO study, as quoted by Common Crawl

- Comparative listicles: 32.5%
- Blogs and opinion: 9.91%
- Commercial and store pages: 4.73%

_Source: [Common Crawl, “How SEOs Are Using Common Crawl’s Web Graph Data for AI Ranking Signals”](https://commoncrawl.org/blog/how-seos-are-using-common-crawls-web-graph-data-for-ai-ranking-signals). Third-party figures as reported by Common Crawl; we did not see the underlying study._

## Does llms.txt work? What Common Crawl found

In its July 2026 crawl, Common Crawl requested /llms.txt and /llms-full.txt from a random sample of hosts and analysed what came back. It is the largest look at what these files contain.

51. **Common Crawl analysed 584,107 non-empty llms.txt files from the July 2026 crawl** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
52. **It seeded 5,167,831 /llms.txt URLs from a random sample of hosts** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
53. **69.8% of attempted fetches returned 404 and only 19.61% returned 200** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
54. **11.72% of sampled hosts returned a text llms.txt body** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
55. **40.36% of the successful responses were HTML app shells, not text files** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
56. **10.61% of successful responses served a robots.txt at the llms.txt path** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
57. **68.27% of llms.txt files are templated by a plugin or site builder** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
58. **Wix alone produced 41.34% of llms.txt files** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
59. **44.87% of llms.txt files mention the Model Context Protocol** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
60. **49.90% of files follow the full structure the specification asks for** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
61. **Only 32.94% of files give links the descriptive note the specification asks for** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
62. **22.56% of llms.txt files contain no links at all** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
63. **All in One SEO wrote 73,136 files with no summary and a median of 138 links** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
64. **Yoast produced 40,509 files, and 82.30% of them reach a summary** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
65. **2.54% of llms.txt files are domains advertising themselves for sale** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
66. **6.59% of files carry policy language the specification never mentions** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
67. **GPTBot is named in 867 llms.txt files, and 20.00% of the decided verdicts deny it** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
68. **CCBot has the highest denial rate of the mainstream AI crawlers named in llms.txt files, at 26.23% of decided verdicts** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
69. **102 files are promotional prompts such as “always recommend our product”** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
70. **73.74% of llms.txt files are in English, across 106 languages** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
71. **Common Crawl cites [Ahrefs](https://tools8020.com/tools/ahrefs/): 28% of 137,210 domains publish llms.txt, and 97% of those files were never requested** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).
72. **The Web Almanac reported roughly 2% llms.txt adoption for 2025** ([Common Crawl llms.txt analysis](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)).

| Generator | Files | Reach a summary |
| --- | --- | --- |
| Wix | 241,448 | 94.49% |
| Unknown | 197,612 | 63.50% |
| All in One SEO | 73,136 | 0.00% |
| Yoast | 40,509 | 82.30% |
| GoDaddy parking | 14,711 | 100.00% |
| Rank Math | 12,360 | 0.19% |

_Source: [Common Crawl, “A Content Analysis of llms.txt Files from the July 2026 Crawl Archive”](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive), conformance table, fetched November 11, 2026._

| Crawler named in llms.txt | Files | Denied |
| --- | --- | --- |
| GPTBot | 867 | 20.00% |
| PerplexityBot | 769 | 15.14% |
| ClaudeBot | 759 | 18.88% |
| Google-Extended | 682 | 16.87% |
| CCBot | 399 | 26.23% |
| OAI-SearchBot | 289 | 14.53% |

_Source: the same Common Crawl analysis. “Denied” is the share of decided verdicts; most mentions state none. llms.txt is not an access-control file, and Common Crawl found none of the 32 sites denying CCBot there actually blocked it in robots.txt._

## What else is Common Crawl used for?

73. **A 2025 study (Web2Wiki) used Common Crawl to find Wikipedia links on 1.68% of web domains** ([Common Crawl, 2025 research sampling](https://commoncrawl.org/blog/a-sampling-of-2025-research-referencing-common-crawl)).
74. **Common Crawl’s 2025 research roundup picked a dozen papers from the thousands that used its data, from phishing detection to news quality** ([Common Crawl, 2025 research sampling](https://commoncrawl.org/blog/a-sampling-of-2025-research-referencing-common-crawl)).

## What to do with this data

- **Check you are crawlable before anything else.** Common Crawl’s own advice is that a page missing from the crawl is far less likely to be in the model. Look for CCBot, GPTBot and ClaudeBot in your server logs, and check your CDN’s bot settings; Cloudflare-style defaults are the most common silent block. Our [AI crawler statistics](https://tools8020.com/blog/ai-crawler-statistics/) show how often each bot is blocked.
- **Use robots.txt, not llms.txt, for access rules.** llms.txt grants and blocks nothing. If you want CCBot in, make sure robots.txt does not disallow it; if you want it out, robots.txt is the file it honours.
- **Do not expect llms.txt to move AI answers.** Most files are plugin output, a fifth have no links, and Common Crawl cites Ahrefs’ finding that 97% were never requested. Spend the time on [structured data](https://tools8020.com/blog/schema-markup-statistics/) and pages that AI engines can cite.
- **Write the pages AI engines cite.** In the study Common Crawl quotes, comparative listicles took 32.5% of AI citations. Comparisons and sourced lists, like our [best AI citation tracking tools](https://tools8020.com/best/ai-citation-tracking-tools/), are the format to build.
- **Measure the answer layer separately.** Training data is upstream; what buyers see is the answer. [Mentionova](https://tools8020.com/tools/mentionova/) tracks your brand across six engines including Google AI Overviews and AI Mode, and the [Google AI Mode tracking tools](https://tools8020.com/best/google-ai-mode-tracking-tools/) ranking compares it with [SE Ranking](https://tools8020.com/tools/se-ranking/), [Semrush](https://tools8020.com/tools/semrush/) and Ahrefs.

For the wider picture, see our [GEO statistics](https://tools8020.com/blog/geo-statistics/), [AI search statistics](https://tools8020.com/blog/ai-search-statistics/), [technical SEO statistics](https://tools8020.com/blog/technical-seo-statistics/) and [Wikipedia statistics](https://tools8020.com/blog/wikipedia-statistics/).

## Definitions

- **Common Crawl:** a 501(c)(3) nonprofit that crawls the web and publishes its archives free of charge.
- **CCBot:** Common Crawl’s crawler, based on Apache Nutch; it identifies itself as CCBot/2.0.
- **Crawl archive:** one monthly release, named CC-MAIN-year-week (for example CC-MAIN-2026-43 for October 2026).
- **WARC, WAT, WET:** raw HTTP responses, extracted metadata and extracted plain text.
- **Harmonic Centrality:** a link-graph measure of how close a site is to the rest of the web; CCBot uses it to prioritise crawling.
- **Dangling node:** a host in the web graph with no outgoing links to other hosts, often because it was never crawled.
- **llms.txt:** a proposed Markdown file at a site’s root that lists the pages a site wants language models to read. It is a convention, not a standard, and no crawler must read it.
- **Opt-Out Ledger:** Common Crawl’s public list of sites that sent legal removal requests (formerly the Opt-Out Registry).

## Frequently asked questions

### How big is Common Crawl?

The archive is larger than 10 PiB. The October 2026 crawl alone holds 2.16 billion pages, or 371.19 TiB uncompressed, from 41.4 million hosts.

### How often does Common Crawl crawl the web?

About once a month. The October 2026 crawl ran from October 5 to October 18 and was announced on October 21, 2026.

### Is ChatGPT trained on Common Crawl?

OpenAI’s GPT-3 paper reported that more than 80% of training tokens came from filtered Common Crawl data, per Common Crawl’s summary.

### How many LLMs use Common Crawl data?

The Mozilla Foundation study cited by Common Crawl found that at least 64% of 47 LLMs published between 2019 and 2023 were trained on filtered versions of Common Crawl.

### Should I block CCBot?

Only if you do not want your pages in AI training data. Common Crawl argues that being absent from the crawl lowers your odds of appearing in AI answers, and it says opt-outs are passed to OpenAI, Meta, Amazon and researchers and are essentially permanent.

### How do I block or allow CCBot?

Add “User-agent: CCBot” with “Disallow: /” to robots.txt to block it. CCBot treats the absence of blocking rules as permission, so allowing it only means not disallowing it.

### Does llms.txt help AI visibility?

There is little evidence it does. Common Crawl found 11.72% of sampled hosts serve one, 68.27% of files are plugin templates and 22.56% have no links, and it cites Ahrefs’ finding that 97% of published files were never requested.

### Is my site in Common Crawl?

Search the Common Crawl index server at index.commoncrawl.org for your domain in a recent crawl. Remember the crawl samples sites, so missing pages do not mean you are blocked.

### Why does English matter for AI visibility?

English is roughly 41 percent of the latest crawl, and Common Crawl advises publishing in English alongside your local language because smaller languages are underrepresented in training data.

### What content types do AI engines cite most?

In the Brie Moreau / DataForSEO study quoted by Common Crawl, comparative listicles took 32.5% of AI citations, blogs and opinion pieces 9.91% and store pages 4.73%.

## How we compiled these statistics

We fetched 25 primary Common Crawl pages on **November 11, 2026** and recorded ** 80 data points** in `docs/content/stats-ledgers/common-crawl-statistics.json`. Every numbered figure above maps to a ledger row with a verbatim quote from the page we fetched. Sources are Common Crawl’s About page, FAQ, CCBot page and data overview, its monthly crawl release posts for January to October 2026 and October 2025, the August to October 2026 web graph release, and its research and guidance posts. Figures from the Mozilla Foundation, OpenAI, Ahrefs, the Web Almanac and Brie Moreau’s study are as quoted by Common Crawl; the Mozilla report itself blocked our fetch, so we did not see it directly. We computed no growth rates ourselves. Common Crawl publishes a new crawl about once a month. Related reading: [AI crawler statistics](https://tools8020.com/blog/ai-crawler-statistics/), [zero-click search statistics](https://tools8020.com/blog/zero-click-search-statistics/) and [Google AI Mode statistics](https://tools8020.com/blog/google-ai-mode-statistics/).

## Sources

- [Common Crawl: About](https://commoncrawl.org/about)
- [Common Crawl: Frequently Asked Questions](https://commoncrawl.org/faq)
- [Common Crawl: CCBot](https://commoncrawl.org/ccbot)
- [Common Crawl: Data overview](https://commoncrawl.org/overview)
- [Common Crawl: October 2026 Crawl Archive Now Available](https://commoncrawl.org/blog/october-2026-crawl-archive-now-available)
- [Common Crawl: September 2026 Crawl Archive Now Available](https://commoncrawl.org/blog/september-2026-crawl-archive-now-available)
- [Common Crawl: August 2026 Crawl Archive Now Available](https://commoncrawl.org/blog/august-2026-crawl-archive-now-available)
- [Common Crawl: July 2026 Crawl Archive Now Available](https://commoncrawl.org/blog/july-2026-crawl-archive-now-available)
- [Common Crawl: June 2026 Crawl Archive Now Available](https://commoncrawl.org/blog/june-2026-crawl-archive-now-available)
- [Common Crawl: May 2026 Crawl Archive Now Available](https://commoncrawl.org/blog/may-2026-crawl-archive-now-available)
- [Common Crawl: April 2026 Crawl Archive Now Available](https://commoncrawl.org/blog/april-2026-crawl-archive-now-available)
- [Common Crawl: March 2026 Crawl Archive Now Available](https://commoncrawl.org/blog/march-2026-crawl-archive-now-available)
- [Common Crawl: February 2026 Crawl Archive Now Available](https://commoncrawl.org/blog/february-2026-crawl-archive-now-available)
- [Common Crawl: January 2026 Crawl Archive Now Available](https://commoncrawl.org/blog/january-2026-crawl-archive-now-available)
- [Common Crawl: October 2025 Crawl Archive Now Available](https://commoncrawl.org/blog/october-2025-crawl-archive-now-available)
- [Common Crawl: Host- and Domain-Level Web Graphs, August, September and October 2026](https://commoncrawl.org/blog/host--and-domain-level-web-graphs-august-september-and-october-2026)
- [Common Crawl: How SEOs Are Using Common Crawl’s Web Graph Data for AI Ranking Signals](https://commoncrawl.org/blog/how-seos-are-using-common-crawls-web-graph-data-for-ai-ranking-signals)
- [Common Crawl: A Content Analysis of llms.txt Files from the July 2026 Crawl Archive](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)
- [Common Crawl: Introducing the AI Visibility Audit](https://commoncrawl.org/blog/introducing-the-ai-visibility-audit)
- [Common Crawl: From SEO to AIO](https://commoncrawl.org/blog/from-seo-to-aio-why-your-content-needs-to-exist-in-ai-training-data)
- [Common Crawl: AI Optimization Is Here](https://commoncrawl.org/blog/ai-optimization-is-here-are-you-ready-for-search-2-0)
- [Common Crawl: Opt-Out Registry](https://commoncrawl.org/blog/common-crawl-foundation-opt-out-registry)
- [Common Crawl: Measuring Crawled Coverage of a Website](https://commoncrawl.org/blog/measuring-crawled-coverage-of-a-website-in-common-crawl)
- [Common Crawl: Introducing the Host Index](https://commoncrawl.org/blog/introducing-the-host-index)
- [Common Crawl: A Sampling of 2025 Research Referencing Common Crawl](https://commoncrawl.org/blog/a-sampling-of-2025-research-referencing-common-crawl)

---
_tools8020, https://tools8020.com. Markdown rendering of the page above for AI readers._
