Technical SEO Statistics (2026): 97 Stats From 16 Million Websites

A data roundup of how the web is actually built for search: crawl rules, indexing signals, page experience, on-page markup, structured data and international targeting. Every figure comes from HTTP Archive’s 2025 Web Almanac SEO chapter, its methodology page, or Google’s own documentation.

Technical SEO is mostly boring and mostly working: 84.9% of sites return a valid robots.txt, 68% of desktop pages carry a canonical tag, 91.7% use HTTPS, and half of home pages now ship structured data. The new part is AI. gptbot is named in 4.5% of desktop robots.txt files, up from 2.9% a year earlier, and about 2% of sites already serve an llms.txt file. The data comes from the Web Almanac, which analyzed 16,213,084 websites from HTTP Archive’s July 2025 crawl. If you are tracking whether AI engines can read and cite your pages, the checks in this page are the ones that gate that outcome, and a tool such as Mentionova shows you the result on the citation side. Each figure below links to its source and has a verbatim quote in our ledger.

Key takeaways

• 84.9% of sites return a valid 200 for robots.txt, and 13.3% of desktop sites return a 404 (Web Almanac)
• 4.5% of desktop robots.txt files name gptbot and 3.6% name claudebot, up from 2.9% and 1.9% in 2024 (Web Almanac)
• About 2% of sites have an llms.txt file, and 39.6% of those were made by one SEO plugin (Web Almanac)
• 68% of desktop pages have a canonical tag, and mismatches affect only 0.72% (Web Almanac)
• 48% of mobile pages had a good overall Core Web Vitals experience in June 2025, against 56% on desktop (Web Almanac)
• 50% of home pages now carry structured data, and JSON-LD is on 43% of them (Web Almanac)
• 20.3% of desktop pages use hreflang, roughly double the share a year earlier (Web Almanac)

What data is this based on?

The Web Almanac is an annual report by HTTP Archive. It reads the same crawl for every chapter, so the numbers are comparable across topics.

1. The 2025 Web Almanac dataset covers 16,213,084 websites (Web Almanac methodology).
2. Almanac metrics come from the July 2025 HTTP Archive crawl unless a chapter says otherwise (Web Almanac methodology).

What it means. These are measurements of how sites are built, not how they rank. A page can pass every check here and still not rank or get cited. But a page that fails them often cannot be crawled, indexed or parsed at all, which makes this the floor for both classic SEO and AI search. Our SEO statistics page covers rankings and traffic, and our local SEO statistics page covers the local pack.

How well does the web handle crawling and robots.txt?

Almost every site now serves a valid robots.txt, and file size is a non-issue. The problems are at the edges: sites with no file, empty files and 5xx errors.

3. robots.txt returned a valid 200 status for 84.9% of sites on both desktop and mobile (Web Almanac SEO).
4. A 404 (no robots.txt file) was returned for 13.3% of desktop and 13.2% of mobile sites (Web Almanac SEO).
5. Valid robots.txt responses rose to 85% in 2025 from 84% in 2024 (Web Almanac SEO).
6. Timeouts hit 1.0% of desktop and 1.1% of mobile robots.txt requests (Web Almanac SEO).
7. Completely empty robots.txt files were served by 1.8% of desktop sites (Web Almanac SEO).
8. Only 0.1% of sites exceeded the 500 KB parsing cutoff Google enforces (Web Almanac SEO).
9. Google enforces a robots.txt size limit of 500 kibibytes and ignores content after it (Google Search Central).
10. The wildcard user agent appears in 77.04% of desktop robots.txt files (Web Almanac SEO).
11. adsbot-google is the most named specific crawler at 9.82% of desktop robots.txt files (Web Almanac SEO).
12. ahrefsbot is named in 9.29% of desktop robots.txt files (Web Almanac SEO).
13. mj12bot is named in 7.31% of desktop robots.txt files (Web Almanac SEO).
14. googlebot is named in 6.22% of desktop robots.txt files (Web Almanac SEO).
15. bingbot ranks 22nd among named user agents and appears in under 3% of robots.txt files (Web Almanac SEO).

What it means. Missing files are not an error, because no robots.txt means no restrictions. An empty file is a weaker choice because the standard does not define how every bot should treat it. The wildcard dominates (77.04%) because most owners want one rule for all bots. The bot names that appear most are not search engines. They are an ad crawler and SEO-tool crawlers, which suggests owners use robots.txt partly to manage crawl load.

How many sites are managing AI crawlers?

Robots.txt has become the place where site owners state whether AI companies may use their content. The share doing so is still small in absolute terms, and it grew quickly.

16. gptbot is named in 4.5% of desktop robots.txt files, up from 2.9% in 2024 (Web Almanac SEO).
17. claudebot is named in 3.6% of desktop robots.txt files, up from 1.9% in 2024 (Web Almanac SEO).
18. ccbot (Common Crawl) is named in 3.5% of desktop robots.txt files (Web Almanac SEO).
19. amazonbot is named in 3.3% of desktop robots.txt files (Web Almanac SEO).
20. google-extended is named in 3.4% of desktop robots.txt files (Web Almanac SEO).
21. perplexitybot is named in 2.8% of desktop robots.txt files (Web Almanac SEO).
22. Valid llms.txt files were found on 324,184 mobile sites (Web Almanac SEO).
23. llms.txt adoption is about 2% of sites (Web Almanac SEO). The Almanac added llms.txt monitors for the first time in 2025.
24. 39.6% of llms.txt files were created by All in One SEO (Web Almanac SEO).
25. 3.6% of llms.txt files were created by Yoast SEO (Web Almanac SEO).
26. The Almanac says no Google service currently uses llms.txt (Web Almanac SEO).

Source: HTTP Archive, Web Almanac 2025 SEO chapter, figure 5.5. Share of sites naming each user agent in robots.txt. Checked October 28, 2026.

What it means. Naming a bot is a statement of intent: allow it, block it or limit it. The Almanac notes that site owners now face two separate questions, one about search indexing and one about AI training. For the matching crawl-traffic data from the bot side, see our AI crawler statistics. On llms.txt, treat the numbers with care. Almost 40% of files come from one plugin’s default, so adoption reflects plugin defaults more than a deliberate choice, and the Almanac says Google does not use the file.

How well are pages indexed and canonicalized?

Robots meta tags and canonical tags tell engines what to index and which version of a page is primary. Both are widely used and mostly correct.

27. Meta robots directives are used on 47.0% of desktop sites (Web Almanac SEO).
28. X-Robots-Tag headers are used on only 0.6% of desktop sites (Web Almanac SEO).
29. The noindex directive appears on 3.5% of desktop sites and 2.4% of mobile sites (Web Almanac SEO).
30. The nofollow directive appears on 2.4% of desktop sites (Web Almanac SEO).
31. Invalid elements in the head were found on 10.1% of desktop pages (Web Almanac SEO).
32. The img element was the most common invalid head element, on 27.2% of desktop pages (Web Almanac SEO).
33. 68% of desktop pages have a canonical tag, against 67% on mobile (Web Almanac SEO).
34. 7% of desktop pages and 9% of mobile pages are canonicalized to a different URL (Web Almanac SEO).
35. Rendered canonical tags are present on 66.01% of desktop pages (Web Almanac SEO).
36. HTTP header canonicals are used on just 0.58% of desktop pages (Web Almanac SEO).
37. Canonical mismatches affect 0.72% of desktop pages (Web Almanac SEO).
38. JavaScript-rendered canonical changes occur on 3.02% of mobile pages (Web Almanac SEO).
39. Yoast SEO is detected on 15.96% of desktop home pages (Web Almanac SEO).
40. RankMath SEO is detected on 3.56% of desktop home pages (Web Almanac SEO).
41. All in One SEO is detected on 2.96% of desktop home pages (Web Almanac SEO).

What it means. Canonical tags are a “strong hint,” not a command, and most sites place them in the raw HTML where they survive rendering. The 3.02% of mobile pages whose canonical changes after JavaScript runs is the figure to audit if you run a JavaScript-heavy site. The invalid head element number matters because a bad element can end the head early, which pushes titles, canonicals and robots tags into the body where they stop working. SEO plugins set much of this by default, with Yoast alone detected on 15.96% of desktop home pages.

How good is page experience?

HTTPS, the viewport tag and legible fonts are essentially solved. Speed on mobile is not.

42. HTTPS is used on 91.7% of desktop pages and 91.5% of mobile pages (Web Almanac SEO).
43. The viewport meta tag appears on 95.2% of mobile pages (Web Almanac SEO).
44. Only about 1% of pages send a Vary: User-Agent header (Web Almanac SEO).
45. 92% of mobile pages passed Lighthouse legible font size (Web Almanac SEO).
46. Only 48% of mobile pages had a good overall Core Web Vitals experience in June 2025 (Web Almanac SEO).
47. Good mobile LCP was reached by 62% of pages in June 2025 (Web Almanac SEO).
48. Good mobile INP was reached by 77% of pages in June 2025 (Web Almanac SEO).
49. Good mobile CLS was reached by 81% of pages in June 2025 (Web Almanac SEO).
50. On desktop, 56% of pages had a good overall Core Web Vitals experience in June 2025 (Web Almanac SEO).
51. Good desktop LCP was reached by 74% of pages in June 2025 (Web Almanac SEO).
52. Good desktop CLS was reached by 72% of pages in June 2025 (Web Almanac SEO).
53. Good desktop INP was reached by 97% of pages (Web Almanac SEO).
54. Google says LCP should occur within 2.5 seconds of page load start (web.dev).
55. Google says INP should be 200 milliseconds or less (web.dev).
56. Google says CLS should be 0.1 or less (web.dev).
57. Google says to judge Core Web Vitals at the 75th percentile of page loads (web.dev).
58. 91.5% of desktop pages declare no iframe loading attribute (Web Almanac SEO).

Source: HTTP Archive, Web Almanac 2025 SEO chapter, figures 5.21 and 5.22. Thresholds from Google web.dev. Checked October 28, 2026.

What it means. On mobile, loading speed (LCP) is the weakest of the three metrics and the one that holds the overall score under 50%. Desktop looks very different, with INP almost solved at 97%. If you have one performance project this quarter, it is mobile LCP. The thresholds are unchanged: 2.5 seconds for LCP, 200 milliseconds for INP and 0.1 for CLS, judged at the 75th percentile of page loads.

What do titles, descriptions, headings and images look like?

On-page markup is stable. The median title is longer than most advice suggests, and the median meta description is now more than double its 2022 length.

59. Title tags are present on 98.6% of desktop pages (Web Almanac SEO).
60. Meta descriptions are present on 67.7% of desktop pages (Web Almanac SEO).
61. The median title tag is 12 words, unchanged from 2024 (Web Almanac SEO).
62. The median title is 77 characters on desktop and 79 on mobile (Web Almanac SEO).
63. At the 90th percentile, titles reach 150 characters on desktop (Web Almanac SEO).
64. The median meta description is 274 characters, double the 2022 median (Web Almanac SEO).
65. At the 90th percentile, meta descriptions reach 537 characters on desktop (Web Almanac SEO).
66. H1 elements are present on 71% of desktop pages (Web Almanac SEO).
67. H2 elements are present on 72% of desktop pages (Web Almanac SEO).
68. Non-empty H1 elements are present on 66% of pages (Web Almanac SEO).
69. The median home page has 20 images on desktop and 19 on mobile (Web Almanac SEO).
70. At the median, 60% of images on mobile pages have alt attributes, up from 58% in 2024 (Web Almanac SEO).
71. At the median, 15% of images on mobile pages have blank alt values (Web Almanac SEO).
72. Video elements are on 6.40% of desktop pages and 5.68% of mobile pages (Web Almanac SEO).
73. Only 0.9% of pages used VideoObject markup in 2024 (Web Almanac SEO).
74. The median mobile home page has 378 rendered words, up from 364 in 2024 (Web Almanac SEO).
75. The median desktop home page has 413 rendered words, up from 400 in 2024 (Web Almanac SEO).
76. The median inner page has 339 rendered words on desktop and 323 on mobile (Web Almanac SEO).
77. The median home page has 340 raw HTML words on desktop and 316 on mobile (Web Almanac SEO).

Source: HTTP Archive, Web Almanac 2025 SEO chapter, figures 5.29 and 5.31. Checked October 28, 2026.

Source: HTTP Archive, Web Almanac 2025 SEO chapter, figures 5.44 and 5.45. Checked October 28, 2026.

What it means. Publishers are not writing to a 60-character title limit, probably because search engines rewrite titles anyway. The median inner page is short: 339 rendered words on desktop. If you publish long, well-structured pages, you are already above the median on depth. Alt text is the clearest gap: at the median, only 60% of images on mobile pages have alt attributes.

How do sites handle links?

Internal links dominate, outbound links are rare, and fine-grained link attributes have not caught on.

78. Home pages pass the descriptive link text audit on 84.64% of desktop sites (Web Almanac SEO).
79. Inner pages pass the descriptive link text audit on 92.42% of desktop sites (Web Almanac SEO).
80. The median desktop page has 43 internal links and the median mobile page has 39 (Web Almanac SEO).
81. The median page has just six outbound links to other sites (Web Almanac SEO).
82. noopener appears on 40.9% of pages and noreferrer on 25% (Web Almanac SEO).
83. nofollow appears on 32.3% of pages (Web Almanac SEO).
84. The sponsored and ugc attributes are each stuck at 0.5% of pages (Web Almanac SEO).

What it means. Only 0.5% of pages use sponsored or ugc, while nofollow sits on 32.3%. Most publishers still use the old binary choice. Descriptive anchor text is mostly fine on inner pages (92.42%) and weaker on home pages (84.64% on desktop), where generic calls to action dominate.

How much structured data is out there?

Half of home pages now ship structured data, almost all in the raw HTML.

85. Structured data reached 50% of home pages on desktop and mobile, up from 48% and 49% in 2024 (Web Almanac SEO).
86. JSON-LD is on 43% of desktop home pages and 43% of mobile home pages (Web Almanac SEO).
87. Microdata is on 17% of desktop home pages and 16% of mobile home pages (Web Almanac SEO).
88. Only 2% of crawls have structured data added via JavaScript (Web Almanac SEO).
89. The WebSite type is the most common, on 37% of desktop home pages (Web Almanac SEO).
90. Organization markup is on 26.74% of desktop home pages (Web Almanac SEO).
91. ListItem is the most common inner page type, on 29% of desktop inner pages (Web Almanac SEO).
92. Google recommends JSON-LD as the format that is easiest to implement and maintain (Google Search Central).
93. rel=amphtml appears on 0.9% of mobile pages, so AMP is a fringe technology (Web Almanac SEO).

Source: HTTP Archive, Web Almanac 2025 SEO chapter, figure 5.50. Checked October 28, 2026.

What it means. JSON-LD is both the most common format and Google’s recommendation. The chapter also notes that Microsoft has confirmed Bing uses schema.org markup to help its models tell articles from products, reviews and FAQs. That is a statement of the Almanac authors about Microsoft, so we treat it as context, not a measured stat. AMP, once a mobile search requirement, is now a fringe technology.

How common is hreflang?

International targeting doubled in a year, but it concentrates on a few languages.

94. hreflang appears on about 20.3% of desktop pages (Web Almanac SEO).
95. That roughly doubles 2024, when about 9 to 10% of pages carried hreflang (Web Almanac SEO).
96. English (en) leads hreflang values at 7.4% of desktop home pages (Web Almanac SEO).
97. French is next at 3.1% of desktop home pages (Web Almanac SEO).

What it means. About one in five pages now carries hreflang, which tracks the broader shift toward multilingual sites. The tail is concentrated: English, x-default, French, German and Spanish account for most values. The Almanac says HTTP header hreflang is now nearly gone.

What to do with this data

• Audit the floor first. Check robots.txt returns 200, your canonical tags sit in the raw HTML, and your head contains only valid elements. These are the checks the median site passes and a problem site fails.
• Decide your AI crawler policy on purpose. Name the bots you want in or out, such as gptbot, claudebot, google-extended and perplexitybot, instead of relying on the wildcard. See our AI crawler statistics.
• Do not expect llms.txt to move rankings. Adoption is about 2%, plugins create much of it, and the Almanac says no Google service uses it. Ship it only if it costs you nothing.
• Fix mobile LCP before anything else. It is the weakest Core Web Vital, and the reason fewer than half of mobile pages pass.
• Ship JSON-LD in the raw HTML. Do not inject schema with JavaScript. Only 2% of sites do, and it adds failure modes.
• Measure the outcome, not just the inputs. Technical SEO makes you eligible to be cited. Mentionova tracks whether ChatGPT, Perplexity, Claude, Gemini and Google AI actually cite you and which Reddit threads they pull from. It is #1 in our best AI visibility tools and LLM brand monitoring tools roundups, and its agents draft the content briefs that close citation gaps, as covered in our best AI SEO agents.
• Pick a site platform that handles the basics. If you are building a new site, our best AI website builders and website builder statistics cover which platforms ship these defaults.

Frequently asked questions

How many websites does the Web Almanac analyze?

The 2025 Web Almanac dataset has 16,213,084 websites, of which 15,426,398 are mobile and 12,155,374 are desktop. Most sites appear in both subsets. Metrics come from the July 2025 HTTP Archive crawl unless a chapter says otherwise.

Do most websites have a robots.txt file?

Yes. A valid 200 response came back for 84.9% of sites on both desktop and mobile, up from 84% in 2024. About 13% return a 404, which means no restrictions apply.

How many sites block AI crawlers in robots.txt?

The Almanac reports how many files name each bot, not whether they allow or block it. gptbot is named in 4.5% of desktop files, claudebot in 3.6%, ccbot in 3.5%, google-extended in 3.4%, amazonbot in 3.3% and perplexitybot in 2.8%.

What share of websites use llms.txt?

About 2% of sites. The Almanac found valid files on 324,184 mobile sites. It also reports that 39.6% of the files come from All in One SEO and 3.6% from Yoast SEO, so many owners may not have chosen it on purpose.

Does Google use llms.txt?

The Almanac says Google has often stated it does not use llms.txt and that no Google service currently does. That is a claim about Google’s position as of the chapter, and it can change.

What percentage of pages pass Core Web Vitals?

In June 2025, 48% of mobile pages and 56% of desktop pages had a good overall Core Web Vitals experience. Mobile LCP was the weakest metric at 62% good, and desktop INP was the strongest at 97% good.

How long should a title tag be?

The median title is 77 characters on desktop and 79 on mobile, and the 90th percentile reaches 150 characters. The Almanac says this is well above the often-cited 50 to 60 character range, likely because search engines rewrite titles anyway.

How many sites use structured data?

Structured data reached 50% of home pages in 2025, up from 48% on desktop and 49% on mobile in 2024. JSON-LD is on 43% of home pages, and Google recommends it.

Is AMP still used?

Barely. The Almanac measured rel=amphtml on 0.9% of mobile pages and described AMP as a fringe technology. Core Web Vitals replaced the performance role AMP once played.

How common is hreflang?

About one in five pages. The Almanac measured 20.3% on desktop, roughly double the 9 to 10% reported a year earlier. English leads the language values at 7.4% of desktop home pages.

Why are these numbers from 2025 and not 2026?

The Web Almanac publishes once a year from a mid-year crawl. The SEO chapter uses the July 2025 crawl, so it is the latest edition available when we fetched it. We show the data year in every ledger row, and we did not blend in newer figures from other sources.

How we compiled these statistics

We fetched 5 primary source pages on October 28, 2026 and recorded 97 data points in docs/content/stats-ledgers/technical-seo-statistics.json. Every numbered figure above maps to a ledger row with a verbatim quote from the page we fetched. The data comes from HTTP Archive’s Web Almanac 2025 (SEO chapter and methodology page), plus Google’s web.dev and Search Central documentation for definitions and thresholds. We did not use traffic estimators or third-party SEO research. Where the chapter’s figure descriptions and prose disagree, or where a chart label looked inconsistent (for example, the image loading attribute split and the indexifembedded share), we left the figure out. We report the chapter’s rounding as printed, and we did not compute any rates ourselves. Related reading: AI search statistics, Google AI Overviews statistics and landing page statistics.

Sources

• HTTP Archive: Web Almanac 2025 methodology
• HTTP Archive: Web Almanac 2025, SEO chapter
• Google Search Central: How Google interprets the robots.txt specification
• Google web.dev: Web Vitals
• Google Search Central: Intro to structured data markup in Google Search