We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read

Ahrefs research into 137,000 websites found that 97% of `llms.txt` files are never accessed. This file, designed to allow website owners to control how large language models (LLMs) interact with their content, was introduced by Google in late 2023. The analysis, which utilized Ahrefs' web and bot analytics tools to examine server logs and live traffic, revealed that very few bots, including those from major LLM providers, are currently checking for or adhering to these directives.

This finding suggests that, for now, the `llms.txt` file has minimal practical impact on how AI models crawl and use website content. SEO professionals and site owners might consider delaying the implementation or extensive management of this file, as current bot behavior indicates it is largely being ignored. Resources could be better allocated to established SEO practices that demonstrably influence search engine visibility and content indexing.