An llms.txt file is a short Markdown file at your site root that gives AI tools a summary of your website and links to its most useful pages. It is an open proposal, not a standard. Google Search ignores it, so it will not help rankings, but it is cheap to add and can help some AI tools.
Business owners keep hearing that their website "needs an llms.txt file for AI". Some agencies now sell it as a separate service. The truth is less exciting and more useful: the file is easy to create, it does no harm, and it does very little on its own. What decides whether ChatGPT, Gemini or Google's AI Overviews mention your business is still the quality and clarity of the pages themselves.
This guide explains what the file is, who reads it, how it differs from robots.txt, and how to write one for a small business website in under an hour.
What is an llms.txt file?
The idea was proposed by Jeremy Howard in September 2024 and is documented at llmstxt.org. The reasoning is simple. Large language models (LLMs) have a limited context window, and a normal web page is full of menus, scripts, cookie banners and ads. A plain Markdown file that says "here is who we are and here are the pages that matter" is much easier for an AI tool to read.
The proposal describes a file with a fixed order:
- An H1 with the name of the site or project. This is the only required part.
- A blockquote with a short summary.
- Optional paragraphs or lists with more detail.
- One or more H2 sections with lists of links, each written as
[name](url)with an optional note after a colon. - By convention, an H2 called "Optional" for secondary links that an AI tool can skip when it needs less context.
The proposal also suggests offering a clean Markdown version of important pages (for example page.md). The file usually lives at /llms.txt, though the current version of the proposal also allows one inside a folder such as /docs/llms.txt.
Does Google use llms.txt?
No. Google's guide to optimising for generative AI features in Search says you do not need to create new machine-readable files, AI text files, markup or Markdown to appear in Google Search, "as Google Search itself doesn't use them". The same page says it is fine to keep llms.txt files for other services, and that doing so "will neither harm nor help" your visibility in Google Search.
So if your main goal is Google, including AI Overviews and AI Mode, spend your time on the content and technical basics we cover in our guide on getting cited in Google AI Overviews.
Do ChatGPT, Claude and other AI tools read it?
There is no public commitment from the major AI companies that their crawlers read llms.txt files on third-party websites. Their published crawler documentation is built around robots.txt instead. Where llms.txt is clearly useful today is in coding assistants and AI agents that are pointed at a specific site, for example a developer asking an AI tool to read a product's documentation. Many software companies publish llms.txt files for their developer docs for exactly this reason.
For a typical Indian service business, that means: add the file if you like, but do not expect it to change your leads.
How is llms.txt different from robots.txt and sitemap.xml?
| File | Purpose | Who reads it | Effect |
|---|---|---|---|
| robots.txt | Tells crawlers what they may and may not fetch | Search engines and most AI crawlers | Controls access. This is the file that actually matters for AI bots. |
| sitemap.xml | Lists every URL you want indexed | Search engines | Helps discovery and indexing |
| llms.txt | A curated, human-readable summary with key links | Some AI tools and agents, on request | Optional context. No ranking effect in Google Search. |
Which robots.txt rules matter for AI?
If you care about how AI systems use your website, robots.txt is where the real decisions are made. Each company documents its own user agents:
- OpenAI:
OAI-SearchBotis used to show websites in ChatGPT's search features, whileGPTBotcollects content that may be used to train its models. OpenAI's crawler documentation says a site that blocks OAI-SearchBot will not appear in ChatGPT search answers, apart from navigational links. - Anthropic: its help article lists
ClaudeBot(training),Claude-User(fetching a page when a user asks) andClaude-SearchBot(search). Blocking the last two may reduce your visibility in Claude's answers. - Google:
Google-Extendedcontrols whether content may be used to train Gemini models and for some grounding uses. Google's crawler documentation states it does not affect inclusion or ranking in Google Search.
A sensible default for most businesses that want to be found is to allow search-type bots and make a separate, deliberate choice about training bots. Blocking everything "for safety" often removes you from the AI answers your customers are reading.
How do you create an llms.txt file?
- Pick your 10 to 20 most important pages. Usually the home page, main service or product pages, pricing, contact, and two or three of your best guides.
- Write a one-line summary of your business. Who you are, what you sell, where you work. No slogans.
- Group the links under clear H2 headings such as "Services", "Products", "Guides" and "Company".
- Add a short note to each link saying what the reader will find there.
- Save it as plain text named
llms.txtand upload it to the root of your domain. - Keep it in sync. When you remove or rename a page, update the file. A file full of broken links is worse than no file.
Here is a simple example for a fictional interior design studio:
# Sample Interiors
> Sample Interiors designs homes and offices in Ahmedabad and Gandhinagar, Gujarat. We handle design, 3D views, material selection and site execution.
## Services
- [Home interiors](https://example.com/home-interiors): 2BHK to villa projects, process and timelines
- [Office interiors](https://example.com/office-interiors): workspaces for 10 to 200 people
## Company
- [Projects](https://example.com/projects): photos of completed work
- [Contact](https://example.com/contact): phone, WhatsApp and enquiry form
## Optional
- [Blog](https://example.com/blog): design guides and material comparisons
What should you not put in llms.txt?
- Claims you cannot back up. "Best in India" or invented numbers do not become true because they are in a text file, and AI tools that cross-check sources may ignore them.
- Private or internal URLs. The file is public. Do not list admin panels, staging sites or customer portals.
- Different facts from your website. Prices, phone numbers and service lists must match your pages exactly.
- Keyword stuffing. The file is meant for machines to understand you, not to repeat search terms.
When is llms.txt worth the effort?
It is most useful when:
- you run a software product or API and developers use AI coding tools to read your documentation;
- you have a large site and want to point AI agents to the handful of pages that matter;
- you already publish clean Markdown versions of your pages and want an index for them.
For a small business website, it is a ten-minute task to do once the real work is done: clear service pages, honest FAQs, correct structured data, a complete Google Business Profile and sensible robots.txt rules.
FAQs
Is llms.txt an official web standard?
No. It is a public proposal published at llmstxt.org. It has not been adopted by a standards body, and the major search engines have not said they use it for ranking.
Will adding llms.txt improve my Google rankings?
No. Google says Google Search ignores llms.txt and similar files, so adding one will neither help nor harm your rankings or your appearance in AI Overviews.
Can llms.txt block AI companies from using my content?
No. llms.txt only describes your site. To control crawling and AI training, use robots.txt rules for each crawler, such as GPTBot, ClaudeBot or Google-Extended.
Where should I upload the file?
Usually at the root of your domain, so it opens at yourdomain.com/llms.txt. The current proposal also allows a file inside a folder, which then covers only the pages in that folder.
Do I need a developer to create llms.txt?
Not really. It is a plain text file in Markdown. Anyone who can upload a file to the website can add it. On sites with many pages, a developer can generate it automatically so it never goes out of date.
Want your website ready for AI search and AI assistants?
We fix the things that actually decide AI visibility: clear page content, technical SEO, structured data and crawler rules. See our SEO services and website revamping. If you want your own AI assistant that answers visitors from your website content, look at our AI chatbot development service, or contact our team for a quick review.



