Skip to content
Digital Marketing

What Is an llms.txt File and Does Your Website Need One

Chetan MakwanaCEO & Founder, Dreams TechnologyPublished 8 min read
What Is an llms.txt File and Does Your Website Need One
Table of contents

An llms.txt file is a short Markdown file at your site root that gives AI tools a summary of your website and links to its most useful pages. It is an open proposal, not a standard. Google Search ignores it, so it will not help rankings, but it is cheap to add and can help some AI tools.

Business owners keep hearing that their website "needs an llms.txt file for AI". Some agencies now sell it as a separate service. The truth is less exciting and more useful: the file is easy to create, it does no harm, and it does very little on its own. What decides whether ChatGPT, Gemini or Google's AI Overviews mention your business is still the quality and clarity of the pages themselves.

This guide explains what the file is, who reads it, how it differs from robots.txt, and how to write one for a small business website in under an hour.

What is an llms.txt file?

The idea was proposed by Jeremy Howard in September 2024 and is documented at llmstxt.org. The reasoning is simple. Large language models (LLMs) have a limited context window, and a normal web page is full of menus, scripts, cookie banners and ads. A plain Markdown file that says "here is who we are and here are the pages that matter" is much easier for an AI tool to read.

The proposal describes a file with a fixed order:

  1. An H1 with the name of the site or project. This is the only required part.
  2. A blockquote with a short summary.
  3. Optional paragraphs or lists with more detail.
  4. One or more H2 sections with lists of links, each written as [name](url) with an optional note after a colon.
  5. By convention, an H2 called "Optional" for secondary links that an AI tool can skip when it needs less context.

The proposal also suggests offering a clean Markdown version of important pages (for example page.md). The file usually lives at /llms.txt, though the current version of the proposal also allows one inside a folder such as /docs/llms.txt.

Does Google use llms.txt?

No. Google's guide to optimising for generative AI features in Search says you do not need to create new machine-readable files, AI text files, markup or Markdown to appear in Google Search, "as Google Search itself doesn't use them". The same page says it is fine to keep llms.txt files for other services, and that doing so "will neither harm nor help" your visibility in Google Search.

So if your main goal is Google, including AI Overviews and AI Mode, spend your time on the content and technical basics we cover in our guide on getting cited in Google AI Overviews.

Do ChatGPT, Claude and other AI tools read it?

There is no public commitment from the major AI companies that their crawlers read llms.txt files on third-party websites. Their published crawler documentation is built around robots.txt instead. Where llms.txt is clearly useful today is in coding assistants and AI agents that are pointed at a specific site, for example a developer asking an AI tool to read a product's documentation. Many software companies publish llms.txt files for their developer docs for exactly this reason.

For a typical Indian service business, that means: add the file if you like, but do not expect it to change your leads.

How is llms.txt different from robots.txt and sitemap.xml?

FilePurposeWho reads itEffect
robots.txtTells crawlers what they may and may not fetchSearch engines and most AI crawlersControls access. This is the file that actually matters for AI bots.
sitemap.xmlLists every URL you want indexedSearch enginesHelps discovery and indexing
llms.txtA curated, human-readable summary with key linksSome AI tools and agents, on requestOptional context. No ranking effect in Google Search.

Which robots.txt rules matter for AI?

If you care about how AI systems use your website, robots.txt is where the real decisions are made. Each company documents its own user agents:

  • OpenAI: OAI-SearchBot is used to show websites in ChatGPT's search features, while GPTBot collects content that may be used to train its models. OpenAI's crawler documentation says a site that blocks OAI-SearchBot will not appear in ChatGPT search answers, apart from navigational links.
  • Anthropic: its help article lists ClaudeBot (training), Claude-User (fetching a page when a user asks) and Claude-SearchBot (search). Blocking the last two may reduce your visibility in Claude's answers.
  • Google: Google-Extended controls whether content may be used to train Gemini models and for some grounding uses. Google's crawler documentation states it does not affect inclusion or ranking in Google Search.

A sensible default for most businesses that want to be found is to allow search-type bots and make a separate, deliberate choice about training bots. Blocking everything "for safety" often removes you from the AI answers your customers are reading.

How do you create an llms.txt file?

  1. Pick your 10 to 20 most important pages. Usually the home page, main service or product pages, pricing, contact, and two or three of your best guides.
  2. Write a one-line summary of your business. Who you are, what you sell, where you work. No slogans.
  3. Group the links under clear H2 headings such as "Services", "Products", "Guides" and "Company".
  4. Add a short note to each link saying what the reader will find there.
  5. Save it as plain text named llms.txt and upload it to the root of your domain.
  6. Keep it in sync. When you remove or rename a page, update the file. A file full of broken links is worse than no file.

Here is a simple example for a fictional interior design studio:

# Sample Interiors

> Sample Interiors designs homes and offices in Ahmedabad and Gandhinagar, Gujarat. We handle design, 3D views, material selection and site execution.

## Services
- [Home interiors](https://example.com/home-interiors): 2BHK to villa projects, process and timelines
- [Office interiors](https://example.com/office-interiors): workspaces for 10 to 200 people

## Company
- [Projects](https://example.com/projects): photos of completed work
- [Contact](https://example.com/contact): phone, WhatsApp and enquiry form

## Optional
- [Blog](https://example.com/blog): design guides and material comparisons

What should you not put in llms.txt?

  • Claims you cannot back up. "Best in India" or invented numbers do not become true because they are in a text file, and AI tools that cross-check sources may ignore them.
  • Private or internal URLs. The file is public. Do not list admin panels, staging sites or customer portals.
  • Different facts from your website. Prices, phone numbers and service lists must match your pages exactly.
  • Keyword stuffing. The file is meant for machines to understand you, not to repeat search terms.

When is llms.txt worth the effort?

It is most useful when:

  • you run a software product or API and developers use AI coding tools to read your documentation;
  • you have a large site and want to point AI agents to the handful of pages that matter;
  • you already publish clean Markdown versions of your pages and want an index for them.

For a small business website, it is a ten-minute task to do once the real work is done: clear service pages, honest FAQs, correct structured data, a complete Google Business Profile and sensible robots.txt rules.

FAQs

Is llms.txt an official web standard?

No. It is a public proposal published at llmstxt.org. It has not been adopted by a standards body, and the major search engines have not said they use it for ranking.

Will adding llms.txt improve my Google rankings?

No. Google says Google Search ignores llms.txt and similar files, so adding one will neither help nor harm your rankings or your appearance in AI Overviews.

Can llms.txt block AI companies from using my content?

No. llms.txt only describes your site. To control crawling and AI training, use robots.txt rules for each crawler, such as GPTBot, ClaudeBot or Google-Extended.

Where should I upload the file?

Usually at the root of your domain, so it opens at yourdomain.com/llms.txt. The current proposal also allows a file inside a folder, which then covers only the pages in that folder.

Do I need a developer to create llms.txt?

Not really. It is a plain text file in Markdown. Anyone who can upload a file to the website can add it. On sites with many pages, a developer can generate it automatically so it never goes out of date.

Want your website ready for AI search and AI assistants?

We fix the things that actually decide AI visibility: clear page content, technical SEO, structured data and crawler rules. See our SEO services and website revamping. If you want your own AI assistant that answers visitors from your website content, look at our AI chatbot development service, or contact our team for a quick review.

Found this useful? Share it

Written by

Chetan Makwana

CEO & Founder, Dreams Technology

Building websites, apps and AI solutions for businesses across India since 2014.

View all →

Turn ideas into results

Tell us about your project and get a free consultation with a fixed-price quote within one working day.

or call +91 84607 65785

Book a Free Consultation with Us!

Share a few details — we'll call you back with ideas and a quote.

Wait! Don't miss this

Claim your free gift before you go

Worth₹10,000FREE for you

Choose one

Phone or email: one is enough.

No spam. We only contact you about your gift.

Get a Free Quote