Free llms.txt generator
What is llms.txt?
llms.txt is a plain-text file at a site’s root — /llms.txt — that tells AI assistants what the site contains and where to find it. It was proposed in 2024 as a companion to robots.txt: where robots.txt says what a crawler may not take, llms.txt says what is worth reading and what each part is. It is Markdown, so a model reads it without parsing anything.
- An H1: the name of the site.
- A blockquote: one or two sentences on what the site is.
- H2 sections: Docs, API, Blog, Pricing — each a list of links, ideally with a line on what each page covers.
llms.txt, llms-full.txt, sitemap.xml and robots.txt
Four files at a site’s root, each answering a different question. None replaces another.
| File | Who reads it | What it says |
|---|---|---|
| robots.txt | Crawlers | What you may not fetch, and how fast |
| sitemap.xml | Search engines | Every URL that exists, and when each changed |
| llms.txt | AI assistants | What the site is about and which pages matter, by section |
| llms-full.txt | AI assistants | The content itself — every usable page’s text in one file |
The map here, the descriptions from a crawl
This tool builds the file from what your site already declares: its robots.txt, the sitemaps named there, and their URLs, grouped into sections by the first part of each path. That gives you the structure and the links in seconds, free, without an account.
What a sitemap cannot give you is the line under each link that says what the page covers — that has to be read off the page itself. A crawl fills those in from each page’s own title and description, writes llms-full.txt alongside, and rebuilds both on a schedule. The generated file says which of the two it is in its own header, so nobody publishes the thinner one thinking it is the other.
Where to put it, and keeping it true
- Serve it at the root: https://yoursite.com/llms.txt, as text/plain or text/markdown.
- Read it once yourself: the sections come from your URL structure, so a site whose paths do not match its meaning will produce a map that does not either. Edit it — it is Markdown.
- Rebuild it: a file generated once is wrong within a month: pages ship, pages move, pages go. A weekly crawl rebuilds it from what the site says now.
Frequently asked questions
What is llms.txt used for?
It gives AI assistants a reliable map of a website, so an assistant answering about your product reads your documentation rather than guessing from whatever it happened to crawl. It is a proposal, not a standard: support is growing rather than universal, and publishing one costs nothing.
What is the difference between llms.txt and llms-full.txt?
llms.txt is the map — the usable pages as links with their descriptions, grouped by section. llms-full.txt is the corpus — every usable page’s Markdown in one file, each under its title and source URL, for an assistant that reads a whole site at once.
Does llms.txt replace robots.txt or my sitemap?
No. robots.txt controls access, the sitemap lists every URL for search engines, and llms.txt curates what matters for a reader that cannot see your navigation. Publish all three.
Where do I put the llms.txt file?
At the root of the domain: https://yoursite.com/llms.txt, served as plain text or Markdown.
Why does this need my sitemap?
Because it is free and does not crawl you. The file is built from the sitemaps your robots.txt declares. A site with no sitemap needs a crawl to be mapped at all, which needs an account.
Give assistants a map of your site
Generate it here from your sitemaps, or have a crawl write the full one every week.