BrowserTools
Home / AI / llms.txt & AI Robots.txt Generator

llms.txt & AI Robots.txt Generator

Create an llms.txt file that tells AI assistants what your site is about, and robots.txt rules to allow or block AI crawlers.

Maintained by BrowserTools · Last updated 2026-09-29

Loading llms.txt & AI Robots.txt Generator… If nothing happens, please enable JavaScript.

AI assistants now read the web in two different ways, and site owners need a say in both. The first is llms.txt, a proposal published at llmstxt.org for a plain Markdown file at the root of a site that tells a language model what the site is and where its most useful pages live. The format is deliberately small: an H1 with the site name, a blockquote with a short summary, optional paragraphs of context, and then H2 sections that each hold a list of links in the form [Title](url): description. A section called Optional marks links an agent can skip when its context window is tight. This generator builds that file from a simple form, with sections and links you can add, remove and reorder, and shows the finished Markdown as you type.

Frequently asked questions

What is llms.txt?
It is a proposed standard, published at llmstxt.org in 2024, for a Markdown file served at /llms.txt that gives language models a curated overview of a site. It starts with the site name as an H1 heading and a one-paragraph summary in a blockquote, followed by sections of links to the pages that matter most. The idea is that a model reading a clean list of well-described links wastes far less context than one parsing navigation menus, cookie banners and scripts.
Do ChatGPT, Claude or Google actually read llms.txt?
None of the big AI companies has committed to reading llms.txt when answering questions, and Google representatives have said Search does not use it. Where it clearly helps today is with coding assistants, documentation tools and agents that are pointed at a site and look for the file, which is why many developer documentation sites publish one. It costs little to add, but it is a convenience for tools that choose to use it, not a ranking factor.
What goes in the Optional section?
Links that are useful but not essential, such as changelogs, press pages, old versions of the docs or background reading. The proposal says an agent that needs a shorter context can skip the Optional section entirely, so put your core pages in the normal sections and move the rest there. This tool always writes the Optional section last, which is where tools expect to find it.
What is llms-full.txt and can this tool create it?
llms-full.txt is a community convention rather than part of the proposal: a single file that contains the full text of every page listed in llms.txt, so a tool can load all of your documentation in one request. It has to be built from the actual page contents, usually by your documentation generator at build time. Because this page never fetches your site, it cannot produce that file, but the guide below the output explains how to assemble one.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
They are three separate OpenAI agents with different jobs. GPTBot collects content that may be used to train models, OAI-SearchBot indexes pages so they can appear and be linked in ChatGPT search, and ChatGPT-User visits a page only when a person in ChatGPT asks for it. Anthropic follows the same pattern with ClaudeBot, Claude-SearchBot and Claude-User, and Perplexity with PerplexityBot and Perplexity-User. That is why blocking training does not have to mean disappearing from AI answers.
Does blocking Google-Extended remove my site from Google?
No. Google-Extended is a control token, not a crawler: pages are still fetched by the normal Googlebot, and the token only tells Google whether that content may be used to train and ground Gemini models. Google states it has no effect on inclusion or ranking in Google Search. Applebot-Extended works the same way for Apple, where blocking it keeps your pages in Siri and Spotlight but out of Apple model training.
Will robots.txt really stop AI crawlers?
Only the ones that choose to obey it. robots.txt is a voluntary protocol with no enforcement, and the major vendors say their automated crawlers respect it. Fetchers that act on a user request, such as Perplexity-User and ChatGPT-User, are documented as not always following it, and some crawlers such as Bytespider are widely reported to ignore it. If you need a hard block, add rules in your web server, CDN or firewall as well.
What happens to my existing robots.txt rules?
They are kept as they are. The tool removes only groups that name the AI crawlers you are managing, so there is a single clear rule for each of them, collects your Sitemap lines at the end of the file and appends the AI block. If you enable explicit Allow groups, the rules from your User-agent: * group are copied into them, because a crawler that finds a group with its own name ignores the * group completely.
Is anything I paste sent to a server?
No. The llms.txt builder, the sitemap parser and the robots.txt merger all run as JavaScript in your browser, and the page makes no network request while you work. That makes it safe to paste sitemaps from staging environments or robots.txt files that mention private paths.

About llms.txt & AI Robots.txt Generator

If your site already has a sitemap.xml, you do not have to type the links by hand. Paste the sitemap contents, or any list of URLs, into the import box and the tool pulls out every address, guesses a readable title from the path and groups the pages into sections by their first path segment, so /docs/ pages land under Docs and /blog/ pages under Blog. The parsing happens in your browser and nothing is fetched, which means it works for staging sites, intranets and sitemaps you have not published yet. Relative links are resolved against the site URL you enter, duplicates are flagged, and a notes panel points out anything that would make the file harder for a model to use, such as missing titles or headings inside the details block.

The second tab handles the other side: which AI crawlers may visit at all. It lists the robots.txt tokens that the major vendors document, grouped by what they do, so you can see the difference between model training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot), AI search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and fetches a user asked for (ChatGPT-User, Claude-User, Perplexity-User). One click applies a common policy such as blocking training while staying visible in AI search. Paste your current robots.txt and your own rules are kept, old rules for the same bots are replaced, your Sitemap lines move to the end, and the AI block is appended. Copy or download both files and upload them to the root of your site.

A 1994 mailing list rule now decides who trains AI

robots.txt was proposed by Martijn Koster in 1994 after a badly behaved crawler overloaded his server. It was agreed on a mailing list rather than by a standards body, and for 28 years it remained an informal convention that every major search engine nonetheless followed. It only became an official internet standard in September 2022, as RFC 9309, largely thanks to a push from Google.

The arrival of large language models gave the old file a new job. When OpenAI published the GPTBot user agent in August 2023, thousands of news sites added a block for it within weeks, and Google followed with the Google-Extended token a month later so publishers could refuse AI training without leaving Google Search. Studies of the most visited websites since then show a steady rise in AI-specific rules, to the point where robots.txt has become one of the main places where the argument over training data is actually settled.

llms.txt comes from the opposite direction. Jeremy Howard of Answer.AI proposed it in September 2024, not to keep models out but to give them a tidier way in, inspired by how much of a model's context is wasted on HTML clutter. Its format is plain Markdown on purpose, so it can be read equally well by a person, a parser or a language model.

Related tools

URL Parser

Split a web address into its parts: domain, path, query parameters and more.

XML Validator

Check whether XML is valid, see where it breaks, and format it neatly.

JSON Validator

Check whether JSON is valid, see exactly where it breaks, and format it neatly.

Token Counter

See how many tokens your text uses in ChatGPT and other AI models, plus an estimate for Claude. Nothing leaves your browser.
Support