BrowserTools
Home / Validators / Robots.txt Tester

Robots.txt Tester

Check whether a crawler like Googlebot or GPTBot may visit a page, and which line of robots.txt decides it.

Maintained by BrowserTools · Last updated 2026-09-29

Loading Robots.txt Tester… If nothing happens, please enable JavaScript.

A robots.txt file looks simple, but the way crawlers read it has a few rules that surprise almost everyone. A crawler does not combine its own group with the User-agent: * group: once a group names it, the general rules stop applying. Between Allow and Disallow, the order of the lines does not matter; the rule with the longest matching path wins, and when an Allow and a Disallow are the same length, Allow wins. Paths are case-sensitive, a * matches any run of characters, a $ at the end anchors the match to the end of the URL, and an empty Disallow blocks nothing. Since 2022 these rules are written down in RFC 9309, the Robots Exclusion Protocol standard that Google, Bing and most other large crawlers follow.

Frequently asked questions

If a crawler has its own group, does it still obey User-agent: *?
No. A crawler uses only the most specific group that names it. If your file has a group for Googlebot, Googlebot ignores everything in the User-agent: * group, so any rule you want it to follow must be repeated in its own group. If several groups name the same crawler, their rules are merged. The tester shows which group was chosen for the selected crawler.
Allow and Disallow both match my URL. Which one wins?
The one with the longer path, counted in characters, regardless of which comes first in the file. Disallow: /shop with Allow: /shop/sale means /shop/sale/shoes is allowed, because /shop/sale is longer. When both rules have the same length, for example Allow: /page and Disallow: /page, Allow wins. The results table shows the deciding rule and the rule it beat.
How do * and $ work in robots.txt?
An asterisk matches any sequence of characters, including none, and a dollar sign at the very end means the URL must end there. Disallow: /*.php blocks /index.php and /folder/page.php?id=1, while Disallow: /*.php$ blocks /index.php but not /index.php?id=1. A trailing asterisk adds nothing, so /fish* is the same as /fish. Every rule is a prefix match from the start of the path, so /fish also blocks /fish.html and /fishing.
Are robots.txt paths case-sensitive?
Yes. Disallow: /Private/ does not block /private/. The directive names and user-agent tokens are case-insensitive, so user-agent: GOOGLEBOT and User-agent: Googlebot mean the same thing, but the paths in Allow and Disallow are compared exactly.
Does blocking a page in robots.txt remove it from Google?
Not necessarily. robots.txt controls crawling, not indexing. A blocked URL can still appear in search results, usually without a description, if other sites link to it. To keep a page out of the index, let it be crawled and add a noindex robots meta tag or an X-Robots-Tag header. The Noindex line in robots.txt stopped working in Google in 2019.
How do I block AI crawlers like GPTBot and ClaudeBot?
Add a group that names them, for example User-agent: GPTBot on one line, User-agent: ClaudeBot on the next, then Disallow: /. Pick each crawler in this tester to confirm the result. Remember that some tokens, like Google-Extended and Applebot-Extended, are control tokens rather than crawlers, and that fetchers acting on a user's request may not follow robots.txt at all.
Why is /robots.txt always allowed?
Because a crawler has to read the file before it can obey it. RFC 9309 says /robots.txt is implicitly allowed, so a Disallow: / rule never stops a crawler from fetching the robots file itself. The tester reflects this for any path equal to /robots.txt.
How big can a robots.txt file be?
RFC 9309 requires crawlers to parse at least 500 kibibytes, and Google stops there: rules after the first 500 KiB are ignored. If your file is close to that size, it is usually generated with one line per URL, and grouping paths with wildcards makes it far shorter and easier to maintain. The tester greys out any line past the limit.
Why does my URL show a different path under it?
The tester compares paths the way crawlers do. A full URL is reduced to its path and query string, and percent-encoding is normalised: non-ASCII characters are encoded as UTF-8, and escapes of plain letters, digits and - . _ ~ are decoded. So /caf%C3%A9 and /café, or /%7Euser and /~user, match the same rules. The normalised form is shown under your input when it differs.

About Robots.txt Tester

This tester implements that standard directly in your browser. Paste your robots.txt, list the URLs or paths you care about, one per line, and pick a crawler: Googlebot and its image, news and video variants, Bingbot, Applebot, DuckDuckBot, YandexBot, the AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and CCBot, link preview bots, SEO tools, or any custom token. For every path you get Allowed or Blocked, the exact rule that decided it with its line number, and the competing rule it beat, so a result like "blocked by Disallow: /*.pdf$ on line 5" is easy to trace. The file view highlights the group that applies to the chosen crawler and colours every line that was used.

The tester also points out problems that make a robots.txt behave differently from what its author intended: rules placed before any User-agent line, unknown or misspelled directives, full URLs where a path is expected, a UTF-8 byte order mark at the start, directives like Noindex that crawlers no longer honour, and files larger than the 500 KiB that Google reads. It lists the Sitemap lines and the Crawl-delay that applies, with a reminder that Google ignores Crawl-delay. Crawler-specific quirks are built in: Googlebot-Image falls back to the Googlebot group, Applebot follows Googlebot rules when it is not named, and the AdsBot and AdSense crawlers ignore the * group entirely. Nothing is fetched and nothing is uploaded, so you can test a draft before it goes live.

A standard that took 28 years to become one

Martijn Koster proposed the robots exclusion rules in February 1994 on the www-talk mailing list, after a badly behaved crawler overloaded his server at Nexor. By the end of June the robot authors of the time had agreed to follow it, and for the next quarter century the whole web relied on a convention that had never been formally standardised. Every search engine interpreted the edge cases a little differently, from wildcards to the length of the file.

In July 2019, on the 25th anniversary of the original proposal, Google submitted a draft to the IETF together with Koster and other search engines, and open-sourced the C++ parser that Googlebot had used for twenty years. The draft went through three more years of review and was published as RFC 9309 in September 2022. The core rules barely changed from 1994, but the RFC finally settled the questions people had argued about for decades: Allow is official, the longest match wins, and crawlers must read at least 500 KiB.

Related tools

llms.txt & AI Robots.txt Generator

Create an llms.txt file that tells AI assistants what your site is about, and robots.txt rules to allow or block AI crawlers.

URL Parser

Split a web address into its parts: domain, path, query parameters and more.

Regex Tester

Try out a regular expression on your own text and see what it matches.

Link Preview

See how a link to your page will look when shared on social apps and in search, and fix the missing tags.
Support