Robots.txt Tester

How to use

1. 1Paste your robots.txt body into the left panel, or click Load sample to see an example.
2. 2Type the URL or site path you want to test, then pick a crawler user-agent or type a custom one.
3. 3Read the Verdict panel: green Allowed or rose Disallowed, the matched rule, line number, and group used.
4. 4Scroll to the Parsed groups, Sitemaps, and Syntax notes panels to verify every directive parses correctly.

About this tool

Robots.txt Tester evaluates any URL against a robots.txt body for any crawler user-agent and returns a clear allow or disallow verdict, the exact rule that matched, the line it came from, and which user-agent group the crawler fell into. Paste a robots.txt body (yours, a competitor's, or a draft you are working on), pick a URL or path to test, and switch between common crawlers (Googlebot, Googlebot-Image, Bingbot, DuckDuckBot, YandexBot, facebookexternalhit, Twitterbot, GPTBot) or type any custom user-agent token. The matcher follows RFC 9309 and Google's documented behavior: rules are grouped by User-agent, the longest path-pattern wins, Allow beats Disallow on a tie, paths are case-sensitive, the * wildcard matches any sequence, and $ pins the end of URL. The tool also lists every parsed group with its rules, surfaces all Sitemap and Host directives, and flags syntax issues like missing colons, unknown directives, orphan rules outside any group, or non-numeric Crawl-delay values. Everything runs in your browser, so the robots.txt body and the URLs you check never leave your device, which makes the tool safe for staging sites, internal subdomains, and competitive analysis.

Free to use. Works in your browser. No signup, no login.