Robots.txt Generator

How to use

1. 1Pick a quick preset (Allow all, Block all, WordPress defaults, eCommerce defaults, Block AI scrapers, or Googlebot only) to seed sensible defaults, or start with the empty default and build the file from scratch.
2. 2Add User-agent groups for each crawler set you want to target. Use * for all bots, or list names like Googlebot, Bingbot, GPTBot, or ClaudeBot, one per line.
3. 3Inside each group, add Allow or Disallow rules. Paths must start with /. You can use * as a wildcard and $ to anchor the end of a URL (for example /*.pdf$).
4. 4Add one or more Sitemap URLs in the file-level box. Each URL must include https:// or http://.
5. 5Read the live validation panel, then click Copy to copy the generated robots.txt or Download to save it to disk and upload it to the root of your domain so it is reachable at /robots.txt.

About this tool

Robots.txt Generator builds a valid robots.txt file from structured input. Add as many User-agent groups as you need (one for *, one for Googlebot, one for AI crawlers, one for image bots, anything you like), give each group its own Allow and Disallow paths, and optionally a Crawl-delay value. File-level Sitemap URLs and an optional Host directive sit at the bottom of the output, which is the order major crawlers expect. Quick presets cover the jobs site owners ask for most often: Allow all crawlers, Block all crawlers (great for staging), WordPress sensible defaults that protect /wp-admin while keeping admin-ajax reachable, eCommerce defaults that hide carts, checkouts, account, search, and faceted parameter URLs, an Allow Googlebot only preset for fully gated sites, and a one-click Block AI scrapers preset that opts out of GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, PerplexityBot, Bytespider, CCBot, and Amazonbot in a single block. Live validation flags the mistakes that silently break crawl directives: paths that do not start with /, paths with raw spaces, an empty Allow rule (it has no effect), Crawl-delay values that are not numeric, the same User-agent appearing in two groups, sitemap URLs that are not valid http or https URLs, and a Host value that includes a protocol or path. The output panel shows the generated file with a live byte and line count, a Copy button, and a one-click Download that saves the file as robots.txt with the right MIME type. Useful for SEOs preparing a client launch, developers configuring a new site, content teams adding faceted-URL rules, store owners hiding cart and checkout from search, and anyone who wants to opt out of AI training crawlers without writing the file by hand. Everything runs in your browser, so the rules and sitemap URLs you draft never leave your device.

Free to use. Works in your browser. No signup, no login.