Robots.txt Generator

How it works

robots.txt is the plain-text file at the root of your site that tells crawlers which paths they may fetch. Each rule block names a User-agent followed by Allow or Disallow paths; crawlers match their own agent name first and fall back to *. The generator builds those blocks row by row — agent, directive, path — and renders the final file ready to paste at your domain root.

Optional fields cover the two standard extras: a Sitemap: line pointing crawlers at your XML sitemap (supported by every major engine), and Crawl-delay, honored by Bing and Yandex but ignored by Google — for Google, rate limiting lives in Search Console settings, not robots.txt.

The AI-crawler preset adds disallow blocks for the common LLM bots — GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot and others — for sites that want to opt out of training crawls while staying fully visible to search. Blocking AI bots does not affect Googlebot or rankings; the directives are independent.

Remember what robots.txt is not: it is a crawl hint, not access control. Well-behaved bots honor it, malicious ones ignore it, and a disallowed URL can still appear in search results without a description if other pages link to it. To keep a page out of the index, use a noindex meta tag — and never block CSS or JS that Google needs to render your pages.

Worked example

Build a typical configuration: everything allowed, one private section excluded, sitemap declared.

1. Add your rules: One row per directive: user-agent, Allow or Disallow, path prefix.
2. Use a preset: Allow all, block all, or block common AI crawlers in one click.
3. Declare the sitemap: Add the sitemap URL so engines discover every indexable page.
4. Copy and verify: Paste at your domain root, then test in Search Console and Bing WMT.

Matching is prefix-based: Disallow: /admin also blocks /administrator. Google supports * wildcards and $ end-anchors (/*.pdf$), but those extensions are not universal — keep rules simple for cross-engine safety.

Common errors and edge cases

• Disallow is not noindex. A blocked URL can still be indexed (without snippet) if links point to it. Use a noindex meta tag for deindexing; robots.txt only controls crawling.
• Blocking assets breaks rendering. Disallowing /css/ or /js/ prevents Google from rendering your pages correctly — a direct ranking risk. Never block render resources.
• Order and specificity. For Google, the most specific matching rule wins (Allow can override a broader Disallow); simpler bots may read rules in order. Test edge cases in Search Console's robots.txt tester.
• Crawl-delay does not apply to Google. Only Bing, Yandex and some smaller bots honor it. Google crawl rate is managed in Search Console.
• Wrong file location. The file must live at the domain root — /robots.txt — plain text, publicly accessible. A robots.txt in a subdirectory is ignored.

Code equivalents

Verify the result with the engines themselves before shipping — robots.txt mistakes are silent and expensive.

Related tools

• Generate the noindex meta tag for pages robots.txt alone cannot keep out of the index.
• Check that your robots.txt returns a clean 200 — not a redirect or soft 404.
• Clean, stable URL slugs make simple, durable robots rules possible.

Frequently asked questions

Privacy note

All processing happens in your browser. The values you enter never leave your device and are never transmitted to ToolPlex.