Robots.txt Generator

robots.txt is a plain-text file at the root of a website that tells search-engine crawlers and other bots which parts of the site they may fetch. A typical file contains rule groups — a User-agent line followed by Allow and Disallow paths — and often a pointer to the sitemap. Hand-writing it is simple but easy to get subtly wrong, such as shadowing an Allow rule with a Disallow rule. This generator assembles the file from editable rule rows and presets, giving site owners and SEO professionals a correct starting point every time. The generated file follows the widely recognized robots.txt syntax, so it works with Google, Bing and the major crawlers out of the box.

What the generator builds

• 🤖 Per-crawler rule groups, from a specific bot to the catch-all *
• 🚫 Disallow and ✅ Allow multiple paths per group
• ⏱️ Optional Crawl-delay line
• 🗺️ One or more Sitemap lines
• 🧩 Presets to start from: allow all, block all, WordPress defaults
• 📋 Copy the finished file or download it directly

When you would write one

• Blocking a staging or preview environment from search results
• Stopping a specific scraper from hammering your site
• Setting up crawl rules for a fresh WordPress install
• Directing crawlers to your sitemap after a redesign

The file is assembled entirely in your browser; nothing is uploaded. Two things to know: you must upload the finished file to your site's root and confirm it is reachable, and robots.txt is a request, not an enforcement — well-behaved crawlers honor it, but it cannot force a bot to stop.