A robots.txt generator builds the plain-text crawl-control file that sits at the root of your domain and tells search engines — and now AI crawlers — which paths they may fetch. This one runs entirely in your browser: start from a preset (allow everything, block everything, WordPress, Next.js static, or e-commerce), edit rules per user-agent, set an allow/block policy for the six major AI crawlers, add your sitemap URL, then copy or download a valid robots.txt file. Live validation flags dangerous rules as you build. Free, no signup, and nothing you type leaves the page.
Robots.txt is the highest-stakes twenty lines on your site because its failures are silent and total. A single Disallow: / under User-agent: * — the exact line staging environments ship with — tells every search engine to stop crawling your entire site. Blocking your CSS and JavaScript directories (old WordPress advice, still copy-pasted in 2026) makes Googlebot render your pages half-blank and rank them accordingly. And the file cuts the other way now too: GPTBot, ClaudeBot and PerplexityBot all read it, which makes robots.txt the policy document that decides whether AI assistants can read — and cite — your content at all.
Use this tool to create a robots.txt by hand and understand exactly what each directive does before it goes live. It's the manual version of what Black & Gold SEO runs continuously across your whole site — crawling it the way Google does, catching accidental blocks, indexation traps and sitemap drift, and shipping the fixes instead of just listing them.
How to use it
- 1
Start from a preset
Pick the preset closest to your stack. Allow everything is the correct default for most public sites. WordPress blocks /wp-admin/ while keeping admin-ajax.php crawlable and shuts off internal search URLs. Next.js (static) allows everything but keeps /api/ out. E-commerce blocks cart, checkout, account and the faceted ?sort=/?filter= parameter URLs that burn crawl budget. Block everything is for staging and private sites only — the validator will warn you loudly if it's still set when you ship.
- 2
Add or edit rules
Each rule is a user-agent (use * to match all crawlers), a directive (Allow or Disallow), and a path. Paths are case-sensitive and must start with /. Google and Bing also support * as a wildcard and $ to anchor the end of a URL — so Disallow: /*?sort= kills sort-parameter duplicates and Disallow: /*.pdf$ blocks every PDF. Rules for the same user-agent are grouped automatically in the output.
- 3
Set your AI-crawler policy
Toggle GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini training), CCBot (Common Crawl) and Bytespider (ByteDance) individually. The default is allow, because crawlable content is what gets cited in ChatGPT, Claude and Perplexity answers — block a bot only when keeping content out of model training matters more than that visibility.
- 4
Add your sitemap and optional crawl-delay
The Sitemap: line takes a full absolute URL (the generator prepends https:// if you type a bare domain) and is the one directive that helps crawlers find pages rather than restricting them. Crawl-delay throttles compliant bots like Bingbot and Yandex to one request every N seconds — Google ignores it completely, so never rely on it to protect a struggling server from Googlebot.
- 5
Review the validation warnings
The generator flags site-killing rules (Disallow: / under User-agent: *), paths missing their leading slash (which never match anything), blocked CSS/JS asset paths that break Google's rendering, an all-AI-bots block that erases you from AI answers, and crawl-delays high enough to starve large sites of crawling.
- 6
Download and upload to your site root
Copy the output or download robots.txt, then place it at your domain root so it resolves at https://yourdomain.com/robots.txt — exact lowercase filename, and one file per host, so subdomains need their own. After deploying, verify it with the robots.txt report in Google Search Console and spot-check a few blocked and allowed URLs.
To write good rules, know how crawlers actually read the file. A crawler picks the single most specific user-agent group that matches it and ignores the rest — a User-agent: Googlebot group replaces, not extends, your User-agent: * rules for Google. Within a group, Google applies the most specific (longest) matching rule, and when an Allow and a Disallow tie, the Allow wins — which is exactly why Allow: /wp-admin/admin-ajax.php successfully punches a hole through Disallow: /wp-admin/. Paths are case-sensitive (/Search/ and /search/ are different rules), the file only governs the exact host and protocol it's served from, and Google stops parsing after 500 KiB — anything past that is ignored.
The most misunderstood fact in technical SEO: robots.txt controls crawling, not indexing. A URL you disallow can still be indexed from external links and will appear in results as “Indexed, though blocked by robots.txt” — title only, no snippet. Worse, the block prevents Google from ever seeing a noindex tag on that page, so the two directives fight each other. To actually keep a page out of the index, leave it crawlable and use a noindex meta tag or X-Robots-Tag header. The file's HTTP status matters too: a 404 on /robots.txt just means everything is crawlable (harmless), but persistent 5xx errors make Google treat the entire site as disallowed — a misconfigured server can de facto deindex you through this one URL.
In 2026 the sharpest robots.txt decision is the AI-crawler section. Allowing GPTBot, ClaudeBot and PerplexityBot is how your content becomes quotable in ChatGPT, Claude and Perplexity answers — a discovery channel that keeps growing while classic blue-link clicks decline. Blocking them keeps your content out of training data and live retrieval, a legitimate call for paywalled or proprietary content, but understand the trade: no citations, no brand mentions, no referral clicks from those surfaces. Two nuances this generator encodes: Google-Extended only opts you out of Gemini model training and does not affect Google Search or your rankings, and Bytespider has a documented history of ignoring robots.txt — treat a block there as a request, not enforcement. Content that genuinely must stay private needs authentication, not a directive.
More free SEO tools
SERP Snippet Preview
Preview how your title and meta description appear in Google results, with live pixel-width and character limits.
Meta Tag Generator
Build copy-paste title, description, canonical, Open Graph and Twitter Card meta tags with a live preview.
Schema Markup Generator
Create valid JSON-LD structured data for Organization, LocalBusiness, Article, FAQ, Product and Breadcrumb.
See all free SEO tools or compare Black & Gold SEO vs other platforms.