A robots.txt tester fetches your live robots.txt file, validates it line by line, and answers the only question that matters: can a specific crawler reach a specific URL? Enter a domain and this tool pulls the file, reports whether it exists and what HTTP status it returns, lists every declared sitemap and user-agent, and flags the mistakes that actually cost traffic. Then test any URL path against Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot or a custom user-agent — evaluated with the same precedence rules Google's crawler uses, with the winning rule highlighted right in the file. The fetch runs through our server so CORS can't block it; all parsing and testing runs in your browser. Free, no signup.
Robots.txt failures are invisible until traffic disappears. A Disallow: / copied from staging silently blocks the whole site. Blocking your CSS or JavaScript directories stops Google from rendering pages the way visitors see them, which drags down everything built on that render. A robots.txt that starts returning a 5xx error can make Google treat your entire site as off-limits while the error persists. And because crawlers ignore directives they don't understand, a simple typo — Dissallow, a missing colon, a rule placed above the first User-agent line — never produces an error anywhere; the rule just quietly stops applying. Google retired its standalone robots.txt testing tool in December 2023, and the Search Console report that replaced it shows fetch errors but won't tell you whether a given URL is blocked for a given bot. This tool restores that workflow.
Testing a URL against a crawler before you ship a robots.txt change is exactly the kind of check most teams skip — and it's the manual version of what Black & Gold SEO does continuously across your whole site: crawling every page, catching accidental blocks, conflicting directives and noindex leaks the moment they appear, and shipping the fix instead of just filing a report.
How to use it
- 1
Enter a domain and fetch the file
Type a bare domain (yourdomain.com) or a full URL. The tool requests https://yourdomain.com/robots.txt through our server and reports the HTTP status, file size and whether the file exists at all. A 404 is harmless — crawlers treat a missing robots.txt as permission to crawl everything. A 5xx is urgent: while robots.txt errors out, Google may treat the whole site as disallowed.
- 2
Review what the file declares
The summary lists every Sitemap: line (each must be an absolute URL) and every user-agent the file addresses, shown as clickable chips. No Sitemap line is a free win waiting to be claimed; a pile of bot-specific groups is your map of exactly who is being singled out. Click any chip to instantly test as that bot.
- 3
Read the lint findings
The validator flags the patterns that bite: Disallow: / under User-agent: * (blocks the entire site), rules placed before any User-agent line (ignored by every crawler), noindex directives (unsupported by Google since 2019), Crawl-delay (Google has never honored it), relative sitemap URLs, and unknown directives that every parser silently skips.
- 4
Test a URL against a crawler
Enter a path like /admin/ or paste a full URL, then pick a user-agent — Googlebot, Googlebot-Image, Bingbot, GPTBot, ClaudeBot, PerplexityBot, the * fallback, or any custom token like Google-Extended. The ALLOWED or BLOCKED verdict is computed instantly in your browser using Google's matching rules, and the winning rule line is highlighted in the file so you can see exactly why. Remember paths are case-sensitive: /Admin/ and /admin/ are different URLs.
- 5
Fix, redeploy, re-test
Edit the file at your web root — it must live at exactly /robots.txt — redeploy, and fetch again here to confirm the fix. Google caches robots.txt for up to 24 hours, so a corrected file doesn't change crawling behavior instantly; the important thing is that the live file is right.
The matching rules are precise, and most people guess them wrong. A crawler first selects one group: the group whose User-agent value is the most specific match for its product token. Googlebot-Image follows a googlebot-image group over a googlebot group, and only falls back to * when nothing else matches — which means a Disallow under * does not apply to any bot that has its own group elsewhere in the file. Within the selected group, the rule with the longest matching path wins regardless of where it sits in the file, and when an Allow and a Disallow match with equal length, Allow wins. Patterns support * as a wildcard and $ to anchor the end of a URL, and all path matching is case-sensitive. Two consequences trip people constantly: rule order means nothing (Disallow: /shop/ followed by Allow: /shop/sale/ behaves identically to the reverse), and Google only processes the first 500 KiB of the file, silently ignoring everything after.
Equally important is what robots.txt cannot do. Disallow stops crawling, not indexing: a blocked URL with inbound links can still appear in search results as “Indexed, though blocked by robots.txt” — with no snippet, because Google never read the page. To keep a page out of the index you do the opposite of what intuition says: allow crawling and serve a noindex meta tag or X-Robots-Tag header, because Google has to fetch a page to see its noindex. And never block resources your pages need to render — CSS, JavaScript, API routes that return content. Google renders pages like a browser, and a page it can't render gets evaluated on incomplete content.
In 2026, robots.txt has a second job: it's the primary control surface for AI crawlers. OpenAI's GPTBot (model training) and OAI-SearchBot (ChatGPT search) are separate tokens; Anthropic's ClaudeBot, PerplexityBot and Google-Extended (Gemini training, entirely separate from Googlebot and Google Search) each honor their own user-agent, so an allow or block decision for one does nothing to the others. Blocking training bots while allowing search-oriented ones is a legitimate strategy — but test each token separately, because a * disallow written years ago may be silently keeping your site out of AI answers you'd actually want to be cited in. That per-bot check is exactly what the tester above is for.
More free SEO tools
SERP Snippet Preview
Preview how your title and meta description appear in Google results, with live pixel-width and character limits.
Meta Tag Generator
Build copy-paste title, description, canonical, Open Graph and Twitter Card meta tags with a live preview.
Schema Markup Generator
Create valid JSON-LD structured data for Organization, LocalBusiness, Article, FAQ, Product and Breadcrumb.
See all free SEO tools or compare Black & Gold SEO vs other platforms.