robots.txt Validator & Rule Tester
Test and explain robots.txt rules against user agents and candidate URLs. Inspect group specificity, equal-length tie-breaks, wildcards, and percent encoding per RFC 9309.
Enter a crawler product token (e.g. "Googlebot") or full User-Agent header.
- • https://draftsbook.com/sitemap.xml
- • https://draftsbook.com/sitemap-articles.xml
RFC 9309 Protocol Evaluation Context
User-agent: Googlebot
Specific Token Match
2 rules
| Decision | Candidate Path | Matched Rule | Line | Specificity & Reason | Inspect |
|---|---|---|---|---|---|
| ALLOWED | /https://draftsbook.com/ | ALLOW: / | #11 | Matched "ALLOW: /" (line 11, pattern length 1). | |
| ALLOWED | /tools/robots-testerhttps://draftsbook.com/tools/robots-tester | ALLOW: / | #11 | Matched "ALLOW: /" (line 11, pattern length 1). | |
| ALLOWED | /admin/settingshttps://draftsbook.com/admin/settings | ALLOW: / | #11 | Matched "ALLOW: /" (line 11, pattern length 1). | |
| ALLOWED | /api/v1/docshttps://draftsbook.com/api/v1/docs | ALLOW: / | #11 | Matched "ALLOW: /" (line 11, pattern length 1). | |
| ALLOWED | /api/private/usershttps://draftsbook.com/api/private/users | ALLOW: / | #11 | Matched "ALLOW: /" (line 11, pattern length 1). |
RFC 9309 robots.txt Validator & Rule Tester Guide
Simulate search engine crawler permissions, specificity, wildcard paths, and equal-length tie-breaks.
Overview
The robots.txt Validator simulates crawler behavior defined in RFC 9309 (Robots Exclusion Protocol). It parses records into user-agent groups, selects the most specific group matching your targeted crawler (with wildcard fallback), extracts path patterns, normalizes unreserved percent-encoded characters, and evaluates candidate URLs with character-length specificity and tie-breaking guarantees.
How to Validate robots.txt Rules
Input your production or staging robots.txt file into the editor, or pick from our built-in RFC 9309 test scenarios.
Enter a search crawler product token (e.g. Googlebot, Bingbot, DuckDuckBot) or use the quick selection pills.
Provide paths or full URLs (one per line) to evaluate. The validator extracts path and query components and normalizes percent encoding.
View an immediate ALLOWED or DISALLOWED decision per URL. Click "Details" to inspect all matching group rules and pattern length rankings.
Copy a formatted Markdown evaluation report or download the tested robots.txt file.
Disallow: /catalog Allow: /catalog Candidate URL: https://example.com/catalog/items
ALLOWED (Matched "ALLOW: /catalog", Line 3, Pattern Length 8)
When an Allow rule and a Disallow rule have equal pattern lengths and both match, RFC 9309 mandates following the most permissive rule (Allow).
- RFC 9309 Compliance: Full fidelity parsing of User-agent groups, member rules, and wildcard * / end-anchor $ syntax.
- Percent-Encoding: Automatically decodes unreserved ASCII octets (A-Z, a-z, 0-9, -, ., _, ~) per RFC 3986.
- Sitemap & Delay Extraction: Discovers all Sitemap: and Crawl-delay directives across the file.
- Zero Network Requests: The validator runs purely in local JavaScript memory and never pings tested domains.
- Private Infrastructure Safety: Unreleased staging routes, confidential admin paths, and internal URLs remain strictly in your browser.