🤖 Robots.txt Tester
Test whether a URL path is allowed or disallowed for a crawler product token, inspect the matching rule and review sitemap directives. Fetch a public robots.txt file or analyze text locally.
Robots.txt Source
The tool requests the website's root /robots.txt. This protected endpoint blocks local and private network addresses.
URL Rule Test
The selected product token is compared with the rules in the loaded file. This is a local parser result, not a live Google/Bing crawl decision.
File Summary
No robots.txt loaded
User-agent tokens
0
Allow rules
0
Disallow rules
0
Sitemap entries
0
Unknown directives
0
Notes
1
Review notes
- The robots.txt file is empty and contains no directives.
🔎 How to use the result
The tool reports which Allow or Disallow rule the selected product token matches in the supplied file. Path matching is case-sensitive. A matching Allow rule can override a less-specific Disallow rule; the parser applies its own RFC-oriented matching behavior.
⚠️ Important limitation
robots.txt is a crawler request, not access control. It does not protect private pages, guarantee indexing, or prove what a search engine has crawled. Crawlers may interpret extensions differently or ignore the file.