Browser-side · Free & Instant
Robots.txt Path Tester
Test whether a URL path is blocked for Googlebot by a disallow rule.
User-agent: * Allow: / Disallow: /admin/ User-agent: GPTBot Allow: / User-agent: Google-Extended Disallow: / Sitemap: https://yourdomain.co.uk/sitemap.xml
Result for Googlebot:Allowed for crawling (Allow: /)
How to use this tool
- 01
Enter the disallow prefix from your robots.txt.
- 02
Enter the URL path you want to test, starting with a slash.
- 03
Read the result: blocked means the path starts with the disallowed prefix.
What to know
- Prefixes match from the start of the path. Disallow: /admin also blocks /administration-guide, while Disallow: /admin/ does not.
- A crawler follows the most specific user-agent group that names it and ignores the rest. Rules in the * group do not add to a group written for that crawler.
Common questions
What happens when an Allow and a Disallow rule both match?
Under RFC 9309 the most specific rule wins, meaning the one with the longest matching path. If the two are equally specific, Allow should be used.
Related guide
How to run a technical SEO audit that earns measurable clicks
In Semmesh
First-Party Technical Site Audit & Fix Verifier
More free tools
Turn single-page checks into continuous verified SEO progress
Connect Google Search Console free inside Semmesh to prioritise technical fixes on pages that actually receive impressions and clicks.