Skip to content
SemmeshSemmesh

Browser-side · Free & Instant

Robots.txt Path Tester

Test whether a URL path is blocked for Googlebot by a disallow rule.

User-agent: *
Allow: /
Disallow: /admin/

User-agent: GPTBot
Allow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://yourdomain.co.uk/sitemap.xml

Result for Googlebot:Allowed for crawling (Allow: /)

How to use this tool

  1. 01

    Enter the disallow prefix from your robots.txt.

  2. 02

    Enter the URL path you want to test, starting with a slash.

  3. 03

    Read the result: blocked means the path starts with the disallowed prefix.

What to know

  • Prefixes match from the start of the path. Disallow: /admin also blocks /administration-guide, while Disallow: /admin/ does not.
  • A crawler follows the most specific user-agent group that names it and ignores the rest. Rules in the * group do not add to a group written for that crawler.

Common questions

What happens when an Allow and a Disallow rule both match?
Under RFC 9309 the most specific rule wins, meaning the one with the longest matching path. If the two are equally specific, Allow should be used.

Turn single-page checks into continuous verified SEO progress

Connect Google Search Console free inside Semmesh to prioritise technical fixes on pages that actually receive impressions and clicks.

See how the Action Engine works