Skip to content
SemmeshSemmesh

Browser-side · Free & Instant

Robots.txt Generator (with AI Crawler Controls)

Generate a robots.txt with a disallow rule, a sitemap line and allow or block rules for GPTBot and Google-Extended.

User-agent: *
Allow: /
Disallow: /admin/

User-agent: GPTBot
Allow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://yourdomain.co.uk/sitemap.xml

Result for Googlebot:Allowed for crawling (Allow: /)

How to use this tool

  1. 01

    Enter the path prefix you want to keep crawlers out of, such as /admin/.

  2. 02

    Choose whether GPTBot and Google-Extended are allowed.

  3. 03

    Copy the output, replace the sitemap URL with your own, and publish it at /robots.txt on your host.

What to know

  • A disallow rule stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it; use noindex for pages that must stay out.
  • Do not block the CSS and JavaScript your pages need. Google renders pages before indexing them.

Common questions

Does blocking Google-Extended affect my Google Search rankings?
Google’s documentation says Google-Extended does not affect a site’s inclusion or ranking in Google Search. It controls whether content is used for Gemini models. Blocking Googlebot is what removes crawl access for Search.

Turn single-page checks into continuous verified SEO progress

Connect Google Search Console free inside Semmesh to prioritise technical fixes on pages that actually receive impressions and clicks.

See how the Action Engine works