Browser-side · Free & Instant
Robots.txt Generator (with AI Crawler Controls)
Generate a robots.txt with a disallow rule, a sitemap line and allow or block rules for GPTBot and Google-Extended.
User-agent: * Allow: / Disallow: /admin/ User-agent: GPTBot Allow: / User-agent: Google-Extended Disallow: / Sitemap: https://yourdomain.co.uk/sitemap.xml
Result for Googlebot:Allowed for crawling (Allow: /)
How to use this tool
- 01
Enter the path prefix you want to keep crawlers out of, such as /admin/.
- 02
Choose whether GPTBot and Google-Extended are allowed.
- 03
Copy the output, replace the sitemap URL with your own, and publish it at /robots.txt on your host.
What to know
- A disallow rule stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it; use noindex for pages that must stay out.
- Do not block the CSS and JavaScript your pages need. Google renders pages before indexing them.
Common questions
Does blocking Google-Extended affect my Google Search rankings?
Google’s documentation says Google-Extended does not affect a site’s inclusion or ranking in Google Search. It controls whether content is used for Gemini models. Blocking Googlebot is what removes crawl access for Search.
Related guide
How to run a technical SEO audit that earns measurable clicks
In Semmesh
First-Party Technical Site Audit & Fix Verifier
More free tools
Turn single-page checks into continuous verified SEO progress
Connect Google Search Console free inside Semmesh to prioritise technical fixes on pages that actually receive impressions and clicks.