Robots.txt
Generator
Create a clean robots.txt with your sitemap, blocked paths and optional AI crawler rules.
Choose whether to allow crawling, add any paths you want crawlers to avoid, include your sitemap URL and optionally block specific AI crawlers. Copy or download the file and place it in your site’s root folder. Robots.txt is a request, not a security measure.
Your rules
Your robots.txt
Robots.txt is a request that well-behaved crawlers follow, not a lock. It does not hide a page: a blocked page can still appear in results if other sites link to it, so use a noindex tag or a password for private content. Blocking AI crawlers may reduce how often your content is used in their answers, so decide deliberately.
What robots.txt can and cannot do
A robots.txt file tells search engine crawlers which parts of your site they should not crawl. It is also the usual place to point to your XML sitemap.
- It does not hide pages. Use noindex or a password for pages that must stay private.
- Blocking a page can stop crawlers from reading a noindex tag on it.
- Never block the CSS and JavaScript your pages need to display.
- Use "Disallow: /" only on staging sites, never on your live site.
Our SEO service includes a technical review of crawling and indexing. Also see the AI-search readiness score.
Frequently Asked
Where does robots.txt go?
At the root of your domain, so it loads at yourdomain.com/robots.txt.
Does blocking a page remove it from Google?
Not reliably. A blocked page can still be listed if other sites link to it. Use a noindex tag or password protection instead.
Should I block AI crawlers?
It is your decision. Blocking them may stop your content from being used by those systems, which can also reduce how often you are mentioned in their answers.
Do all crawlers obey robots.txt?
Reputable search engines do. Bad bots may ignore it, so it is not protection against scraping or attacks.
Need Help Putting This
Into Practice?
Tell us what you need. We reply with an honest recommendation and a clear plan.
