robots.txt Generator

What is robots.txt?

A plain-text file at the root of your domain that tells well-behaved crawlers (Googlebot, Bingbot, etc.) which URLs they may or may not request. It is a request, not a wall — anyone determined can ignore it. For real access control, use authentication or server-side blocks.

Common patterns

• Block staging sites: User-agent: * with Disallow: /
• Block /admin: Disallow: /admin/
• Allow Googlebot, block everyone else: Two groups — one for Googlebot with no Disallow, one for * with Disallow: /
• Crawl-delay: Some crawlers honour it; Googlebot ignores it (use Search Console instead)

How CustomMind uses your robots.txt

When you sign up for CustomMind AI and ask us to crawl your site for chatbot training, we honour your robots.txt by default. Block /admin, /checkout, or any other path you don't want indexed and we'll skip it.