# Robots.txt Generator

Build a valid robots.txt: allow search engines, block AI training crawlers like GPTBot, keep folders private and add your sitemap. Warnings for common mistakes.

Use it at https://lobsite.com/tools/robots-txt-generator

## How to use it

1. Pick a starting point: allow everything, block AI training, block all AI crawlers, or block everything.
2. Add your sitemap, folders to keep private, and any other crawlers to block.
3. Copy or download `robots.txt` and put it at the root of your site, so it's at `https://your-site/robots.txt`.

## Search engines, AI search and AI training

Crawlers now come in three kinds, and robots.txt can treat each one differently:

- **Search engines** (Googlebot, Bingbot) index pages for search results.
- **AI search and assistants** (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) fetch pages to answer questions, and usually cite and link to them.
- **AI training** crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and others) collect pages to train models.

**Block AI training** keeps the last group out while you stay in search results and AI answers. Google-Extended and Applebot-Extended don't affect Google or Apple search; they only control training.

## The mistake that hides a site

A crawler obeys only the most specific group that names it. Once GPTBot has a group of its own, the rules under `User-agent: *` no longer apply to it. So if you keep `/admin/` private for `*` and add a group for another crawler that allows the rest of the site, that crawler can read `/admin/`. The generator warns about it; repeat the lines in each group.

## Frequently asked questions

### Does robots.txt keep a page out of Google?

Not reliably. It stops crawling, not indexing: a blocked page can still appear in results if other sites link to it. To keep a page out, let it be crawled and add `<meta name="robots" content="noindex">`, which the [Meta Tag Generator](https://lobsite.com/tools/meta-tag-generator) writes.

### Do crawlers have to obey it?

It's a voluntary standard (RFC 9309). The major search engines and AI companies say their crawlers follow it; others may not. Don't use it to hide anything private: it's a public file that lists the paths you'd rather not have crawled.

### Where does the file go on Lobsite?

Put `robots.txt` at the top of the folder you deploy, next to `index.html`. Sites without an account are never indexed anyway: Lobsite serves them with `X-Robots-Tag: noindex` until they're kept in an account.

### Should I add my sitemap?

Yes. The `Sitemap:` line is the simplest way for crawlers to find every page. Make one with the [XML Sitemap Generator](https://lobsite.com/tools/sitemap-generator).