How to use it
- Pick a starting point: allow everything, block AI training, block all AI crawlers, or block everything.
- Add your sitemap, folders to keep private, and any other crawlers to block.
- Copy or download
robots.txtand put it at the root of your site, so it's athttps://your-site/robots.txt.
Search engines, AI search and AI training
Crawlers now come in three kinds, and robots.txt can treat each one differently:
- Search engines (Googlebot, Bingbot) index pages for search results.
- AI search and assistants (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) fetch pages to answer questions, and usually cite and link to them.
- AI training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and others) collect pages to train models.
Block AI training keeps the last group out while you stay in search results and AI answers. Google-Extended and Applebot-Extended don't affect Google or Apple search; they only control training.
The mistake that hides a site
A crawler obeys only the most specific group that names it. Once GPTBot has a group of its own, the rules under User-agent: * no longer apply to it. So if you keep /admin/ private for * and add a group for another crawler that allows the rest of the site, that crawler can read /admin/. The generator warns about it; repeat the lines in each group.
Frequently asked questions
Does robots.txt keep a page out of Google?
Not reliably. It stops crawling, not indexing: a blocked page can still appear in results if other sites link to it. To keep a page out, let it be crawled and add <meta name="robots" content="noindex">, which the Meta Tag Generator writes.
Do crawlers have to obey it?
It's a voluntary standard (RFC 9309). The major search engines and AI companies say their crawlers follow it; others may not. Don't use it to hide anything private: it's a public file that lists the paths you'd rather not have crawled.
Where does the file go on Lobsite?
Put robots.txt at the top of the folder you deploy, next to index.html. Sites without an account are never indexed anyway: Lobsite serves them with X-Robots-Tag: noindex until they're kept in an account.
Should I add my sitemap?
Yes. The Sitemap: line is the simplest way for crawlers to find every page. Make one with the XML Sitemap Generator.