Learn · AI Search · Beginner
Learn what Google-Extended, GPTBot, OAI-SearchBot and PerplexityBot are each documented to do, then write, upload and check a robots.txt file that reflects your choice.
By Dean Cruddace · 45 minutes to do · Updated · Last reviewed
An AI crawler is an automated program that fetches web pages for an AI company's products. Each company publishes a short name for its crawlers, and you can use that name in a plain text file called robots.txt to ask it to stay away from some or all of your site.
The names are not one group. Each company documents separate jobs for separate names, such as training a model or finding pages for a search answer. Blocking one does not block the others.
Google says a robots.txt file tells crawlers which URLs they can access and that it cannot enforce behaviour: it is up to the crawler to obey (Google's robots.txt introduction). So a rule is a request, and you need to know what you are requesting. OpenAI, for example, says its crawler settings are independent of each other, so a site can allow one for search results while disallowing another to indicate its content should not be used for training (OpenAI's crawler overview). Choosing without reading this can block something you wanted, or leave open something you did not.
Type your site's address followed by /robots.txt into the browser. Google says the file must sit at the root of the site and applies only to the protocol, host and port it is posted on, so a subdomain needs its own. Copy what you see into a note. On a hosted builder you may not be able to edit it directly. See how to create a robots.txt file.
You will know it worked when You have a saved copy of the current file, or you know there is none (Google says that without rules, everything is implicitly allowed to be crawled).
Write one line per name, from the vendors' own pages. Google-Extended: manages whether crawled content may be used for training future Gemini models and for grounding in Gemini Apps and Vertex AI; Google says it does not affect inclusion in Google Search (Google's common crawlers). GPTBot: crawls content that may be used for training. OAI-SearchBot: surfaces sites in ChatGPT's search features. ChatGPT-User: acts on user requests; OpenAI says robots.txt rules may not apply. PerplexityBot: surfaces and links sites in Perplexity search results and is not used for foundation model training. Perplexity-User: generally ignores robots.txt.
You will know it worked when You have a list of names, each with one documented purpose and a note of whether robots.txt controls it.
For each name robots.txt can control, choose allow or block. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and recommends allowing it to appear in search results; Perplexity recommends allowing PerplexityBot for the same reason. Neither page says that allowing or blocking affects traffic. Write down each decision and the reason.
You will know it worked when Each name has a decision and a one-line reason, agreed by whoever owns the decision.
In a plain text editor, add one group per crawler: a User-agent: line naming it, then a Disallow: line. Google gives Disallow: / as the way to block a named agent from the whole site. Hypothetical example, blocking only training use by two vendors:
User-agent: GPTBotDisallow: /
User-agent: Google-ExtendedDisallow: /
Google says rules are case-sensitive and the file should be saved as UTF-8.
You will know it worked when Reading the file back, every group begins with a User-agent line and the names match the vendors' spelling.
Put the file at the root of your site, or change the setting in your CMS. Then open a private browsing window and visit /robots.txt, as Google advises. OpenAI says its search systems can take around 24 hours to adjust; Perplexity says up to 24 hours.
You will know it worked when The private window shows your new rules at the root address.
Google says the robots.txt report in Search Console helps fix issues with markup, for files already accessible on your site. Open it and look for problems. This confirms Google can parse the file; the pages we read do not say it tests other companies' crawlers.
You will know it worked when The report shows no errors for your file.
Go deeper
Send Dean your current robots.txt and what you want to protect or be found for. Access decisions sit between technical SEO and AI search visibility.
