What is GPTBot?

GPTBot is OpenAI’s training crawler, and it is only one of the agents OpenAI runs - which is why “block OpenAI” is the wrong instruction:

• GPTBot is for training - OpenAI documents it as the crawler “used to crawl content that may be used in training our generative AI foundation models” (OpenAI).
• OAI-SearchBot is the one that carries citations - it feeds ChatGPT search, and OpenAI says sites opted out of it “will not be shown in ChatGPT search answers”.
• ChatGPT-User is a person clicking - it fetches a page because someone asked ChatGPT to open it, and OpenAI notes that for user-initiated actions “robots.txt rules may not apply”.
• The tokens are independent - a robots.txt rule names one user-agent at a time, so refusing training does not have to mean refusing citation.

Ten per cent of Australian small-business sites name GPTBot in robots.txt at all^*. If you block one thing, make sure it is GPTBot and not OAI-SearchBot.