Do AI crawlers obey robots.txt?

The major ones publish named user agents and state that they honour robots.txt, but the picture is not uniform:

• Named agents can be addressed - GPTBot, ClaudeBot, PerplexityBot and Google-Extended each have their own directive.
• Compliance is voluntary - robots.txt is a request, not an enforcement mechanism, for any crawler.
• Blocking has a cost - a crawler you exclude cannot cite you either.
• Few have decided - only 13% of Australian small business sites publish any AI crawler rules at all^*.

Decide deliberately rather than by default: whether being quoted by an AI assistant is worth your content being read.