What is robots.txt?

robots.txt is a small text file at yourdomain.com.au/robots.txt that tells search-engine crawlers which parts of your site they may and may not visit. It is one of the first things Google reads when it arrives:

• The one dangerous line — a single careless Disallow: / tells every crawler to stay off the entire site, and is one of the most common reasons a website silently vanishes from Google after a redesign or a move off a staging server.
• Crawling, not indexing — robots.txt controls crawling, not indexing: a page blocked here can still appear in results without a description.
• Not for hiding pages — it is the wrong tool for hiding private pages, which need proper authentication instead.

After any site rebuild, check yourdomain.com.au/robots.txt for an accidental Disallow: / before you check anything else.