Robots.txt
A file telling search engine crawlers which pages they can and can't visit.
The Tradie Take
Robots.txt is like a sign on your workshop door that says "apprentice entrance at the back." It tells Google's bot where to enter and where not to go. If you put the sign on the wrong door or write it in a language the apprentice can't read, they can't get in at all. Check your robots.txt to make sure you're not accidentally blocking Google.
Definition
Robots.txt is a text file on your website that tells search engine crawlers (like Googlebot) which parts of your site they are allowed to crawl and index. It's the first thing Googlebot checks when it visits your site. You can use it to block crawlers from admin pages, duplicate content, or files that don't need to be indexed. Blocking the wrong pages by accident can hide your entire site from Google.
How It Works
- 1 The file lives at the root of your site: yoursite.com.au/robots.txt
- 2 It contains rules: "Allow" or "Disallow" for specific crawlers and paths
- 3 Googlebot reads this file before crawling any pages on your site
- 4 A misconfigured robots.txt can accidentally block Google from your entire site
Tips for Tradies
- Don't block CSS, JavaScript, or image files - Google needs these to see your pages properly
- Use it to block duplicate pages, admin sections, and internal search results pages
- Test your robots.txt in Google Search Console's robots.txt tester