robots.txt

robots.txt is a small text file at example.com.au/robots.txt that tells search engines which parts of your site they shouldn’t crawl.

Why it matters

Used well, it keeps Google away from pages that don’t matter, such as admin areas or internal search results. Used badly, it can block Google from your whole site.

Things to know

• It doesn’t hide pages from Google. A blocked page can still appear in results if other sites link to it. To keep a page out of Google, use noindex instead.
• Don’t block pages you’ve marked noindex. If Google can’t crawl the page, it never sees the noindex.
• Watch for Disallow: /. This line blocks the entire site. It’s often left behind after a site is built. If you see it and your site isn’t meant to be hidden, remove it.
• Add a Sitemap: line pointing to your XML sitemap.

A typical robots.txt for a small business site is very short:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com.au/sitemap_index.xml

Where to edit it

Platform	Where
WordPress (Yoast SEO)	Yoast SEO > Tools > File editor
WordPress (Rank Math)	Rank Math > General Settings > Edit robots.txt
Shopify	Shopify creates it for you. Changes need a robots.txt.liquid template in your theme, so ask your developer.
Wix	Settings > SEO > Robots.txt editor (search “robots.txt” in your dashboard if you can’t find it).
Squarespace	Squarespace manages it for you and it can’t be edited.
Custom-built site	Ask your developer to edit the file in the site’s root folder.

Check it worked

Type your domain followed by /robots.txt into your browser to see the live file. In Search Console, the robots.txt report under Settings shows the version Google last fetched and any errors.

Google’s guide

Introduction to robots.txt (Google Search Central)