Robots.txt

Robots.txt is a plain text file at the root of your domain that gives crawlers instructions about which paths they may fetch. It is a crawl-control tool, not a security or privacy tool — anything you truly need private must be behind authentication, because the file itself is public and only well-behaved bots obey it.

Use Disallow to steer crawlers away from low-value areas — internal search results, cart and account pages, endless parameter combinations — so crawl demand flows to pages that matter. Keep it minimal and test every change in Search Console before shipping; a stray Disallow: / has taken entire sites out of Google. Reference your XML sitemap at the bottom of the file so crawlers can find it.

The subtle, career-defining detail: robots.txt controls crawling, not indexing. A URL you block can still appear in results as a bare link if other pages point to it, because Google indexed the reference without seeing the page. To actually remove something from the index you must allow the crawl and serve a noindex — blocking it in robots.txt guarantees Google never sees that noindex. Get this backwards and you cement unwanted pages in the index permanently.