robots.txt vs noindex: the distinction that causes the most damage

This is the most consequential misunderstanding in technical SEO, and it produces a failure that is both common and self-perpetuating: a page somebody wanted removed from search, blocked in robots.txt, still appearing in results months later with no way to shift it.

The mechanism is worth understanding precisely, because once you see it the correct procedure is obvious.

What each one actually does

The third row is the whole problem. noindex is an instruction written on the page. If you have told the crawler it may not fetch the page, it never reads the instruction. The page stays in the index, usually with no description, and nothing you add to it will ever be seen.

The correct procedure for each goal

<!-- In the page head -->
<meta name="robots" content="noindex, follow">

<!-- Or as an HTTP header, for non-HTML files such as PDFs -->
X-Robots-Tag: noindex

Note noindex, follow rather than noindex, nofollow. You generally still want links on the page to be followed so that authority flows through to pages you do want indexed. Using nofollow as well cuts that off for no benefit.

The staging site case

Neither control is the right answer for a staging environment. robots.txt is a request that well-behaved crawlers honour and nothing else does, and noindex requires the page to be fetched by anyone who asks. Neither prevents a human or a badly behaved bot from reading your unreleased site.

Use HTTP authentication. It is the only control that actually restricts access, and it has the useful side effect of making the staging site uncrawlable by definition. This is also how the single most damaging robots.txt accident happens — a staging Disallow: / promoted to production along with everything else.

How long removal actually takes

Adding noindex does not remove a page immediately. The crawler has to revisit the URL, read the instruction and then drop it, which can take days or weeks depending on how often that page is crawled. Low-value pages are crawled infrequently, which means the pages you most want gone are often the slowest to go.

If you need something out quickly, the removals tool in Search Console↗ suppresses a URL from results temporarily while the noindex does its work. It is a stopgap rather than a fix — the suppression expires — so use both together: removals tool for speed, noindex for permanence. Resubmitting the affected URLs can also prompt a faster recrawl.

Bing and other engines have their own removal mechanisms and their own timelines, so a page gone from Google may persist elsewhere. If removal matters for a legal or reputational reason rather than a tidiness one, handle each engine separately through its webmaster tools — and remember that neither mechanism affects third-party archives or caches, which are outside your control entirely.