Noindex vs Disallow: Choosing the Right Control for a Page
Disallow and noindex are often treated as two ways of saying the same thing, yet they act at different stages. Disallow tells a crawler not to fetch a URL. Noindex tells a search engine not to show a page it has fetched. Choosing the wrong one leaves pages in results or wastes crawling on pages nobody needs.
What each control actually does
A disallow rule in robots.txt stops compliant crawlers from requesting the URL. Because the page is never fetched, the crawler cannot see anything inside it, including a noindex tag. The URL may still be indexed from links alone, usually with little or no description.
A noindex directive can be placed in a robots meta tag or sent as an X-Robots-Tag HTTP header. The page must be crawlable for the directive to be read. Once processed, the page is dropped from results even if other sites link to it.
Why combining them backfires
Adding noindex to a page and disallowing the same URL is a common mistake. The disallow rule hides the noindex directive, so a page that was already indexed can remain visible. When removing pages from an index, keep them crawlable until the noindex has been processed, then decide whether blocking is still useful.
For files that cannot carry HTML tags, such as PDFs or images, the X-Robots-Tag header is the practical way to send noindex.
X-Robots-Tag: noindex
<meta name="robots" content="noindex">Picking a control per page type
Use noindex for pages people may reach but that should not appear in search, such as thank-you pages, internal search results or thin tag archives. Use disallow for areas that create large numbers of low-value URLs, such as endless filter combinations, where saving crawler requests matters more than the index state.
Never rely on either control to protect private information. Anything confidential needs authentication, because robots rules are public and only respected by well-behaved crawlers.