Noindex vs robots.txt Disallow at a Glance
| Category | noindex | robots.txt Disallow | Winner |
|---|---|---|---|
| Best for | Keeping a reachable page out of search results | Stopping crawlers from fetching URLs | Depends |
| Starting price | Free | Free | Tie |
| What it controls | Indexing (and display in results) | Crawling (access) | Depends |
| Keeps the URL out of Google results | Yes, once the page is recrawled | Not reliably; the URL can be indexed from links | noindex |
| Saves crawl capacity | No; the page must be fetched | Yes | Disallow |
| Where it lives | Page <head> or X-Robots-Tag HTTP header | One file at the site root | Depends |
| Ease of use for many URLs | Needs a template or server rule | One pattern can cover a whole path | Disallow |
| PDFs and other non-HTML files | Via X-Robots-Tag header | Blocks crawling; keeps images, video and audio out of Google | Depends |
| Standard | Search-engine documentation | RFC 9309 plus search-engine documentation | Disallow |
| Security | None | None; paths are public | Tie (neither) |
| Overall | Use noindex to keep pages out of results; use Disallow to manage crawling | Depends | |
Each row is drawn from the Google, Bing and IETF documents cited below, checked October 8, 2026.
What Is noindex?
noindex is a rule, given in a robots <meta> tag or an X-Robots-Tag HTTP header, that tells search engines which support it not to show a page or file in their results.
Google: “When Googlebot crawls that page and extracts the tag or header, Google will drop that page entirely from Google Search results, regardless of whether other sites link to it” (Block Search indexing with noindex). The two forms “have the same effect”. Google adds that “Specifying the noindex rule in the robots.txt file is not supported by Google.” Bing supports the same tag and header, and lets you replace robots with bingbot to target only Bing (Bing robots meta tags).
<meta name="robots" content="noindex">
X-Robots-Tag: noindexWhat Is a robots.txt Disallow Rule?
A Disallow rule is a line in a site's robots.txt file that asks crawlers not to request URLs whose path matches a given pattern.
RFC 9309, the Robots Exclusion Protocol, says allow and disallow lines “indicate whether accessing a URI that matches the corresponding path is allowed or disallowed”, and that “The most specific match found MUST be used” (RFC 9309). Its scope is access: crawlers are “requested to honor” the rules when accessing URIs. Google describes robots.txt as “used mainly to avoid overloading your site with requests” (Introduction to robots.txt).
User-agent: *
Disallow: /search/
Disallow: /cart/Noindex vs robots.txt Disallow: Cost and Access
| Item | noindex | robots.txt Disallow |
|---|---|---|
| Cost | Free | Free |
| Access needed | Edit page templates or HTTP headers; Google notes it is useful “if you don't have root access to your server” | Write access to /robots.txt at the root of each host |
| CMS support | Google says many CMSs expose a search-settings page instead of raw HTML | Google says the same applies to robots.txt |
| Size limits | None documented | Google reads the first 500 KiB; RFC 9309 sets 500 KiB as the minimum parsing limit |
Neither has a vendor price; the cost is developer time. “Pricing” in the usual sense does not apply.
Keeping a Page Out of Search Results
noindex is the reliable way to keep a page out of Google and Bing results; robots.txt is not. Google says a disallowed URL can still be indexed “if it is linked from other places on the web”, so “the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google Search results” (Introduction to robots.txt).
Search Console even has a status for this: “Indexed, though blocked by robots.txt”. Google's advice there: “robots.txt is not the correct mechanism to avoid being indexed. To avoid being indexed, remove the robots.txt block and use 'noindex'” (Page indexing report). Google's robots.txt introduction says such a result “won't have a description”, and the Page indexing help says any snippet “will probably be very limited”.
Bing's current guidelines agree: “robots.txt controls crawl access, not indexing” (Bing Webmaster Guidelines, section 10). A separate Bing help page, How to create a robots.txt file, states: “Here, it is important to note that a page that is not crawled will not be indexed”, and links from that sentence to guidance on preventing indexing (Bing robots.txt help). Bing does not reconcile the two statements; the guidelines, and Bing's robots meta tag help (“we need to be able to crawl the page to see this tag, so do not block access to the page”), are the more specific instructions. For background, see our explainer Does blocking a crawler remove you from search?
Classification from Google Search Central documentation, not a measurement: full bar = Google says the URL can still appear
Source: Google Search Central, Block Search indexing with noindex and Introduction to robots.txt, read October 10, 2026.
Google says a blocked URL can be indexed when other pages link to it, and that noindex works only if robots.txt does not block the page.
Controlling Crawling and Server Load
robots.txt Disallow is the right tool for reducing crawling; noindex does not reduce it at all. Google's crawl budget guide tells large sites to block unimportant URLs, such as infinite scroll or differently sorted versions of a page, with robots.txt, and says: “Don't use noindex, as Google will still request, but then drop the page when it sees a noindex meta tag or header in the HTTP response, wasting crawling time” (Large site owner's guide to managing your crawl budget).
That guide is written for very large or fast-changing sites (Google names 1 million+ pages, or 10,000+ pages changing daily). Most company sites will not run short of crawl capacity, so for them this is a minor factor. Bing's guidelines likewise list “Block unnecessary crawl paths with robots.txt” under crawl efficiency, while warning against “Blocking important URLs unnecessarily”.
One caution from Google: do not disallow scripts or styles that pages need, because “Google Search won't render JavaScript from blocked files or on blocked pages” (JavaScript SEO basics).
PDFs, Images and Video
For non-HTML files, both methods work but do different jobs. A PDF cannot carry a meta tag, so noindex goes in the X-Robots-Tag header, which Google says “can be used for non-HTML resources, such as PDFs, video files, and image files”. Bing gives the same advice and adds that the file must not be blocked in robots.txt, “because Bing cannot read the header or confirm removal if it cannot reach the file” (Bing removal help).
For media in Google, robots.txt is documented to work: Google says it can be used “to prevent image, video, and audio files from appearing in Google Search results”, though it “won't prevent other pages or users from linking to” the file (Introduction to robots.txt). For PDFs and other text-based files, Google's warning against using robots.txt to hide pages applies.
Speed and Reversibility
Neither rule acts instantly, and both depend on the crawler coming back. A robots.txt change is picked up when the file is refetched: Google “generally caches the contents of robots.txt file for up to 24 hours” (How Google interprets the robots.txt specification), RFC 9309 says crawlers “SHOULD NOT use the cached version for more than 24 hours”, and Bing says changes “can take a few hours to be reflected”.
A noindex takes effect page by page, when each page is recrawled. Google: “Depending on the importance of the page on the internet, it may take months for Googlebot to revisit a page”; you can request a recrawl in URL Inspection (Block indexing with noindex). For urgent cases, Google's Removals tool and Bing's Block URLs tool (90 days, extendable) hide URLs temporarily while the permanent rule takes effect (Bing Block URLs); see how to remove outdated content from Google Search.
Privacy and Security
Neither rule protects anything. RFC 9309 states: “These rules are not a form of access authorization”, and in its security section warns that “Listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable” (RFC 9309). Google adds that well-behaved crawlers obey robots.txt but “other crawlers might not” (Introduction to robots.txt).
A noindexed page is still public to anyone with the link. For staging sites, client portals or internal documents, use password protection or HTTP authentication, which both Google and RFC 9309 point to.
Standards and Search-Engine Support
robots.txt has a formal standard; noindex does not. RFC 9309, published in September 2022 by Martijn Koster and three authors from Google LLC, “specifies and extends” the protocol “originally defined by Martijn Koster in 1994”. Its rules govern access to URIs; it defines no rule for keeping a page out of an index. noindex is defined in each search engine's documentation; Google warns that “some search engines might interpret the noindex rule differently”.
Both Google and Bing support both mechanisms. Watch for crawler-specific groups: Bing says that when Bingbot finds a group for itself it ignores the generic * group, so general rules must be repeated in a Bingbot group (Bing robots.txt help). The same applies to AI crawlers; our guide to blocking or allowing AI crawlers in robots.txt covers those tokens.
Why Combining Them Fails
Adding noindex to a page and also disallowing it in robots.txt is the most common mistake, because each rule stops the other working. Google: “If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored” (Robots meta tag specifications).
Search Console shows the effect. In URL Inspection, if a page is blocked by robots.txt, “Indexing allowed?” “will always be ‘Yes’ because Google can't see and respect any noindex directives. When this happens, your page might appear in Search results even if indexing is blocked by your site” (URL Inspection tool). To check a page, see how to check if a page is indexed by Google.
Where noindex Wins
- Removal from results: Google drops the page “regardless of whether other sites link to it”.
- Precision: works on one URL at a time without affecting neighbours.
- Bing and Copilot: Bing names NOINDEX for keeping URLs out of search, Copilot and grounding results.
- No root access: works through templates, CMS settings or headers.
- PDFs: via
X-Robots-Tag.
Where robots.txt Disallow Wins
- Crawl load: stops requests outright; Google's recommended tool for low-value URLs on large sites.
- Scale: one pattern covers thousands of faceted, sorted or search-result URLs.
- Media in Google: documented to keep images, video and audio out of Google Search.
- Formal standard: RFC 9309 defines parsing, errors and caching.
- Per-crawler control: separate groups for different user agents, including AI crawlers.
Who Should Choose noindex?
Best for: any page that should stay reachable for visitors but not appear in search results.
Starting price: free.
Primary strength: Google and Bing both say a crawled noindex keeps the page out of results, even if other sites link to it.
Main limitation: the page must stay crawlable, and the rule takes effect only when each page is recrawled, which Google says can take months for less important pages.
Verdict: the correct default for keeping pages out of search.
Choose noindex if: you have thank-you pages, internal search results, thin tag archives, gated-content landing pages or old press pages you want to keep online but out of Google and Bing.
Avoid noindex if: the content is confidential (use a password), the page has moved (use a 301 redirect), or your problem is crawl volume on a very large site.
Who Should Choose robots.txt Disallow?
Best for: sections crawlers should not request at all, such as endless filter and sort combinations, cart and session URLs, or media files you want out of Google.
Starting price: free.
Primary strength: stops crawling at the source with one file, under a formal standard.
Main limitation: it does not stop indexing; Google and Bing both say so.
Verdict: a crawl-management tool, not a removal tool.
Choose Disallow if: crawlers are wasting requests on URLs with no search value, your server struggles under crawl load, or you want specific crawlers kept out.
Avoid Disallow if: your goal is to get a page out of search results, the page already carries a noindex, or the blocked files are scripts and styles that pages need to render.
How to Choose Between noindex and robots.txt Disallow
| If you need… | Choose |
|---|---|
| A live page kept out of Google and Bing results | noindex |
| A page that is already indexed removed from results | noindex (crawlable), plus a temporary Removals or Block URLs request if urgent |
| A PDF kept out of results | X-Robots-Tag: noindex |
| Crawlers kept away from millions of filter URLs | robots.txt Disallow |
| Images or video kept out of Google Search | robots.txt Disallow |
| A specific AI or other crawler kept out | robots.txt group for that user agent |
| A duplicate page consolidated with its original | Neither; use rel="canonical" or a redirect |
| A deleted page gone from results | Neither; return 404 or 410 |
| Private content protected | Neither; use password protection |
Switching a Page From Disallow to noindex
If a page you want out of results is currently disallowed, reverse the order. Remove the Disallow rule that covers it; add noindex in the page or header; confirm with a live URL Inspection test that “Indexing allowed?” now reports the noindex; request a recrawl if the page is important; and leave crawling allowed so Google and Bing keep seeing the rule. Google's own fix for “Indexed, though blocked by robots.txt” is exactly this: “remove the robots.txt block and use 'noindex'”.
Final Verdict: noindex or robots.txt Disallow?
noindex is better for keeping pages out of search results, and robots.txt Disallow is better for controlling what crawlers request. They answer different questions, and the choice depends on one thing: whether your problem is a page appearing in results or crawlers spending requests where you do not want them. For most company sites the first problem is far more common, so noindex is the tool to reach for, with crawling left open so the rule can be read. Use Disallow for crawl waste, media files and specific crawlers, never on a page whose noindex you need search engines to see, and use neither for anything confidential.
Frequently Asked Questions
Can a page blocked by robots.txt still appear in Google?
Yes. Google says a disallowed URL can still be indexed if other pages link to it, and that such a result won't have a description. Search Console reports this as Indexed, though blocked by robots.txt.
Can I put noindex in robots.txt?
No. Google says specifying the noindex rule in the robots.txt file is not supported. Use a robots meta tag or an X-Robots-Tag HTTP header on the page or file itself.
Should I use noindex and Disallow together?
Not on a page you want out of search results. If robots.txt blocks the page, Google and Bing cannot crawl it and never see the noindex rule, so the URL can stay in results.
How long does noindex take to remove a page from Google?
It works when Google next crawls the page, which Google says can take months for less important pages. You can request a recrawl in URL Inspection, and use the Removals tool to hide the URL temporarily in the meantime.
Does noindex save crawl budget?
No. Google still has to request the page to see the rule, and its crawl budget guide tells large sites to use robots.txt rather than noindex for URLs they do not want crawled.
Does robots.txt Disallow keep content out of AI training?
Only for crawlers that honour robots.txt and are named or covered by your rules. RFC 9309 says the rules are not a form of access authorization, so crawlers that ignore them are not stopped.
Is noindex respected by Bing?
Yes. Bing supports noindex as a meta tag or X-Robots-Tag header, can be targeted with name=bingbot, and says a noindexed page will not appear in the Bing index once Bingbot recrawls it, provided the page is not blocked.
Should I noindex or Disallow a staging site?
Neither on its own. Both leave the content publicly reachable. Google and RFC 9309 both point to password protection or HTTP authentication for content that must stay private.
What does nofollow add to noindex?
nofollow tells Google not to follow the links on the page, and none is equivalent to noindex, nofollow. It does not change whether the page itself is indexed.
How We Compared Noindex and robots.txt Disallow
We compared the two mechanisms on the criteria above using Google Search Central and Search Console documentation, Bing Webmaster help and guidelines, and RFC 9309, all read on October 8, 2026. Quotations are verbatim. Where Bing pages disagree, we quote both and follow the more specific instruction. We did not test either mechanism on a live site for this comparison; “Winner” entries reflect what the documentation says each method does, not measured results.
- official Google Search Central, Block Search indexing with noindex. Read October 8, 2026.
- official Google Search Central, Robots meta tag, data-nosnippet, and X-Robots-Tag specifications. Read October 8, 2026.
- official Google Search Central, Introduction to robots.txt. Read October 8, 2026.
- official Google Search Central, How Google interprets the robots.txt specification. Caching and size limit. Read October 8, 2026.
- official Google Search Central, Large site owner's guide to managing your crawl budget. Read October 8, 2026.
- official Google Search Central, Understand the JavaScript SEO basics. Read October 8, 2026.
- official Google Search Console Help, Page indexing report. Read October 8, 2026.
- official Google Search Console Help, URL Inspection tool. Read October 8, 2026.
- official M. Koster, G. Illyes, H. Zeller and L. Sassman, IETF, RFC 9309: Robots Exclusion Protocol, September 2022. Read October 8, 2026.
- official Bing Webmaster Tools Help, Robots meta tags and attributes that Bing supports. Read October 8, 2026.
- official Bing Webmaster Tools Help, How to create a robots.txt file. Read October 8, 2026.
- official Bing Webmaster Tools Help, Bing Webmaster Guidelines, sections 8, 10 and 21. Read October 8, 2026.
- official Bing Webmaster Tools Help, How to Permanently Remove a URL or Page from Bing or Copilot. Read October 8, 2026.
- official Bing Webmaster Tools Help, How to Temporarily Block URLs from Bing and Copilot Search Results. Read October 8, 2026.