What Is a Robots Meta Tag, in Its Defining Sources' Words?
No standards body owns the robots meta tag. The Web Robots Pages, the long-standing reference site for robots.txt, describe it as “a special HTML <META> tag to tell robots not to index the content of a page, and/or not scan it for links to follow.” The same page says that “Like the /robots.txt, the robots META tag is a de-facto standard. It originated from a ‘birds of a feather’ meeting at a 1996 distributed indexing workshop” (robotstxt.org).
MDN Web Docs gives the current developer definition: the robots value for the name attribute of a <meta> element “defines the crawl and indexing behavior that web crawlers should use with the page” (MDN). Google's specification describes it as a way to take “a granular, page-specific approach to controlling how an individual HTML page should be indexed and served to users in Google Search results” (Google).
Robots Meta Tag at a Glance
| Question | Answer | Source |
|---|---|---|
| What it looks like | <meta name="robots" content="noindex"> in the page head | |
| Who reads it | Cooperative crawlers; MDN notes “Only cooperative robots follow these rules” | MDN |
| Origin | A 1996 workshop meeting; a de facto standard | robotstxt.org |
| HTTP equivalent | X-Robots-Tag response header, for PDFs, images and other files | |
| Rules Google uses | 12 valid rules, including noindex, nofollow, nosnippet and max-snippet | |
| Requires crawling | Yes: a page blocked by robots.txt cannot have its tag read | |
| Related terms | robots.txt, X-Robots-Tag, data-nosnippet, rel="nofollow", canonical | See table below |
How the Tag Is Written
The tag has two attributes. The name attribute says which crawler the rule is for, and the content attribute holds one or more comma-separated rules. Google says “Both the name and the content attributes are case-insensitive.” Using robots addresses all crawlers; to address Google alone, use a user agent token. Google says it “supports two user agent tokens in the robots meta tag; other values are ignored”: googlebot for all text results and googlebot-news for news results.
<meta name="robots" content="noindex, nofollow">
<meta name="googlebot-news" content="nosnippet">Google expects the tag in the head, but notes that it “will respect robots meta tags in the body section of an HTML document as well.” For files that have no HTML head, the same rules go in an HTTP header: “Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag”, for example X-Robots-Tag: noindex. A CMS may offer a settings page in place of editing HTML.
The Rules Google Recognizes
Google's specification lists these as valid indexing and serving rules, and warns that “It is possible that these rules may not be treated the same by all other search engines.”
| Rule | What Google says it does |
|---|---|
| all | No restrictions; “the default value and has no effect if explicitly listed” |
| noindex | “Do not show this page, media, or resource in search results” |
| nofollow | “Do not follow the links on this page” |
| none | “Equivalent to noindex, nofollow” |
| nosnippet | No text snippet or video preview; also applies to AI Overviews and AI Mode |
| indexifembedded | Allows indexing when embedded via iframes, despite noindex; only works with noindex |
| max-snippet: [number] | Caps the text snippet at that many characters; 0 means no snippet, -1 lets Google choose |
| max-image-preview: [setting] | none, standard or large image preview |
| max-video-preview: [number] | Caps video previews at that many seconds; 0 allows at most a static image, -1 no limit |
| notranslate | Do not offer translation of the page in results |
| noimageindex | “Do not index images on this page” |
| unavailable_after: [date/time] | Do not show the page after the given date and time |
Number of rules listed, by status
Source: Google Search Central, Robots meta tag, data-nosnippet, and X-Robots-Tag specifications, last updated March 24, 2026, read October 10, 2026.
Valid rules: all, noindex, nofollow, none, nosnippet, indexifembedded, max-snippet, max-image-preview, max-video-preview, notranslate, noimageindex, unavailable_after. Ignored: noarchive, nocache, nositelinkssearchbox. Tokens: googlebot and googlebot-news; Google says other values are ignored.
The three ignored rules are worth knowing because they still appear in templates. Google says noarchive “is no longer used by Google Search to control whether a cached link is shown in search results, as the cached link feature no longer exists”, nocache “isn't used by Google Search”, and nositelinkssearchbox no longer applies because the feature is gone. MDN's keyword list still names noarchive as used by Google, which shows why the search engine's own page is the one to trust.
Classification from Google's robots meta specification, not a measurement
Source: Google Search Central, Robots meta tag, data-nosnippet, and X-Robots-Tag specifications, read October 10, 2026.
Ranks how much of a page's presence in Google Search each rule removes, as described in the specification. Ordinal categories, not measured quantities.
What Bing Supports
Bing reads the same tag. In an April 2020 announcement, Bing said it was “extending our support for robots meta tags in HTML or X-Robots-Tag tag in the HTTP Header” to add max-snippet, max-image-preview and max-video-preview, and that “the NOSNIPPET meta tag is still supported” (Bing Webmaster Blog). Bing's example combined those rules with noarchive, which it described as meaning “link to no cache page should be shown”. The post added: “We will use these options as directive statement, not as hints.” Bing's current help page on supported robots meta tags did not load for us without JavaScript, so we cannot list its full current set here.
Combining Rules and Resolving Conflicts
Rules can be combined in one tag or spread across several. Google gives two conflict rules. Within its own reading: “In the case of conflicting robots rules, the more restrictive rule applies. For example, if a page has both max-snippet:50 and nosnippet rules, the nosnippet rule will apply.” Across crawler names: “the search engine will use the sum of the negative rules”, so a page with name="robots" content="nofollow" and name="googlebot" content="noindex" is read by Googlebot as noindex, nofollow. MDN adds that for mutually exclusive values such as index and noindex, “Behavior is undefined when conflicting values are used.”
For text-level control, the data-nosnippet attribute marks parts of a page that should not appear in snippets. Google says it works on span, div and section elements. Our guide to controlling snippets and AI Overviews walks through nosnippet, max-snippet and data-nosnippet with examples.
The robots.txt Trap
A robots meta tag only works if the crawler can fetch the page. Google states: “If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored.” Its noindex page adds that such a page “can still appear in search results, for example if other pages link to it” (Google). The two tools do different jobs, compared in noindex vs robots.txt disallow and does blocking a crawler remove you from search?.
The reverse also holds: a robots meta tag does not reduce crawling. Google's crawl budget guide says it “will still request, but then drop the page when it sees a noindex meta tag”. See what a crawl budget is.
Robots Meta Tags and AI Features
Google's specification now names its AI features. It says nosnippet “applies to all forms of search results (at Google: web search, Google Images, Discover, AI Overviews, AI Mode) and will also prevent the content from being used as a direct input for AI Overviews and AI Mode.” max-snippet likewise “will also limit how much of the content may be used as a direct input for AI Overviews and AI Mode.” Neither rule addresses model training by other companies; for that, see can you tell an AI not to train on your site? and blocking or allowing AI crawlers in robots.txt.
How to Check a Page's Robots Meta Tag
- View the source and search for name="robots" or name="googlebot". Check response headers for X-Robots-Tag, which will not appear in the HTML.
- Use the URL Inspection tool in Search Console. Google's Page indexing help says that under Indexing allowed? the report should show whether noindex is preventing indexing, and that Test live URL checks the current version.
- Review the Page indexing report for the status “URL marked ‘noindex’”. Google's note on it reads: “If you do not want this page indexed, congratulations!”
- Wait for a recrawl. Google says it “may take months for Googlebot to revisit a page” in some cases; see how to check if a page is indexed.
Robots Meta Tag vs Related Terms
| Term | What it is | Relationship to the robots meta tag |
|---|---|---|
| robots.txt | A site-wide file listing URLs crawlers should not fetch | Controls crawling; the meta tag controls indexing and display |
| X-Robots-Tag | An HTTP response header | Carries the same rules, including for non-HTML files |
| data-nosnippet | An HTML attribute on span, div or section | Text-level snippet control inside a page |
| rel="nofollow" | A link attribute on a single link | Distinct from the page-wide nofollow rule; see what a nofollow link is |
| rel="canonical" | A link to the preferred version of a page | Consolidates duplicates; see canonical tag vs noindex |
Common Misconceptions About Robots Meta Tags
| Myth | What the source actually says | Source |
|---|---|---|
| A robots meta tag blocks crawling | It is read only after crawling; MDN suggests robots.txt to reduce bandwidth | MDN |
| noindex plus a robots.txt block is extra safe | A robots.txt block means the noindex is never seen | |
| noarchive still matters for Google | Google says it no longer uses noarchive because the cached link feature no longer exists | |
| All bots obey it | Malware robots and email harvesters “will pay no attention” | robotstxt.org |
| The tag must be in the head or Google ignores it | Google also respects robots meta tags in the body |
Frequently Asked Questions
What is a robots meta tag in simple terms?
It is a line of HTML in a page that tells search engine crawlers whether to index the page, follow its links or show a snippet. The most common form is a meta tag named robots with the content noindex.
Where do I put the robots meta tag?
In the head section of the HTML page. Google says it also respects the tag in the body. For PDFs, images and other non-HTML files, use the X-Robots-Tag HTTP header instead.
What is the difference between noindex and nofollow?
Noindex tells search engines not to show the page in results. Nofollow tells them not to follow the links on the page. The value none combines both, according to Google.
Does the robots meta tag work if robots.txt blocks the page?
No. Google says that if a page is disallowed in robots.txt, its indexing and serving rules will not be found and will be ignored, and the URL can still appear in results if other pages link to it.
Does Google still support noarchive?
Google lists noarchive as a historical rule it no longer uses, because the cached link feature no longer exists. Bing's 2020 announcement still used noarchive in an example.
Can a robots meta tag keep my content out of AI Overviews?
Google says nosnippet prevents content from being used as a direct input for AI Overviews and AI Mode, and max-snippet limits how much can be used. Neither addresses training by other AI companies.
How do I check whether a page has a noindex tag?
Search the page source and response headers for noindex, then run the URL Inspection tool in Search Console, which shows whether indexing is allowed, and test the live URL.
Sources and Methodology
This explainer is built from Google's robots meta tag specification and noindex page, the Search Console Page indexing and crawl budget documentation, Bing's April 2020 announcement, MDN Web Docs and the Web Robots Pages, read on October 10, 2026. Definitions and rules are quoted verbatim. No standards body defines the tag, so the rule list is Google's, and other engines may differ. Bing's current help page would not load without JavaScript, so Bing support is reported only from its 2020 announcement. We did not test how any search engine responds to each rule.
- official Robots meta tag, data-nosnippet, and X-Robots-Tag specifications, Google Search Central (last updated March 24, 2026). Read October 10, 2026.
- official Block Search indexing with noindex, Google Search Central (last updated December 10, 2025). Read October 10, 2026.
- official Announcing new options for webmasters to control their snippets at Bing, Bing Webmaster Blog, Fabrice Canel, April 8, 2020. Read October 10, 2026.
- reference <meta name="robots">, MDN Web Docs. Read October 10, 2026.
- reference About the Robots <META> tag, The Web Robots Pages (robotstxt.org). Origin and de facto status. Read October 10, 2026.
- official Page indexing report, Search Console Help. Read October 10, 2026.
- official Optimize your crawl budget, Google crawling documentation (last updated July 22, 2026). Read October 10, 2026.