The meta robots tag is a small piece of HTML sitting in a page's head section that controls two separate things: whether that specific page gets indexed by search engines, and whether the links on it should be followed and pass authority onward. It is frequently confused with robots.txt, but the two operate at completely different levels — robots.txt controls crawling site-wide via a separate file, while the meta robots tag controls indexing and link-following on a per-page basis, directly in each page's own code. Understanding the distinction, and the specific directives available, prevents both missed opportunities and genuinely damaging mistakes.
Meta robots vs robots.txt: two different jobs
This distinction trips up a lot of people, so it is worth stating plainly: robots.txt tells crawlers which paths they are allowed to crawl at all — if a page is disallowed there, crawlers generally never even read its content, including its meta robots tag. The meta robots tag, by contrast, is read only once a crawler has already been allowed to access and read the page, and it then instructs what to do with that specific page once read — index it or not, follow its links or not. A page you want indexed must be crawlable (not blocked in robots.txt) and carry index-permitting meta robots directives (or none at all, since indexing is the default) — blocking it at either level prevents indexing, but for different underlying reasons.
The core directives
index / noindex controls whether the page should appear in search results at all — "noindex" is the directive to actively keep a specific page out of search results while still allowing it to be crawled and its links followed (useful for a thank-you page, an internal search results page, or content you want live on the site but not surfaced in search). follow / nofollow controls whether the links on that page should be followed and pass ranking signal onward — "nofollow" tells search engines not to pass authority through the page's outbound links, commonly used for user-generated content or paid/sponsored links where you do not want to implicitly vouch for the destination. The Meta Robots Tag Generator builds the correctly formatted tag from whichever combination of directives you select.
Common, sensible combinations
Most real-world pages use one of a few standard combinations. The default (no meta robots tag at all, or explicitly "index, follow") applies to the vast majority of a site's normal, public pages — you want them indexed and their links followed. "Noindex, follow" suits a page you do not want appearing in search results but whose links should still pass value onward — a paginated archive page, for instance, where you want the individual articles it links to indexed even though the archive listing page itself is not a useful search result. "Noindex, nofollow" is the strongest exclusion, appropriate for genuinely low-value pages like internal admin areas, duplicate filtered views, or thank-you pages you want completely out of the search ecosystem's consideration.
The catastrophic mistake: accidentally noindexing a live site
Just as with robots.txt, the single most damaging meta-robots mistake is a "noindex" directive left over from a staging or development environment that makes it into the production site — every page silently carries an instruction telling search engines to remove it from the index, with no visible symptom until organic traffic mysteriously collapses over the following days and weeks as search engines act on the directive. This is arguably even more insidious than a robots.txt mistake, because it can be set differently per page or per template, meaning it might affect only certain sections of a site rather than announcing itself as an obvious, uniform, site-wide problem — always specifically check the meta robots tag on production pages after any deployment involving templates or configuration that might set it.
Noindex vs removing content entirely
A subtlety worth understanding: "noindex" does not delete a page or make it inaccessible to visitors who have the direct URL — it only asks search engines not to include it in their index. If you genuinely want content gone from the internet entirely, noindex is the wrong tool; you need to actually remove or restrict access to the content itself. Conversely, if you want a page to remain live and usable (for existing bookmarks, direct links, or internal navigation) but simply not show up when people search, "noindex" is exactly the right, lightweight tool — it achieves that specific goal without the disruption of actually taking the page down.
International sites and hreflang
For sites serving multiple languages or regions, a related but distinct piece of technical SEO is the hreflang tag, which tells search engines which language or regional version of a page to show to users in different locations — it operates alongside, not instead of, the meta robots tag, since hreflang addresses "which version for which audience" while meta robots addresses "should this specific page be indexed at all." The Hreflang Tag Generator builds the correct set of reciprocal tags needed across all language/region variants of a page, a detail that is easy to get subtly wrong by hand since every variant needs to reference every other variant correctly.
Checking your directives are correct
Because meta robots mistakes are both easy to make and slow to notice (search engines act on a noindex directive gradually, not instantly), it is worth periodically auditing key pages — especially templates that apply site-wide — to confirm the intended directives are actually present and correct. Combining this check with a broader technical audit, such as working through a Page Speed Checklist, catches multiple categories of quiet, high-impact technical issues in one pass rather than discovering them separately, weeks apart, each time organic performance unexpectedly dips.
Key takeaways
- Robots.txt controls crawling site-wide; the meta robots tag controls indexing and link-following per page — different jobs, both needed correctly.
- "Noindex" keeps a page out of search results while still allowing it to be crawled and its links followed (unless paired with "nofollow").
- A stray "noindex" left over from staging is a common, slow-to-notice mistake — always verify after deployments.
- Noindex does not delete or restrict a page — it only asks search engines not to index it; use hreflang separately for multi-language sites.