What is Noindex? (Meta Noindex, Robots Noindex)
Learn what noindex is, how it prevents search engine indexing, and its growing role in controlling AI crawler access to your content.
Noindex is an HTML meta tag or HTTP header that instructs search engine crawlers to exclude a page from their index. While the page can still be crawled and its links followed, it won't appear in search results. This directive has gained new significance as AI companies deploy crawlers to gather training data and power retrieval systems.
Why it matters
Noindex is a precision tool for controlling your search presence. Without it, you're at the mercy of search engines deciding which of your pages matter - and they often get it wrong, indexing login pages, internal search results, and parameter variations that cannibalize your real content. The stakes are higher now that AI systems are actively scraping the web. Your noindex strategy needs to account for both traditional search visibility and AI training data access. A page you never wanted ranking might end up informing how ChatGPT describes your brand. Understanding noindex - and its limitations - is essential for managing your content's presence across both search engines and AI systems.
What people get wrong
- Noindex prevents Google from seeing the page entirely. Google still crawls noindexed pages and follows their links. The page is processed and analyzed - it just won't appear in search results. This is why noindexed pages can still pass PageRank to pages they link to.
- Noindex and robots.txt disallow do the same thing. They're fundamentally different. Robots.txt blocks crawling entirely - Google never sees the content. Noindex allows crawling but prevents indexing. For maximum exclusion, you actually need robots.txt, not noindex.
- Adding noindex will quickly remove a page from search results. Google must recrawl the page to discover the noindex directive. For pages that are rarely crawled, this can take weeks or months. Request removal via Search Console for faster results.
Related terms
Robots.txt: Robots.txt blocks crawling at the directory level, while noindex allows crawling but prevents indexing of specific pages.
Indexing: Noindex is the primary mechanism for excluding pages from the indexing process while keeping them accessible to users.
AI Crawlers: AI crawlers like GPTBot often require separate blocking via robots.txt since they may not respect noindex meta tags.
Index Control Affects AI Visibility
Pages you've noindexed from traditional search can still influence AI responses if AI crawlers accessed them before blocking. Trakkr helps you understand where your brand actually appears in AI-generated responses, regardless of your indexing preferences - revealing gaps between your intended visibility strategy and reality.
Frequently Asked Questions
What is noindex?
Noindex is an HTML directive that instructs search engines not to include a specific page in their search index. The page can still be crawled and its content analyzed, but it won't appear in search results. It's implemented via a meta tag in HTML or an X-Robots-Tag HTTP header.
What's the difference between noindex and nofollow?
Noindex prevents a page from appearing in search results. Nofollow tells search engines not to pass ranking signals through the page's links. They serve different purposes and can be combined: a page can be noindexed but still pass link equity, or indexed but have nofollow links.
Does noindex block AI crawlers like GPTBot?
Generally no. AI crawlers from OpenAI, Anthropic, and others typically don't respect the noindex meta tag. They rely on robots.txt rules instead. To prevent AI crawlers from accessing content, you need to explicitly block them in your robots.txt file using their specific user-agent names.
How do I check if a page is noindexed?
View the page source and search for 'noindex' in the meta robots tag. You can also use Google Search Console's URL Inspection tool, which shows whether Google can index a page and what directives it found. Browser extensions like SEO Meta in 1 Click surface this information automatically.
Should I noindex or delete old content?
It depends on user value. If the content still serves visitors (old event info, archived articles), noindex keeps it accessible while preventing search competition. If it's truly obsolete, deletion with proper redirects is cleaner. Noindex isn't a substitute for content pruning - it's a tool for pages that serve users but not search.