What the message means

In Search Console’s URL Inspection you see:

Indexing allowed? No: ‘noindex’ detected in ‘X-Robots-Tag’ http header

and in the Page indexing report the URL sits under “Excluded by ‘noindex’ tag”. Google fetched the page, and the HTTP response came back with a header like this:

X-Robots-Tag: noindex

Google treats that exactly like <meta name="robots" content="noindex"> in the HTML: it won’t show the page in search. The difference is that you can’t see it. The header travels with the response, before the HTML, and View Source doesn’t show it. That’s why so many people stare at a perfectly normal page and can’t find the problem.

X-Robots-Tag vs meta robots

Both carry the same directives (noindex, nofollow, none, nosnippet, unavailable_after and so on). A few things are specific to the header:

  • It works on any file type. PDFs, images and other non-HTML files can only be kept out of the index with the header, since they have no <head>.
  • It can target one crawler. X-Robots-Tag: googlebot: noindex applies only to Google. Without a crawler name, it applies to all of them.
  • Multiple headers combine. A response can carry several X-Robots-Tag headers, and they’re all read together with any meta robots tag in the HTML.
  • The most restrictive rule wins. If one layer says noindex and another says index, follow, the page is not indexed. index is the default anyway; it doesn’t cancel anything. none means noindex, nofollow.

So the fix is never “add an index header somewhere”. It’s finding the thing that adds noindex and stopping it.

Step 1: look at the header yourself

curl -sI https://example.com/page | grep -i '^x-robots-tag'

-I sends a HEAD request. Some applications answer HEAD differently from GET, so if you get nothing, check with a real GET and print only the headers:

curl -s -o /dev/null -D - https://example.com/page

Then repeat it as Googlebot. Bot management, WAF rules, A/B tools and some plugins treat crawlers differently, and the noindex may only appear for them:

curl -s -o /dev/null -D - \
  -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  https://example.com/page

If the URL redirects, add -L and read every block of headers. What matters for indexing is the final URL, the one you expect Google to index.

The HTTP check shows the X-Robots-Tag value on the final response after following redirects, from outside your network. It uses its own user agent, so for rules that only fire on Googlebot, the curl command above is the test.

If none of this shows a noindex but Search Console still does, two explanations cover most cases:

  • The inspection is showing an old crawl. The default URL Inspection view reports what Google saw on its last crawl. Click “Test live URL” to fetch the page now.
  • Google gets a different response from yours. Rules keyed on Google’s IP ranges or verified-bot status won’t trigger for a curl from your laptop even with the right user agent. The live test is the only view from Google’s side. Under “View tested page”, the “More info” tab lists the HTTP response headers Google received.

Step 2: find which layer adds it

A response passes through several layers, and any of them can add a header. Work from the outside in.

Skip the CDN

Ask the origin directly, with the right hostname, using --resolve to override DNS:

curl -s -o /dev/null -D - --resolve example.com:443:203.0.113.10 https://example.com/page

If the origin response has no X-Robots-Tag but the public one does, the CDN or edge adds it. If the origin already has it, keep going inward.

CDN and edge rules

  • Cloudflare: Transform Rules that modify response headers, any Worker on the route, and the _headers file if the site is on Cloudflare Pages.
  • Netlify: the _headers file and [[headers]] sections in netlify.toml.
  • Vercel: headers in vercel.json, or headers() in next.config.js for Next.js.
  • Any other CDN with header rules or edge functions: search its config for robots.

One more CDN trap: cached responses keep their headers. If the origin once sent noindex and the CDN cached that response, fixing the origin isn’t enough. Purge the cache for the affected URLs.

Hosting preview deployments

Hosting platforms add X-Robots-Tag: noindex to preview builds on purpose, so test deployments don’t end up in Google. Vercel does this for preview deployments, Netlify for Deploy Previews, and Cloudflare Pages for preview deployments on pages.dev.

That’s correct behaviour. It turns into a problem when your public hostname is actually serving a preview: a custom domain assigned to the wrong branch or deployment, or a site that’s still reachable mainly through its preview URL. Check which deployment the domain points at in the platform’s domain settings.

Web server configuration

nginx:

add_header X-Robots-Tag "noindex, nofollow" always;

Search the full running configuration, includes and all:

sudo nginx -T 2>/dev/null | grep -n -i 'x-robots-tag'

Watch nginx’s inheritance rule: a location block inherits add_header lines from the server block only if it has none of its own. A noindex can be sitting in one location and nowhere else, and removing it from the server block won’t touch it.

Apache:

Header set X-Robots-Tag "noindex, nofollow"

It can live in the virtual host, a shared include or any .htaccess file in the document root:

grep -rni 'x-robots-tag' /etc/apache2 /etc/httpd /var/www 2>/dev/null

Configs copied from a staging server are a classic source. Someone added a blanket noindex so staging stayed out of Google, and the file came along to production.

The application

If the web server config is clean, the application sets the header itself. Search the code:

grep -rni 'x-robots-tag' --exclude-dir=node_modules --exclude-dir=.git .

The usual pattern is an environment check: middleware that adds noindex unless NODE_ENV or APP_ENV equals production, and the production environment variable is spelled differently, missing, or set to prod. Also check CMS and SEO plugin settings. In WordPress, the “Discourage search engines from indexing this site” option under Settings → Reading adds a meta robots noindex rather than a header, but it gets left on after launch for the same reason. SEO plugins can mark whole post types or archives as noindex.

A noindex on your XML sitemap files is normal. Some SEO plugins send it on purpose to keep the sitemap file itself out of search results; it applies to that file, not to the URLs listed in it.

robots.txt makes it worse, not better

A common reflex is to also block the URL in robots.txt. Don’t. A Disallow stops Google from fetching the page, so it can never see whether the header is still there or gone. It also doesn’t keep a page out of the index: a blocked URL can still be indexed from links pointing at it, just without its content.

The rule is simple: if you want Google to obey a noindex, or notice that one was removed, the URL must be crawlable.

Step 3: remove it and verify

  1. Remove the directive at the layer that adds it. Don’t add a competing header.
  2. Purge the CDN cache for the affected URLs.
  3. Re-run the curl commands, with and without the Googlebot user agent. No X-Robots-Tag header, or one without noindex, is what you want.
  4. In URL Inspection, run “Test live URL”. “Indexing allowed?” should now say Yes.
  5. Click “Request indexing” for the URLs that matter most. It has a daily quota, so don’t spend it on hundreds of pages.
  6. If many pages were affected, open the “Excluded by ‘noindex’ tag” group in the Page indexing report and use “Validate fix”. Make sure those URLs are in your sitemap with an updated lastmod.

How long until the page comes back

Google has to recrawl the page before the change counts. How soon that happens depends on how often Google crawls your site, which you can’t set directly. Google’s own guidance is that crawling can take anywhere from a few days to a few weeks. Request indexing and sitemaps help Google find the URLs again; they don’t guarantee when or whether a page is indexed.

If you actually want noindex somewhere

For staging and preview environments, password protection or IP allowlisting is more robust than noindex. A noindex is one config line away from leaking into production, which is how many of these incidents start. Authentication keeps the content private too.

For files you never want in search, such as PDFs in a downloads folder, a scoped rule is fine:

location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex" always;
}

Keep it scoped to exactly what you mean, and re-check with curl after every config change that touches headers.

Check your own domain now

Free, no sign-up. Runs the exact check this guide describes and shows what to fix.