JavaScript: Noindex Only in Original HTML

Pages that contain a noindex in the raw HTML, and not in the rendered HTML.

When Googlebot encounters a noindex tag, it skips rendering and JavaScript execution.

Because Googlebot skips JavaScript execution, using JavaScript to remove the ‘noindex’ in the rendered HTML won’t work.

How to Analyse in the SEO Spider

Enable JavaScript rendering mode via ‘Config > Spider > Rendering’ and select ‘JavaScript’ to crawl JavaScript websites.

View URLs with this issue in the ‘JavaScript’ tab and ‘Noindex Only in Original HTML’ filter and export all URLs using the ‘Export’ button.

The ‘HTML Meta Robots’ column shows the robots directives found in the original HTML before JavaScript.

The ‘Rendered HTML Meta Robots’ column shows robots directives found in the rendered HTML after JavaScript has been processed.

Read our tutorial on ‘How To Crawl JavaScript Websites‘.

What Triggers This Issue

This issue is triggered when pages have a meta robots ‘noindex’ directive in the raw HTML, which is then removed or not present in the rendered HTML after JavaScript execution.

For example:

https://www.screamingfrog.co.uk/

Has the meta tag:

<meta name="robots" content="noindex">

Which is then removed after JavaScript is executed.

How To Fix

Carefully review pages with noindex in the raw HTML are expected to not be indexed. Remove the ‘noindex’ if the pages should be indexed.