Ignore non-indexable URLs for Issues

Dan Sharp

Posted 23 November, 2020 by in

Ignore non-indexable URLs for Issues

Configuration > Spider > Advanced > Ignore Non-Indexable URLs for Issues

When enabled, the SEO Spider will only populate issue-related filters if the page is Indexable. This includes all filters under Page Titles, Meta Description, Meta Keywords, H1 and H2 tabs and the following other issues –

  • ‘Low Content Pages’ in the Content tab.
  • ‘Missing’, ‘Validation Errors’ and ‘Validation Warnings’ in the Structured Data tab.
  • ‘Orphan URLs’ in the Sitemaps tab.
  • ‘No GA Data’ in the Analytics tab.
  • ‘No Search Analytics Data’ in the Search Console tab.
  • ‘Pages With High Crawl Depth’ in the Links tab.

E.g. This means URLs won’t be considered as ‘Duplicate’, or ‘Over X Characters’ or ‘Below X Characters’ if for example they are set as ‘noindex’, and hence non-indexable.

We recommend disabling this feature if you’re crawling a staging website which has a sitewide noindex.


Dan Sharp is founder & Director of Screaming Frog. He has developed search strategies for a variety of clients from international brands to small and medium-sized businesses and designed and managed the build of the innovative SEO Spider software.

Comments are closed.

Back to top