Internal hyperlinks

Dan Sharp

Posted 24 October, 2019 by in

Internal hyperlinks

Configuration > Spider > Crawl > Internal Hyperlinks

By default the SEO Spider will crawl and store internal hyperlinks in a crawl.

Internal is defined as URLs on the same subdomain as entered within the SEO Spider. Hyperlinks are URLs contained within HTML anchor tags.

By disabling ‘crawl’, URLs contained within anchor tags that are on the same subdomain as the start URL will not be followed and crawled.

Disabling both ‘store’ and ‘crawl’ can be useful in list mode, when removing the crawl depth. It allows the SEO Spider to crawl the URLs uploaded and any other resource or page links selected, but not anymore internal links.

For example, you can supply a list of URLs in list mode, and only crawl them and the hreflang links. Or you could supply a list of desktop URLs and audit their AMP versions only. You could upload a list of URLs, and just audit the images on them, or external links etc.


Dan Sharp is founder & Director of Screaming Frog. He has developed search strategies for a variety of clients from international brands to small and medium-sized businesses and designed and managed the build of the innovative SEO Spider software.

Comments are closed.

Back to top