Limit crawl total per subdomain

Dan Sharp

Posted 20 October, 2025 by in

Limit crawl total per subdomain

Configuration > Spider > Limits > Limit Crawl Total Per Subdomain

Control the number of URLs that are crawled for each subdomain.

For example, if this was set at ‘100’ and two subdoamins uk.example.com and us.example.com are encountered, then 100 URLs for each would be crawled.

Dan Sharp is founder & Director of Screaming Frog. He has developed search strategies for a variety of clients from international brands to small and medium-sized businesses and designed and managed the build of the innovative SEO Spider software.

Like us on Facebook Connect with us on LinkedIn Follow us on X Subscribe to our channel us YouTube View our RSS feed

Join the mailing list for updates, tips & giveaways

How we use the data in this form

Get in touch

For an in-depth proposal on our services, complete our contact form to request a proposal.

Contact us

We'll get back to you asap.

Comments are closed.

Back to top