Max page size (kb) to crawl

Dan Sharp

Posted 3 February, 2026 by in

Max page size (kb) to crawl

Configuration > Spider > Limits > Max Page Size (KB) to Crawl

Control the size of HTML pages crawled.

There’s a default is 50mb, which can be increased. If a page is extremely large in size, it can take a long time to process.

Dan Sharp is founder & Director of Screaming Frog. He has developed search strategies for a variety of clients from international brands to small and medium-sized businesses and designed and managed the build of the innovative SEO Spider software.

Like us on Facebook Connect with us on LinkedIn Follow us on X Subscribe to our channel us YouTube View our RSS feed

Join the mailing list for updates, tips & giveaways

How we use the data in this form

Get in touch

For an in-depth proposal on our services, complete our contact form to request a proposal.

Contact us

We'll get back to you asap.

Comments are closed.

Back to top