Memory allocation
Dan Sharp
Posted 27 February, 2018 by Dan Sharp in
Memory allocation
File > Settings > Memory Allocation (Windows, Linux)
Screaming Frog SEO Spider > Settings > Memory Allocation (macOS)
The SEO Spider uses Java which requires memory to be allocated at start-up. By default the SEO Spider will allow 1gb for 32-bit, and 2gb for 64-bit machines.
Increasing memory allocation will enable the SEO Spider to crawl more URLs, particularly when in RAM storage mode, but also when storing to database.
We recommend setting the memory allocation to at least 2gb below your total physical machine memory so the OS and other applications can operate.

If you’d like to find out more about crawling large websites, memory allocation and the storage options available, please see our guide on crawling large websites.