I am excited to announce an update to the Screaming Frog SEO spider, to version 1.80.

The following new features are included in the update -

  • Crawl Speed Control – The ability to control the speed of the crawl is something we have wanted to release for a long time. We have had some requests for throttle control due to the SEO spider crawling too fast for some older servers. Hence, we have introduced a ‘speed’ feature which allows you to limit the number of threads or the number of URI requests per second. This feature also allows you to significantly increase the speed of the SEO spider at the same time, which we just ask everyone uses responsibly :-).
  • XML Sitemaps – We have included the ability to create XML sitemaps in the SEO spider under the top level menu ‘export’ option. Currently this feature is only for HTML pages, so it does not include images, videos etc (which we will be introducing at a later date). We conform to the standards outlined in sitemaps.org protocol. If you have over 49,999 urls the SEO spider will automatically create additional sitemap files and create a sitemap index file referencing the sitemap locations.
  • Exclude Is Now Regex – Much like the ‘include’ feature, the ‘exclude’ option in the SEO spider is now regex. In previously versions it was robots.txt syntax, but regex provides more control and better balance.
  • Bulk Export Missing Alt Text URI – We have included the ability to bulk export all images with missing alt text and the URI that reference them.
  • List Mode Accepts XML Sitemap Files – We added the ability in ‘list’ mode to be able to upload and crawl .xml files. Hence, if you already have an XML sitemap and wish to audit it, simply switch to ‘list’ mode, change the filetype to ‘.xml’ and upload it. We will discover all the URLs automatically and crawl them.

We also fixed up a couple of bugs -

  • Level – We did not always find the shortest path. Now we do.
  • Https – There was a bug crawling secure pages in version 1.70. This has been fixed!

So, what are you waiting for? Go and download it now!

As always, please just let us know if anyone has any queries or feedback about the above. I will be updating the user guide this week (so please bare with me!).

Thanks again to everyone for all their support of the tool!

screamingfrog (29 Posts)

Dan Sharp is founder & Director of Screaming Frog. He has developed search strategies for a variety of clients from international brands to small and medium sized businesses and designed and managed the build of the innovative SEO Spider software.