I am really excited about the latest release of the Screaming Frog SEO spider, version 1.60. It’s without a doubt our biggest update since we launched the tool about 6-months ago and a number of the new features have come from feedback and suggestions from users. So thanks to everyone who taken the time to e-mail, tweet or grab me at a conference with suggestions! It’s genuinely appreciated and really helps us improve the tool.

So, lets take a look at what’s new in version 1.60 -

  • Crawling At Scale – We have a couple of really big features in this release, but probably the update we are most proud of is actually the performance of the spider crawling websites at scale. In old versions of the spider, you could experience slow down of crawl when crawling a significant number of URI. We have worked really hard to optimise the performance here (so CPU use is less intensive!) and it’s now possible to crawl a much larger number of URI, without any speed slowdown. As an example, by upgrading the RAM allocation to 1,500mb the spider easily crawled over half a million pages of the BBC news section. With further RAM allocation, we were able to crawl into millions.
  • Save Crawl Function – We now have a save crawl function. This was probably our most requested feature. It allows you to save a crawl at anytime and re-open the .seospider file in the SEO spider and open or resume the crawl. Thanks to everyone who suggested this feature, I believe it was Ian Macfarlane who was the first in particular.
  • Bulk Export Of Links – The SEO spider stores a huge amount of data on linkage of a site. So, we made this available in the new export function which allows you to export every single instance of a link found by the spider, the source, the destination, alt or anchor text and status code. So for example, you can now export all inlinks to all pages which 404 error instead of exporting individually for each URI. You can now get extremely granular data of linking relationships on a site, so I will be interested to see how people use this data.

While these were the main updates, we also added some other very cool features -

  • Search Function – You can now search for anything in the spider. This is case sensitive and works on all columns in the tab you are viewing. So switch to the appropriate tab if it’s something specific in a page title or meta description etc.
  • Gzip content encoding was improved
  • Support of proxy and web server authentication – We support both Basic and Dgest authentication
  • Smart file names for exporting data
  • Remember last access folder for file open/save and exporting
  • Included a open recent saved crawls option
  • Included a recent crawled domains option. Thanks to Rob Nicholson for the suggestions here (and more to come in this area).

We also fixed a few bugs for -

  • Fixed crash with lower window pane menu right click
  • A parsing issue for relative urls starting with ?
  • A bug effecting urls linked to with an umlaut
  • Fix for commas in urls when uploading list files

We still have a huge ‘todo’ list for the spider and continue to get great feedback from users. However, if there is anything you would love to see, then please leave your feedback below. If we don’t already have it on the list, we will add anything with real value.

Thanks again to everyone for all their support of the tool!

screamingfrog (25 Posts)

Dan Sharp is founder & Director of Screaming Frog. He has developed search strategies for a variety of clients from international brands to small and medium sized businesses and designed and managed the build of the innovative SEO Spider software.