Screaming Frog SEO Spider Update – Version 1.70

Dan Sharp

Posted 15 August, 2011 by in Screaming Frog SEO Spider

Screaming Frog SEO Spider Update – Version 1.70

I am excited to announce another update to the Screaming Frog SEO spider, to version 1.70.

So lets get straight to it, what new features are there in version 1.70 of the tool?

    • Include (REGEX Crawling) – This new feature allows you to control URLs the spider crawls using REGEX. It narrows the default search by only crawling the URLs that match the REGEX which is particularly useful for larger sites, or sites with less intuitive URL structures.

 

    • User Agent Switcher – The SEO spider now have a user-agent switcher with inbuilt preset user agents for Googlebot, Bingbot, Yahoo! Slurp, various browsers and more. We also have a custom user-agent feature which allows you to specify your own user agent. This feature is only available to licensed users and we ask everyone to use this responsibly.

 

    • X-Robots-Tag & Canonical HTTP Headers – The SEO spider now offers support for both the X-robots-tag and canonical http headers.

 

  • Re-Spider URLs – You can now re-spider urls with the right click function. This is particularly useful if you’ve had connection time outs, unexpected responses or you wish to update some URLs when editing the site itself. You can use the control and shift functions to highlight multiple URLs to re-spider. So for example, you could simply go to the ‘response codes’ tab, click the ‘No response’ filter, press ‘control a’ to highlight all the URLs, right click and re-spider to crawl them all again.

We also fixed a few bugs around around case sensitivity of content type, a meta refresh display bug, crawl control for some domain extensions (.com.au etc) and the trailing slash on the root page between host and file path.

We hope everyone enjoys the new features in the tool, as always please do just let us know if you have any feedback or encounter any bugs. To download the new version of the Screaming Frog SEO spider, simply navigate to ‘help’ and ‘check for updates’ in the interface or download it here.

Dan Sharp is founder & Director of Screaming Frog. He has developed search strategies for a variety of clients from international brands to small and medium-sized businesses and designed and managed the build of the innovative SEO Spider software.

4 Comments

  • Dan Nolan 13 years ago

    Love the updates, as always. Does the ‘Include’ work just like the ‘exclude’ regular expressions?

    I seemed to have more control over this before this feature was added, and I’m concerned I’m doing it wrong. I’ve had great success with the ‘exclude’ part…just can’t get ‘include’ to work.

    Do you have any guides on this? Thanks!

    Reply
  • screamingfrog 13 years ago

    Hi Dan,

    Glad you like the update :-)

    The include feature works a little differently to exclude at the moment.

    Include literally is regex (SEOmoz posted a useful guide just the other day – http://www.seomoz.org/blog/an-seos-guide-to-regex), while exclude is a similar protocol to robots.txt (you can read more about it here – https://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/#9).

    I will be updating our own user guide this weekend with more details on how it can be used! Sorry, we were a bit delayed updating it.

    Thanks,

    Dan

    Reply
  • immobilier grenoble 13 years ago

    is SF compliant with 10.5.8 ? i get a message with java revision … ?!

    thanks

    AC

    Reply
  • screamingfrog 13 years ago

    Hi AC,

    I am not entirely sure off the top of my head, but if you’re getting a Java error message I would recommend updating and it certainly will work.

    Thanks.

    Reply

Leave A Comment.

Back to top