Screaming Frog SEO Spider Update – Version 2.20

Dan Sharp

Posted 15 July, 2013 by in Screaming Frog SEO Spider

Screaming Frog SEO Spider Update – Version 2.20

I am delighted to announce version 2.20 of the Screaming Frog SEO spider.

screaming-frog-seo-spider-2-20

We have been busy behind the scenes developing some very cool new features which we hope everyone will enjoy! As always, thank you to everyone for their fantastic feedback and suggestions, we still have plenty more to come. Version 2.20 now includes the following –

Redirect Chains Report

There is a new ‘reports’ menu in the top level navigation of the UI, which contains the redirect chains report. This report essentially maps out chains of redirects, the number of hops along the way and will identify the source, as well as if there is a loop. This is really useful as the latency for users can be longer with a chain, a little extra PageRank can dissipate in each hop and a large chain of 301s can be seen as a 404 by Google (Matt discussed this in a Google Webmaster Help video here).

Another very cool part of the redirect chain report is how it works for site migrations alongside the new ‘Always follow redirects‘ option (in the ‘advanced tab’ of the spider configuration). Now when you tick this box, the SEO spider will continue to crawl redirects even in list mode and ignore crawl depth.

Previously the SEO Spider would only crawl the first redirect and report the redirect URL target under the ‘Response Codes’ tab. However, as list mode is essentially working at a crawl depth of ‘0’, you wouldn’t see the status of the redirect target which, particularly on migrations is required when a large number of URLs are changed. Potentially a URL could 301, then 301 again and then 404. To find this previously, you had to upload each set of target URLs each time to analyse responses and the destination. Now, the SEO Spider will continue to crawl, until it has found the final target URL. For example, you can view redirect chains in a nice little report in list mode now.

Crawl Path Report

Have you ever wanted to know how a URL was discovered? Obviously you can view ‘in links’ of a URL, but when there is a particularly deep page, or perhaps an infinite URLs issue caused by incorrect relative linking, it can be a pain to track down the originating source URL (Tip! – To find the source manually, sort URLs alphabetically and find the shortest URL in the sequence!). However, now on right click of a URL (under ‘export’), you can see how the spider discovered a URL and what crawl path it took from start to finish.

crawl path report

Respect noindex & Canonical

You now have the option to ‘respect’ noindex and canonical directives. If you tick this box under the advanced tab of the spider configuration, the SEO Spider will respect them. This means ‘noindex’ URLs will obviously still be crawled, but they will not appear in the interface (in any tab) and URLs which have been ‘canonicalised’ will also not appear either. This is useful when analysing duplicate page titles, or descriptions which have been fixed by using one of these directives above.

rel=“next” and rel=“prev”

The SEO Spider now collects these html link elements designed to indicate the relationship between URLs in a paginated series. rel=“next” and rel=“prev” can now be seen under the ‘directives’ tab.

Custom Filters Now Regex

Similar to our include, exclude and internal search function, the custom filters now support regex, rather than just query string. Oh and we have increased the number of filters from five to ten and included ‘occurrences’. So if you’re searching for an analytics UA ID or a particular phrase, the number of times it appears within the source code of a URL will be reported as well.

Crawl Overview Report

Under the new ‘reports’ menu discussed above, we have included a little ‘crawl overview report’. This does exactly what it says on the tin and provides an overview of the crawl, including total number of URLs encountered in the crawl, the total actually crawled, the content types, response codes etc with proportions. This will hopefully provide another quick easy way to analyse overall site health at a glance. Here’s a third of what the report looks like –

crawl overview report

We have also changed the max page title length to 65 characters (although it seems now to be based on pixel image width), added a few more preset mobile user agents, fixed some bugs (such as large sitemaps being created over the 10Mb limit) and made other smaller tweaks along the way.

As always, we love any feedback and thank you again for all your support. Please do update your SEO Spider now to try out all the new features.

Small Update – Version 2.21 Released 20th October 2013

We have released a very small update to version 2.21 of the Screaming Frog SEO spider, which essentially fixes a few small bugs & updates a couple of features from feedback we received. These include –

  • Individual cell selection, so contents can be individually copied and pasted.
  • Now when you upload a list of URLs and use the ‘Always follow redirects‘ configuration, the resulting redirect chains report export will now include ALL URLs from the original upload, not just those that redirect. This should make it easier to audit URLs in a site migration by seeing them all in one place with a single report.
  • There was a bug with relative URLs and the new respect canonical configuration which has been corrected.
  • We have fixed a couple of smaller total & percentage calculations in the crawl overview report.

Small Update – Version 2.22 Released 3rd December 2013

We have just released another small update to version 2.22 of the Screaming Frog SEO spider. There’s a couple of tweaks, which include –

  • We are now identified Apple developers! So hopefully Mac users will no longer receive a warning that we are ‘unidentified’ or the ‘file is damaged’ upon install.
  • From feedback of individual cell and row selection in 2.21, we have improved this and introduced row numbers.

That’s it for now, if you spot any issues, please as always do let us know via our support. Thanks again for all the kind comments below, we are busy working on lots of new features for our next release!

Dan Sharp is founder & Director of Screaming Frog. He has developed search strategies for a variety of clients from international brands to small and medium-sized businesses and designed and managed the build of the innovative SEO Spider software.

91 Comments

  • Bob Jones 11 years ago

    Awesome update guys! Sounds like some really useful new features.

    Reply
  • All great new features; every single one useful. Keep up the good work!

    Reply
  • Dan Nolan 11 years ago

    The only thing I’m sad about…

    Is i’m about to walk into a meeting and I can’t play with this for a few hours. :)

    Thanks!! The redirect loop/chain tool will let me do a project I’ve been having to do for a few months super easily!

    Reply
    • screamingfrog 11 years ago

      heheh! Thanks Dan – The redirect chain report has been on the list for a little while, so it’s great to release it finally.

      Reply
  • Thomas Rosenstand 11 years ago

    Guys… Thank you! Great new features in my absolute favorite tool. There is not a day where I don’t use Screaming Frog!

    Reply
  • Peer 11 years ago

    Hey, great new features – thanks for your good work !!!

    Reply
  • Dan Shure 11 years ago

    Nice work! You just saved me hours with crawl paths and redirect chains.

    Reply
  • Sebastian 11 years ago

    Nice one! Whats the Max Memory Limit for crawling large sites?

    Reply
  • Andrew J Baker 11 years ago

    Great work, just the features I was hoping for…

    Reply
  • samuel lavoie 11 years ago

    awesome addition to the tool with the crawl path and redirect chains! thanks :)

    Reply
  • Travis Bailey 11 years ago

    The best crawler just got better. I always recommend Screaming Frog when the subject comes up. It’s worth every red cent.

    Reply
  • Michael 11 years ago

    Very nice work! Thanks :)

    Reply
  • Joris Has 11 years ago

    If i knew this a week ago… It saved me approx 2 days of work… Great Update by the way!

    Reply
  • Baumr 11 years ago

    In the new crawl overview spreadsheet, the depth for clicks from the start URL seems useful.

    Depth (Clicks from Start URL):
    1 2 3 4 5 6+

    However, the percentages are adding up to ~250%.

    How do you arrive at the number of URI in this case?

    Reply
    • screamingfrog 11 years ago

      Good spot. There is a bug on these numbers (including more than just html pages).

      We are awaiting more bugs and feedback (there’s always some!) before updating a release with small fixes.

      Cheers.

      Reply
  • Kevin Wiles 11 years ago

    Great update Guys! Might have to purchase a full license now as seem to be using more and more.

    Reply
  • François 11 years ago

    Thanks for the update, especially the redirect path.

    We’re using Screaming Frog to check our websites both on developpement and staging environnement. Because we have many subdomains and the staging environnent is password protected it’s sometimes painful to to type the same username/password for each subdomains.

    Could you add a function, wich will be able to memorize the requested username and password and try it on any subdomains.

    Reply
    • screamingfrog 11 years ago

      Hey François,

      Good to hear you like the redirect chain report.

      We can certainly add your suggestion to our ‘todo’ list, so there is an option to remember the username/password.

      Thanks for the feedback.

      Dan

      Reply
      • François 11 years ago

        Hey Dan,

        Thanks for your reply.
        I would like to be able to define a default username / password for a all subdomains under the same domain so the spider will try to authenticate using this default username/password.

        Francois

        Reply
  • Aaron Alexander 11 years ago

    Real exciting stuff in this update. Pagination directives, Redirect Chains…Oh My! Thank you for continuing to make a great product even better!

    Reply
  • Stellan 11 years ago

    Niiice! Looking forward to play with the frog again :-)

    Reply
  • James Norquay 11 years ago

    Thanks for this will update my version now, then it is time to crawl thousands of pages =)

    Reply
  • Justin Walmsley 11 years ago

    Fantastic update, loving the crawl path!

    Well done guys

    Reply
  • Rock 11 years ago

    Great update. The redirect chain report is perfect for a task I was just assigned. This tool keeps getting better.

    Reply
  • Steve Krull 11 years ago

    Can you please share a quick “how to read” the crawl path report. It seems intuitive enough but I want to make sure I’ve got it right before sharing with customers.

    Reply
  • ravi 11 years ago

    Can you make this software to crawl Javascript files, for external links or internal links. I hope you can include in 2.3 version.

    Reply
  • Avanz 11 years ago

    Oh! That’s great news. Thanks for sharing SEO Search Engine Optimization spider update.
    Really, an useful tool.

    Reply
  • Ricky Shah 11 years ago

    Definitely a great update indeed.
    How about few more suggestions?
    It would be great if you can fetch GA code from the page and mention whether it is missing or not. Also, the HTTP issues in a tree structure would help a lot. It helps in where the HTTP issue is generating from.

    Reply
  • Sankar Datti 11 years ago

    Thank you so much for the update.

    I am waiting a lot for path report and noindex features. Good job.

    Reply
  • Isreal 11 years ago

    Great work guys. I love the new features, especially the overview report. I can’t wait for the next update – worth every cent!

    Reply
  • Eilidh MacRae 11 years ago

    Really informative. Thanks a lot.

    Reply
  • Erdal Torun 11 years ago

    Great Update!!

    Reply
  • Daniel Vareta 11 years ago

    All these new features are great! my favorite is the redirect chain report but all the others are great too.
    Thanks so much for the great work. You guys rock! (as always by the way)
    Daniel

    Reply
  • Tom Howlett 11 years ago

    The new features look really good. Looking forward to trying out the crawl path report feature. Nice work.

    Reply
  • Click this 11 years ago

    Really great work guys. And also very informative . Actually I have been looking for this for a few days. Thanks again.

    Reply
  • Alex Ramadan 11 years ago

    Great updates. These are going to save me a ton of time with site audits. Just tried the Overview report and there is so much useful information in here.

    Reply
  • Jimmy Love 11 years ago

    Thanks for the update team, tons of new & helpful features! Question, do you guys have plans on including a crawler that scans marketing tags (like Google Analytics, Adsense, KissMetrics, etc.) sometime in the near future? I found a pretty cool tool, Tag Inspector (http://www.taginspector.com) that does just that, but would love to see your tool also add this feature in future updates.

    Thanks again for the great work!

    Reply
  • screamingfrog 11 years ago

    Thanks again to everyone for commenting and all the kind words.

    Lots more to come!

    Reply
  • Craig Bailey 11 years ago

    Beautiful work. Really love how this program continues to go from strength to strength. Well done.

    Reply
  • Jason tailor 11 years ago

    This software used to helpful for me and now upgraded version is awesome….it will rock in seo world.

    Reply
  • Michael Belk 11 years ago

    This looks like a very good tool. I am new here so I wanted to see what other tools are out there in SEO.

    I bet you did a lot of work on this tool. It looks very intuitive.

    Reply
  • Kenneth von Rauch 11 years ago

    Great news. The only thing that I’d like to point out that you have not added the option to check if a specific page has been indexed by Google. :)

    Reply
  • Angela 11 years ago

    Wow, thanks for the update. I love Screaming Frog. :)

    Reply
  • thewebtheorist 11 years ago

    Hey, love Screaming Frog! I have a question about Custom Filters. Is there a way of exporting all filters without having to export each individually? It would be fantastic if it would export all custom filters, with perhaps a tab (Excel) for each Filter.

    It would also be really helpful to be able to see what the actual Filter is from the Screaming Frog software. i.e. “Filter: Filter 1 [Export] could be something like “Filter: UA-123456789 [Export]

    Thanks again for such an amazing contribution to the Internet Marketing world with this tool! :)

    Reply
    • screamingfrog 11 years ago

      Hello thewebtheorist :-)

      There’s no way of exporting all at once currently I am afraid, it has to be individually.

      Thanks for the suggestions though, I agree seeing the contents of the filter would be useful as well.

      Cheers.

      Dan

      Reply
  • Dave Young 11 years ago

    Love the Screaming Frog, thanks for the update!

    Reply
  • raquel 11 years ago

    amazing update, I love the new features. Thanks a lot!

    Reply
  • Harsh Agrawal 11 years ago

    Thanks for new great features…They are really handy….

    Reply
  • Nate 11 years ago

    Awesome guys! I don’t know what I would do without Screaming Frog. Your ongoing dedication to improving an already stellar piece of SEO software is incredible.

    Reply
  • Max Peters 11 years ago

    Another great update to my most used tool.

    I have a suggestion for a feature. It would be great if there was a way to crawl URLs, but ultimately exclude them from being included in final report.

    Say if there is website that has hundreds of category ‘page=’ pages, with valid product pages being linked from them. I don’t actually need the ‘page=’ pages in the report, (as I will be excluding these from being indexed in Google) but I can’t exclude them from being crawled as then it would miss all the products pages linked from them.

    This would decrease the size of the crawl, as I often have gigantic crawls of thousands, upon thousands of pages that often run out of the memory.

    Reply
  • seocon.at 11 years ago

    Awesome update guys! Really nice tool for SEOs :-)

    Reply
  • Martin Oxby 11 years ago

    Thanks for continuing to build this great tool – loving the idea of the redirect loop info and also the respecting noindex and canonical – makes the crawl even more accurate. Going to update my version now. :)

    Reply
  • Marcis Gasuns 11 years ago

    If I had to choose which feature to kill from this update, I would not know. Make as stable as Xenu and so it can crawl millions of pages on a website and not fall off – and it will be the single tool one will ever need.

    Reply
  • Anita Clark 11 years ago

    Thanks for the latest update…Screaming Frog is a fantastic tool that has saved me countless man-hours. Much appreciated!

    Reply
  • Max Peters 10 years ago

    The Individual cell selection , while it may be useful, actually feels like a step backward. If i select a line in the first column, then scroll all the way to the far right column, I no longer have any idea which line I’m on.

    Perhaps you could have the 1st colum where you can select the entire row, like excel.

    Reply
    • screamingfrog 10 years ago

      Hey Max,

      Thanks for the feedback. I agree, I think this can be improved.

      The old click a cell which highlights the whole row was useful when you scroll to know exactly which line you’re looking at.

      We will have a look at making this easier, but for now you can (you may already know) –

      * When you click on a cell and want to scroll right, just hold down the click on the cell and drag right (which scrolls right and highlights the row).

      Or

      * Click on the cell and use the right arrow key, this scrolls right and keeps to the same row.

      Cheers.

      Dan

      Reply
  • Tilo Rupsch 10 years ago

    First of all thank you for your awesome tool. But I have to agree to Max Peters, the individual cell selection of version 2.21 fells like a step back.

    The export to csv function didn’t work well for me because some characters created an unwanted line break in excel. This made it really hard to work with this data. But i found a workaround for this problem. I just copied and pasted everything out of Screaming Frog into Excel and then applied the style of excel to it. This put every piece of content in the right cell, no aditional fixes were needed.

    Now since to new update this workaround isn’t working anymore. Maybe you figure something out to make this work again, or even fix/optimize the export to csv function. Until then i’ll have to downgrade to 2.20 again.

    Reply
  • Max 10 years ago

    Hi SF,
    ur update sounds cool … very cooL! Acutualy theres a problem with installing the new version. Downloading runs witout any problems, but while installation at mac 10.9 an error Message appears: “Screaming Frog SEO Spider.app” ist beschädigt und kann nicht geöffnet werden. Es empfiehlt sich, das Objekt in den Papierkorb zu bewegen.”

    Its sounds like a damaged image?!
    Quick Help would be nice,
    thx
    Max

    Reply
  • George 10 years ago

    The update sound cool with bug fixes.

    Reply
  • Conrad 10 years ago

    Any idea on support for Retina text on Mac?

    Reply
    • screamingfrog 10 years ago

      Hi Conrad,

      It’s something we’ve had feedback on recently, so will be looking into.

      Thanks,

      Dan

      Reply
  • Nicholas 10 years ago

    Getting this message under mavericks.

    Finder[157]: JLRequestRuntimeInstall: CFMessagePortSendRequest: error -1

    I have uninstalled and reinstalled and nothing. It will not launch.

    I would appreciate any feedback or advice. Thx for your time.

    Reply
  • Neha 10 years ago

    loving the idea of redirecting loop..thanks for the tool and the updates..fantastic post..Thank you…!!!

    Reply
  • We use Screaming Frog extensively here at the office. Only question is are you going to provide support for Retina text on Mac? I’d like to see our results in full HD B)

    Reply
    • screamingfrog 10 years ago

      Hey David,

      It’s on our list, we have had a few people request this :-)

      Cheers,

      Dan

      Reply
  • Thanks Dan for the reply. I’ll keep my eyes open for the update!

    Reply
  • Jason Hennessey 10 years ago

    Big Fan of Screaming Flog…this is a must have SEO tool.

    Loving the redirect chain report.

    Thx for all that you do!

    – Jason Hennessey

    Reply
  • George 10 years ago

    Does screaming frog analysis schema structure of blog ?

    Reply
    • screamingfrog 10 years ago

      Hi George,

      It doesn’t as yet, but we do have it on the ‘todo’.

      You can use the custom filters to search for any Schema though :-)

      Thanks,

      Dan

      Reply
  • Vance 10 years ago

    Does SEO Spider have any automation capabilities? Like, I save my list of URLS to run, and it can run it each night at 10 and save out the reports separate for each URL; or even export as a sql file?

    Or am I going to have to do something funky with Applescript and GUI automation?

    Reply
  • BIG shout out to Screaming Frog! I’ve been using your software for a while, but mainly internally. I WOW’d a prospect last week with a report I handed them thanks to data from SF, Moz and Link Assistant’s Rank Tracker, that they pulled the trigger to move forward with us! Your software is invaluable and looking for to more updates as you all evolve. That Retina display would be kick-ass as well from a comment above I saw!! – Patrick @ Whiteboard Creations

    Reply
  • Ted Fligor 10 years ago

    Loaded log on screen first in the list of URL’s to crawl, but the log on screen dialog box never comes up, page says unauthorized.

    Reply
    • screamingfrog 10 years ago

      Hey Ted,

      You’re commenting on a really old blog post now and I’ve just seen you’ve contacted support.

      So, will be in touch via email.

      Cheers,

      Dan

      Reply
  • Raymond 9 years ago

    Guys… Thank you! Great new features in my absolute favorite tool. There is not a day where I don’t use Screaming Frog!

    Reply
  • WorkoutSeven 8 years ago

    Thanks for continuing to build this great tool – love it!

    Reply

Leave A Comment.

Back to top