Screaming Frog SEO Spider Update – Version 2.50

Dan Sharp

Posted 1 July, 2014 by Dan Sharp in Screaming Frog SEO Spider

Screaming Frog SEO Spider Update – Version 2.50

I am pleased to announce a small update of the Screaming Frog SEO spider to version 2.50, codenamed internally as ‘bite’. Thanks again to everyone for their continued feedback, suggestions and support.

This is just a small update, with a couple of new features and bug fixes listed below. Our next update will be much, much larger. Version 2.50 includes just the one main new feature –

Graph View

We have introduced a new ‘graph view’ window into the lower half of the right hand overview window pane. This updates in real time, as you crawl –

When you click the different tabs at the top, or filters in the right hand window pane, the graphs update to help visualise the data.

If you click the ‘Site Structure’ or ‘Response Codes’ tabs, you can see better visualisations of this data as well.

At the moment you can’t export the graphs unfortunately, but we do have this planned for development.

Other Updates

We have also made a number of other updates, these include the following –

We became a verified publisher for the Windows platform.
Improved debugging and saving of log files for support issues directly from the ‘Help’ and ‘debug’ menu.
Improved logging so data is not overwritten with new sessions.
The SEO Spider now detects any crashes on a previous run and asks the user to save logs and send to our support team.
The SEO Spider now has greater tolerance around invalid gzip responses. We will now continue with what was read up to the point of error, logging appropriately in the log file.
Fixed a bug where the ‘ignore robots.txt’ configuration wouldn’t save as default.
Fixed a bug where the ‘respect canonicals’ configuration was case insensitive.
Fixed a bug where we failed to parse some pages containing meta refresh tags resulting in invalid urls.
Fixed a crash caused by the sequence add, add, delete, delete, add on URL Rewriting -> Remove Parameters config menu :-).
Fixed a crash caused by starting in crawl mode, navigating to the outlinks tab and right clicking.

That’s the lot for this fairly small update. We still have plenty more in the pipeline and our next version will be a much larger release!

Thanks for all your support and feedback as always, you can download the new version now.

Move to Java 7

Version 2.50 requires Java 7 to use (rather than Java 6), which was a natural evolution and launched around three years ago now.

It’s also worth noting that Macs below OS X Version 10.7.3 (& 32-Bit Macs) can continue to use version 2.40, as well as anyone who might be experiencing any bugs in the new version.

Dan Sharp is founder & Director of Screaming Frog. He has developed search strategies for a variety of clients from international brands to small and medium-sized businesses and designed and managed the build of the innovative SEO Spider software.

31 Comments

Guilherme 9 years ago

Great update, really useful in reports.

I have a suggestion, why don’t you add a filter to show if any page has the Google Analytics script missing?

Best and continue with the great work!

Reply
- screamingfrog 9 years ago
  
  Hi Guilherme,
  
  Thanks for the kind comments and for the suggestion.
  
  We do have this on our ‘todo’, obviously you can do it manually at the moment via the custom filters, but we were going to introduce something that helps do this automatically / easier :-)
  
  Thanks again,
  
  Dan
  
  Reply
Richard 9 years ago

The last 3 or 4 updates have been great!

Reply
- screamingfrog 9 years ago
  
  Cheers Richard! Lots more to come.
  
  Reply
Guillaume 9 years ago

Hi Guys,
Thanks for that great update !

Don’t worry for graph exports, in the meantime, we will do screen captures.

Keep up that good work :)

Reply
- screamingfrog 9 years ago
  
  Good shout Guillaume!
  
  Thanks!
  
  Reply
Eric 9 years ago

Hi
great news, your product is very efficient
but for me there is somthings missing yet !
:)
it ‘s will be very funny if we can know where you can find easily the external links on the report

best regards

Reply
- screamingfrog 9 years ago
  
  Hi Eric,
  
  Apologies if I don’t understand the question completely.
  
  You can view ‘external links’ under the ‘external’ tab and you can view where a site links out to these, by clicking on the ‘in links’ tab at the bottom.
  
  If you mean, inbound links (ie links coming URLs found in the crawl), this is on our ‘todo’ for the future.
  
  Thanks,
  
  Dan
  
  Reply
Adam Heaton 9 years ago

We are spoilt with these updates, can’t wait for the coming updates! Wonder what else you guys have in store for us!

Reply
Mike Cravens 9 years ago

Would you please consider continuing to ship binaries for Intel Macs running Intel CoreDuo (32 bit address) processors.

In the latest version of Apple’s Xcode developers’ package they default to not supporting this machines, reversing a long standing Apple policy of defaulting to include binaries for practically every machine Apple ever made in standard Universal Binaries.

The Core Duo processor Imacs are fairly recent and are still in use at my customers, in schools, and in our offices and homes.

I promise to upgrade our licenses and to continue to promote your software and services in the seminars we teach if you will only change back your Xcode setup to what is was until just recentl and ship Intel 32 bit Core Duo binaries along with 64 bit binaries.

Thank you for such a wonderful tool. I hope to be able to continue using and promoting it.

Regards,

Mike Cravens
Managing Partner
Southwest Digital Media

Reply
- screamingfrog 9 years ago
  
  Hi Mike,
  
  Thanks for your comment (and the email you popped through).
  
  The team have just responded to you via email and I have put together an FAQ here –
  
  https://www.screamingfrog.co.uk/seo-spider/faq/#45
  
  Essentially, 32-bit Macs are still supported for version 2.40 which you can continue to download and use.
  
  2.50 requires newer technology only availble in Java 7 which is not available for OS X below 10.7.3 (& 32-Bit Macs). The difference, is really just the graphs.
  
  Cheers.
  
  Dan
  
  Reply
Jared Kayes 9 years ago

Another good update I see.

Reply
Matthew 9 years ago

Great update. May I suggest implementing export graphs to jpeg or png functionality. This would make client reporting a lot easier for me. Keep up the good work.

Reply
- screamingfrog 9 years ago
  
  Hi Matthew,
  
  Thanks for the suggestion.
  
  I mentioned in the post we will be introducing this functionality definitely.
  
  Cheers.
  
  Dan
  
  Reply
  - Matthew 9 years ago
    
    My bad. Again, thanks for the great work and support – very impressive tool
    
    Reply
    - screamingfrog 9 years ago
      
      No worries, thanks for the kind comments!
      
      Reply
  - matthew 9 years ago
    
    Hi Dan,
    
    I’ve having issues with Screamingfrog on my mac. It hangs badly whenever I use it. I’m running an I7 MBP running 10.9.4 any suggestions or known bugs?
    
    Reply
    - screamingfrog 9 years ago
      
      Hi Matthew,
      
      It will be much quicker if you pop an email directly through to support with all the details – https://www.screamingfrog.co.uk/seo-spider/support/
      
      We can then advise!
      
      Cheers.
      
      Dan
      
      Reply
Ian McGarvey 9 years ago

Great update. I wanted suggest / request a new feature (First of all, a feature request page / forum would be nice):

I work at a website company and one of the things on our pre-launch checklist is to confirm that all external links open in a new window/tab.

It would be awesome if this software could capture the “target” attribute of links so that I could sort external links by that column and see what external links are not target=”_blank.”

Thanks for your time and effort in general and in response to this request.

Reply
- screamingfrog 9 years ago
  
  Hey Ian,
  
  Thanks for your suggestion, I can confirm we do have this on the ‘todo’.
  
  I do like the idea of a feature request page as well. We have been considering this too!
  
  Cheers,
  
  Dan
  
  Reply
  - Marc 7 years ago
    
    Did you guys ever get around to doing this?
    Is there any other way to effectively extract this information?
    
    Reply
Josh 9 years ago

Hi,
i have a problem with the Graph View.
It works in the first time, but not now.

I uninstalled the software and reinstalled it.
This does not work, I cant see the graph.
The area does not exist anymore.

What can I do?

Reply
- screamingfrog 9 years ago
  
  Hey Josh,
  
  It sounds like you just need to click the little ‘arrow’ in the top right, which makes the right hand overview window open up.
  
  OR if the right hand window is open and just the ‘graph view’ is missing, you might need to click the little ‘arrow’ at the bottom.
  
  You can see them in the screenshots above.
  
  Cheers,
  
  Dan
  
  Reply
Jim Hodson 9 years ago

Like the new visualization.

Was disappointed that the bug related to “Configuration -> Spider -> Crawl all subdomains” was not fixed. It existed in the last release and exists in this one.

When checked, it essentially causes the program to crawl ALL linked domains and subdomains (internal and external), crawl all links on those external domains, ad infinitum.

I thought if checked and crawling http://example.com/, it instructed to the program to crawl all subdomains of example.com (current domain being crawled). But instead it acts like “crawl all linked domains” or “crawl the web” lol.

Reply
- screamingfrog 9 years ago
  
  Hi Jim,
  
  Thanks, excellent to hear you like!
  
  Sorry you’re disappointed we didn’t fix this bug in this update. It is known and on our list, but actually wasn’t a priority as it’s easy to work around.
  
  If you’re crawling from the root (for example http://example.com/), the SEO spider will crawl all subdomains as default anyway. So there isn’t actually any need to use the ‘Crawl all subdomains’ option, which is when you correctly pointed out this bug occurs.
  
  This is similar to a site: query in Google, where Google will show all subdomain URLs in their index if you query the root. If you specify a subdomain in the site: query, it will only show that subdomain results etc.
  
  If you’re crawling from a subdomain, like http://www.example.com/, you can still use the ‘Crawl all subdomains’ configuration fine.
  
  Hopefully that helps for now, we will fix it though!
  
  Cheers,
  
  Dan
  
  Reply
Matt Storms 9 years ago

Increasing the size of the arrows to see the graph view would be helpful or accent them with a color.

Reply
- screamingfrog 9 years ago
  
  Hey Matt,
  
  Thanks for the comment & follow up tweet.
  
  The issue is a bug, where the right hand window doesn’t open for the first time as default (so some don’t realise it’s even there).
  
  The arrow improvements would help in terms of UX, but we have now fixed the bug too :-)
  
  Cheers.
  
  Dan
  
  Reply
Emin Sinanyan 9 years ago

Love the new version. The graphs are amazing. I have one suggestion. When the tool scans for a url, it would be nice to highlight the errors. For example 404 errors red, 301s yellow or some other color. It would be easier on the eye for huge sites. Thank you!

Reply
- screamingfrog 9 years ago
  
  Hey Emin,
  
  Thank you, very kind.
  
  The colours of the graphs/charts have been chosen with that in mind already.
  
  For example for response codes, the colours are as follows –
  
  No Response – Grey
  2XX – Green
  3XX – Orange
  4XX – Red
  5XX – A dark red.
  
  We found this to quite nice to help spot obvious problems. Hope that helps :-)
  
  Dan
  
  Reply
othmane 9 years ago

Hello,

Great update 2.5 , i would suggest to add average depth like average response time it’s would help to know average depth for the website.

Also i recommande when we click on the nombre of uri in depth (graph view) we can have acces to this url without having to export to excel.

Cheers.
Othmane

Reply
- Bruno Pinna 8 years ago
  
  hey team of Screaming Frog
  thank you for develop this tool because my life now is more easy
  really i doing on hand… your tool save my time.
  great job but i want news update :)
  cheers from brazil seo
  
  Reply