Screaming Frog SEO Spider Update – Version 15.0

Dan Sharp

Posted 12 April, 2021 by in Screaming Frog SEO Spider

Screaming Frog SEO Spider Update – Version 15.0

We are delighted to launch Screaming Frog SEO Spider version 15.0, codenamed internally as ‘disparity’.

This is a release we are really excited about, as it focuses on a major feature we’ve wanted to introduce for some time and have been quietly developing in the background.

Here’s an overview of what’s new.


1) Crawl Comparison

You can now compare crawls and see how data, issues and opportunities have changed in tabs and filters over time.

Crawl Comparison

This feature helps track the progress of technical SEO issues and opportunities and provides granular data about what’s changed between the crawls.

To compare, go to ‘File > Crawls’, highlight two crawls, and ‘Select To Compare’.

Select to compare crawls

Or, switch to ‘Mode > Compare’ and click ‘Select Crawl’ via the top menu to pick two crawls you wish to compare. You can adjust the compare configuration (more on that shortly) or just click ‘Compare’.

Mode Compare

The crawl comparison analysis will then run and the right-hand overview tab will populate to show current and previous crawl data and changes.

It will identify whether existing URLs found in the previous crawl have moved from or to a tab or filter (‘added’ and ‘removed’), or if a URL is entirely ‘new’, or now ‘missing’ in the latest crawl. This helps better understand progress and if issues are going up or down for URLs you already know about.

Crawl Comparison Overview tab

You’re able to click on the numbers in the columns to view which URLs have changed, and use the filter on the master window view to toggle between current and previous crawls, or added, new, removed or missing as well.

Crawl Comparison

(Insert joke about the results being more impressive than Arsenal’s)

The compare feature is only available in database storage mode with a licence. If you haven’t already moved, it’s as simple as ‘Config > System > Storage Mode’ and choosing ‘Database Storage’.

Database storage comes with a number of significant benefits, such as improved crawling at scale, auto storing of crawls, super-quick opening and helping to avoid lost crawls if your machine turns off unexpectedly during that 1m URL crawl. Check out our storage modes video guide for an overview.


2) Site Structure Comparison

The right-hand ‘Site Structure’ tab shows a directory tree overview of how the structure of a site has evolved. It allows you to identify which directories have new or missing pages, for example, you can see new files have been found within the /sites/, /fixture/ and /news/ directories below.

Compare Site Structure

This can help provide more context to how a site is changing between crawls.

It aggregates data from the previous and current crawl to show where URLs have either been added to a directory or have been removed. You can click in and drill down to see which specific URLs have changed.

Compare Site Structure Page Changes

You’re also able to visualise how crawl depth has changed between current and previous crawls, which helps understand changes to internal linking and architecture.

Crawl Comparison Crawl Depth

3) Change Detection

While the overview tab is able to identify most key changes between crawls, it doesn’t tell you if an element or key metric has changed – such as a page title, word count, crawl depth, internal links, structured data and more.

It will only tell you if, for example, a page title is now ‘Over X Characters’ or ‘Missing’ based upon the parameters defined within filters. Keeping on top of these kinds of changes is vital as it can make such a big difference in ranking.

So you’re also able to compare all of these within the SEO Spider to see exactly what has changed in the new ‘Change Detection’ tab.

Change Detection When Comparing Crawls

This can be super useful when checking on the implementation of SEO recommendations (has the client really updated those page titles and meta descriptions?), or when monitoring a website where you can now be alerted to changes that perhaps you weren’t expecting!

Another important use-case is parity checks, such as comparing Mobile Vs Desktop, or JavaScript Vs Non-JavaScript crawls for differences.

Mike King recently wrote an excellent piece on the importance and complexities of parity issues (and shared a handy free tool called Parito to aid the process). Our new change detection feature will help perform the heavy lifting for parity checks and find differences at scale.

When in ‘Compare’ mode, you can click on the compare configuration via the cog at the top (or ‘Config > Compare’) and select which metrics you’re interested in identifying changes.

Crawl Comparison Config

When you have clicked ‘Compare’ and the analysis has run, the ‘Change Detection’ tab becomes available and shows what has changed in granular detail.

You’re able to click on an element and view the changes in the master window, such as for page titles where you can see current and previous page titles side-by-side and review how they have changed.

Page Title Changes

We all know how important internal linking can be, and this feature can help alert you to pages that have suddenly lost unique internal links.

Unique Inlinks Changes

You can also keep an eye on pages that the content team are improving –

Word Count Changes

Or have changed in crawl depth –

Crawl Depth Comparison

Another cool feature is that you’re able to adjust the new filter on the lower window to see what’s changed to better understand the ‘why’. So if crawl depth has changed to ‘1’ from ‘3’ between crawls like the above example, you can select ‘inlinks’ at the bottom, and then ‘Added’ to see any new inlinks.

New Inlinks Change Detection

In the above example, the current crawl was a JavaScript enabled crawl, and previous was crawling the raw HTML. So this has helped identify that the Rouge Media homepage has links to the portfolio page which use client-side JavaScript.

The ‘Content’ change detection option requires you to store the HTML of the page in a crawl (‘Config > Spider > Extraction > Store HTML & Store Rendered HTML’) and this analysis takes a little longer to run using the minhash algo. This feature can approximate the page change % from the previous to the current crawl based upon your configured main content area.

Content Change Detection

In this example, our version 14 blog post has changed in content by 1% between the crawls.

You can click on the lower window ‘View Source’ tab, select ‘compare’ and ‘show differences’ to highlight what has changed in the HTML.

View Source Comparison

It’s relatively trivial to add more elements to change detection, so do let us know if there’s anything else you’d like to see included here.


4) URL Mapping For Staging / Different URL Structure Comparison

As part of crawl comparison, we have introduced a URL Mapping feature, which allows you to compare two different URL structures effectively. This is helpful when comparing a production site against staging, or a website that uses separate mobile URLs against its desktop equivalent.

To set this up, click on the compare configuration and ‘URL Mapping’. Then input a regex to map the previous crawl URLs into the current crawl. For example, we recently switched host to Kinsta and tested a staging site pre to migrating using this new feature.

Crawl Comparison URL Mapping

The Kinsta staging site is then rewritten to the production site and the equivalent URLs will be compared against each other for overview tab data, issues, and opportunities, the site structure tab, and change detection.

You can switch between the current and previous crawls to view the different URL structures still.

Crawl Comparison URL Mapping

If a client has made much smaller changes like removed trailing slashes from URLs in staging, this feature can also be used to help compare the different versions of the same pages with minimal effort as well.

This will hopefully make SEOs lives a little bit easier during testing for new site releases and migrations.


Other Updates

Version 15.0 also includes a number of smaller updates and bug fixes, outlined below.

  • Math Solvers and Practice Problems Google rich result features are now supported in structured data validation.
  • There’s a new ‘Crawl Timestamp’ column in the Internal tab, which should help with automation, reporting and debugging.
  • Project folders within the ‘File > Crawls’ menu are now collapsed by default.
  • The URL bar will now default to HTTPS if you don’t specify the protocol.
  • Fixed a blurry interface issue on high resolution screens on both Windows and Linux (for any scaling setting).
  • Fixed many bugs that are too monotonous to include in any detail. You’re not even reading this last bullet point, so why am I writing it?

That’s all for now. We think these features help raise the SEO Spider to a new level, so hopefully, you find them useful. Please see our tutorial on ‘How To Compare Crawls‘ for more on how to use all the features released above. If you experience any issues, please let us know via support and we’ll help.

Thank you to everyone for all their feature requests, feedback, and continued support.

Now, go and download version 15.0 of the Screaming Frog SEO Spider and let us know what you think!


Small Update – Version 15.1 Released 14th April 2021

We have just released a small update to version 15.1 of the SEO Spider. This release is mainly bug fixes and small improvements –

  • Fix bug that allowed selection of a project directory in ‘Compare’ mode triggering a crash.
  • Fix bug in crawls dialog with crawls not being ordered by modified date.
  • Fix tool tip alignment issues in Compare mode.
  • Fix bug in JavaScript rendering causing a stall when crawling PDFs.
  • Fix bug where re-spidered URLs that previously failed rendering appeared in the External tab.
  • Fix crash in Locales that use non ASCII numbering systems when crawling sites with multiple instances of HTML elements.
  • Fix crash removing URLs from Tree View.

  • Small Update – Version 15.2 Released 18th May 2021

    We have just released a small update to version 15.2 of the SEO Spider. This release is mainly bug fixes and small improvements –

    • Include ‘Current Indexability Status’ (and previous) columns in ‘Change Detection > Indexability’.
    • Added right-click ‘Respider’ to details panel.
    • Add status and status code to right click ‘Export > Inlinks’.
    • Improve usability of comparison mode crawls dialog.
    • Improve usability of Crawl Comparison URL Mapping UI.
    • Remove new lines from Link Paths.
    • Fix various issues with Forms based authentication.
    • Fix bug where cells containing lots of data would cause Google Sheets exports to fail.
    • Fix bug reading sitemaps containing a DOCTYPE.
    • Fix bug where not all image types were shown in the Image Preview tab.
    • Fix crash performing a search in the master view after changing a Custom Search or Extraction.
    • Fix bug where doing a respider on loaded in completed crawl with GSC/GA looses columns in master view when saved.
    • Fix bug with change detection metrics missing when upgrading crawl to 15.2.
    • Fix bug with Server Response Times columns showing incorrect data if ‘server response times category’ is also configured.
    • Fix bug with persistent spell check dictionary being cleared on startup.
    • Fix bug with Word Cloud Visualisation HTML export not working.
    • Fix bug with trailing slashes being stripped from file names in screenshots bulk export.
    • Fix bug with class name shown in Gnome menu on Ubuntu 18.04.
    • Fix bug with SERP description clipping behind table.
    • Fix crash in Image Details preview pane.
    • Fix bug with PSI exports showing headings as Secs, but data as MS.
    • Fix Duplicate ‘Max Image Size’ label in Spider Preferences.
    • Fix crash with using an invalid regex in Regex Replace UI.


Dan Sharp is founder & Director of Screaming Frog. He has developed search strategies for a variety of clients from international brands to small and medium-sized businesses and designed and managed the build of the innovative SEO Spider software.

42 Comments

  • Deepak Kumar Das 3 years ago

    This is what i was looking for! Thank you for adding new stuffs and improved user friendly checks to the tool each time you bringing an update.

    Reply
  • Garrett 3 years ago

    “Fixed many bugs that are too monotonous to include in any detail. You’re not even reading this last bullet point, so why am I writing it?” ~ Yes I am! :P

    Reply
    • screamingfrog 3 years ago

      Hahaha!

      Reply
  • Wojciech Pietrzak 3 years ago

    It looks very nice, especially when we need to compare text rendering / javascript rendering differences in the website. I “feel” a quite nice performance increase also.

    With this feature I think that you are working on server-side app :)

    Reply
  • Seo Doktor 3 years ago

    One of best seo tools.

    Reply
  • Hannah Miller 3 years ago

    We certainly did read right to the very last bullet :) Great change, looking forward to getting stuck into it. Thanks team!

    Reply
    • screamingfrog 3 years ago

      Thanks, Hannah!

      Reply
  • Ryan Ip 3 years ago

    Thanks for the great update! The changes are very useful and the comparison between crawls make it easier to detect anomalies.

    Reply
    • screamingfrog 3 years ago

      Good to hear, Ryan – and thank you.

      Reply
  • Playfortuna 3 years ago

    I hope there will be no more bug with skipping pages when crawling

    Reply
  • Nazarii 3 years ago

    Please add comparing of three crawls

    Reply
  • Andrew 3 years ago

    Looks like your Pagespeed API exports are all screwed up. The data is mislabeled all over the place. Likely needs a hotfix before I’ll update.

    Reply
    • screamingfrog 3 years ago

      Hi Andrew,

      We’re on this one. We’ve also added in automated checks so this doesn’t happen again, as it’s an old bug returned.

      Cheers.

      Dan

      Reply
  • Kyle Mack 3 years ago

    oh, I read the last bullet point.

    I would love to be able to select which folder the Google Sheets goes into.

    Still need to test this article here:
    https://www.screamingfrog.co.uk/how-to-automate-screaming-frog-with-google-data-studio/

    Reply
    • Evgeniy 3 years ago

      Automation of Screaming Frog with Google Data Studio becomes a child’s play since 15.0. Since then Screaming Frog has a crawl timestamp in the internal_all tab, so you must just pull an export into Google drive and archive the overwritten tab with https://workspace.google.com/u/0/marketplace/app/archive_data/187729264527. With these data you can automatically create time series charts in GDS with an affort of 10 minutes.

      Reply
  • Krzysztof Parfieniuk 3 years ago

    Hi can you give in Validation -> Check HTML with W3C Validation this version : https://validator.w3.org/nu/ ?

    Reply
  • J 3 years ago

    Hey ran a crawl for kukui dot com, said that most pages were Non-Indexable but when I did manual check, no issues on SERP/Page source. Possible bug? may want to check it out

    Great job on this update! Thanks so much

    Reply
    • Mark Porter 3 years ago

      Hi J,

      It looks like you may have some issues with the canonical link elements on your site. Your pages are canonicalised to the non-secure version (http://), but the non-secure version 301 redirects back to the secure version (https://). You may want to take a look and investigate further, but this explains why it’s showing as non-indexable in the spider.

      Thanks!

      Mark

      Reply
  • Erick 3 years ago

    You guys rock! ✌

    Reply
  • Get Noticed 3 years ago

    Crawl comparison is great for analysis but also for reporting. Thank you for this feature!

    Reply
  • Matan Cohen 3 years ago

    Thanks for the update!

    Reply
  • Rakesh 3 years ago

    I am big fan of screamingfrog. Crawl Comparison & Site Structure Comparison is much needed change. Thanks for this update. I would love to explore this new features right now.

    Reply
  • Suraj Rana 3 years ago

    your guys are rocking with change detection tab.

    Reply
  • Playfortuna 3 years ago

    Added right-click ‘Respider’ to details panel.
    finally, I really missed this feature

    Reply
  • Instabaza 3 years ago

    Great update. I love this tool

    Reply
  • Sam 3 years ago

    I have been using screaming frog for years now. It’s really good, thanks you

    Reply
  • alphabeta 3 years ago

    I have to admit that Screaming Frog impresses me more and more with each new update. It is one of the basic and obligatory tools for SEO activities. I have been using it for a long time and its functionality definitely facilitates and speeds up my work.

    Reply
  • Maxime Clementz 3 years ago

    Another great update. Screaming frog is an essential tool for all seo agencies

    Reply
  • Jimit 3 years ago

    I think URL Mapping will be handy for me. As I am always involved in SEO from planning, development & deployment.

    Reply
  • Mill Ex 3 years ago

    Great update. I am very fond of this SEO tool. Screaming Frog is an amazing tool which I have been using for years.

    Reply
  • Jvalin 3 years ago

    Great to know the update. The screaming frog keeps proving the best tool in this niche..

    Reply
  • Novitech 3 years ago

    Good software. Imo best & fast crawler on market.

    Reply
  • Occeo 3 years ago

    Always the best SEO crawler in the game !

    Reply
  • Logic 3 years ago

    I did not know dark mode was an option in screaming frog, thats amazing!

    Reply
  • Piotr Starzynski 3 years ago

    Great piece of software ;) And for me a crawl depth change is a really nice feature!

    Reply
  • Samuel Fillmore 3 years ago

    The new change detection feature is a lifesaver. Keep up the good work – I’m trying to consolidate my tools and I just crossed off Parito.

    Reply
  • Krasimir 3 years ago

    Great to see this. Continue with the good work :)

    Reply
  • SeoTest 3 years ago

    Oh my goodness. This is a fantastic implementation, thank you very much :)

    Reply
  • SeoGig 3 years ago

    In version 15.2, the Google analytics API does not work for me – it shows 0 everywhere.
    I connect it in the program, press connect, allow access in the browser.
    Next, the API writes, 0 in all graphs.

    It started after moving to ga v4. I connect it via google tag manager.
    I don’t understand what the problem is, google Nothing is like about this.
    Same error on all my sites in different accounts.

    At the same time, the Search Concole API seems to be working properly.
    Please help me solve the problem.

    Reply
  • Forcognito 3 years ago

    Thanks a lot for the elaborated ubdate!

    Using your tool for a long time now and it seems it’s only getting better over time. your efforts are much appreciated.

    Reply

Leave A Comment.

Back to top