Screaming Frog SEO Spider Update – Version 15.0

We are delighted to launch Screaming Frog SEO Spider version 15.0, codenamed internally as ‘disparity’.

This is a release we are really excited about, as it focuses on a major feature we’ve wanted to introduce for some time and have been quietly developing in the background.

Here’s an overview of what’s new.

1) Crawl Comparison

You can now compare crawls and see how data, issues and opportunities have changed in tabs and filters over time.

This feature helps track the progress of technical SEO issues and opportunities and provides granular data about what’s changed between the crawls.

To compare, go to ‘File > Crawls’, highlight two crawls, and ‘Select To Compare’.

Or, switch to ‘Mode > Compare’ and click ‘Select Crawl’ via the top menu to pick two crawls you wish to compare. You can adjust the compare configuration (more on that shortly) or just click ‘Compare’.

The crawl comparison analysis will then run and the right-hand overview tab will populate to show current and previous crawl data and changes.

It will identify whether existing URLs found in the previous crawl have moved from or to a tab or filter (‘added’ and ‘removed’), or if a URL is entirely ‘new’, or now ‘missing’ in the latest crawl. This helps better understand progress and if issues are going up or down for URLs you already know about.

You’re able to click on the numbers in the columns to view which URLs have changed, and use the filter on the master window view to toggle between current and previous crawls, or added, new, removed or missing as well.

(Insert joke about the results being more impressive than Arsenal’s)

The compare feature is only available in database storage mode with a licence. If you haven’t already moved, it’s as simple as ‘Config > System > Storage Mode’ and choosing ‘Database Storage’.

Database storage comes with a number of significant benefits, such as improved crawling at scale, auto storing of crawls, super-quick opening and helping to avoid lost crawls if your machine turns off unexpectedly during that 1m URL crawl. Check out our storage modes video guide for an overview.

2) Site Structure Comparison

The right-hand ‘Site Structure’ tab shows a directory tree overview of how the structure of a site has evolved. It allows you to identify which directories have new or missing pages, for example, you can see new files have been found within the /sites/, /fixture/ and /news/ directories below.

This can help provide more context to how a site is changing between crawls.

It aggregates data from the previous and current crawl to show where URLs have either been added to a directory or have been removed. You can click in and drill down to see which specific URLs have changed.

You’re also able to visualise how crawl depth has changed between current and previous crawls, which helps understand changes to internal linking and architecture.

3) Change Detection

While the overview tab is able to identify most key changes between crawls, it doesn’t tell you if an element or key metric has changed – such as a page title, word count, crawl depth, internal links, structured data and more.

It will only tell you if, for example, a page title is now ‘Over X Characters’ or ‘Missing’ based upon the parameters defined within filters. Keeping on top of these kinds of changes is vital as it can make such a big difference in ranking.

So you’re also able to compare all of these within the SEO Spider to see exactly what has changed in the new ‘Change Detection’ tab.

This can be super useful when checking on the implementation of SEO recommendations (has the client really updated those page titles and meta descriptions?), or when monitoring a website where you can now be alerted to changes that perhaps you weren’t expecting!

Another important use-case is parity checks, such as comparing Mobile Vs Desktop, or JavaScript Vs Non-JavaScript crawls for differences.

Mike King recently wrote an excellent piece on the importance and complexities of parity issues (and shared a handy free tool called Parito to aid the process). Our new change detection feature will help perform the heavy lifting for parity checks and find differences at scale.

When in ‘Compare’ mode, you can click on the compare configuration via the cog at the top (or ‘Config > Compare’) and select which metrics you’re interested in identifying changes.

When you have clicked ‘Compare’ and the analysis has run, the ‘Change Detection’ tab becomes available and shows what has changed in granular detail.

You’re able to click on an element and view the changes in the master window, such as for page titles where you can see current and previous page titles side-by-side and review how they have changed.

We all know how important internal linking can be, and this feature can help alert you to pages that have suddenly lost unique internal links.

You can also keep an eye on pages that the content team are improving –

Or have changed in crawl depth –

Another cool feature is that you’re able to adjust the new filter on the lower window to see what’s changed to better understand the ‘why’. So if crawl depth has changed to ‘1’ from ‘3’ between crawls like the above example, you can select ‘inlinks’ at the bottom, and then ‘Added’ to see any new inlinks.

In the above example, the current crawl was a JavaScript enabled crawl, and previous was crawling the raw HTML. So this has helped identify that the Rouge Media homepage has links to the portfolio page which use client-side JavaScript.

The ‘Content’ change detection option requires you to store the HTML of the page in a crawl (‘Config > Spider > Extraction > Store HTML & Store Rendered HTML’) and this analysis takes a little longer to run using the minhash algo. This feature can approximate the page change % from the previous to the current crawl based upon your configured main content area.

In this example, our version 14 blog post has changed in content by 1% between the crawls.

You can click on the lower window ‘View Source’ tab, select ‘compare’ and ‘show differences’ to highlight what has changed in the HTML.

It’s relatively trivial to add more elements to change detection, so do let us know if there’s anything else you’d like to see included here.

4) URL Mapping For Staging / Different URL Structure Comparison

As part of crawl comparison, we have introduced a URL Mapping feature, which allows you to compare two different URL structures effectively. This is helpful when comparing a production site against staging, or a website that uses separate mobile URLs against its desktop equivalent.

To set this up, click on the compare configuration and ‘URL Mapping’. Then input a regex to map the previous crawl URLs into the current crawl. For example, we recently switched host to Kinsta and tested a staging site pre to migrating using this new feature.

The Kinsta staging site is then rewritten to the production site and the equivalent URLs will be compared against each other for overview tab data, issues, and opportunities, the site structure tab, and change detection.

You can switch between the current and previous crawls to view the different URL structures still.

If a client has made much smaller changes like removed trailing slashes from URLs in staging, this feature can also be used to help compare the different versions of the same pages with minimal effort as well.

This will hopefully make SEOs lives a little bit easier during testing for new site releases and migrations.

Other Updates

Version 15.0 also includes a number of smaller updates and bug fixes, outlined below.

• Math Solvers and Practice Problems Google rich result features are now supported in structured data validation.
• There’s a new ‘Crawl Timestamp’ column in the Internal tab, which should help with automation, reporting and debugging.
• Project folders within the ‘File > Crawls’ menu are now collapsed by default.
• The URL bar will now default to HTTPS if you don’t specify the protocol.
• Fixed a blurry interface issue on high resolution screens on both Windows and Linux (for any scaling setting).
• Fixed many bugs that are too monotonous to include in any detail. You’re not even reading this last bullet point, so why am I writing it?

That’s all for now. We think these features help raise the SEO Spider to a new level, so hopefully, you find them useful. Please see our tutorial on ‘How To Compare Crawls‘ for more on how to use all the features released above. If you experience any issues, please let us know via support and we’ll help.

Thank you to everyone for all their feature requests, feedback, and continued support.

Now, go and download version 15.0 of the Screaming Frog SEO Spider and let us know what you think!

Small Update – Version 15.1 Released 14th April 2021

We have just released a small update to version 15.1 of the SEO Spider. This release is mainly bug fixes and small improvements –

• Fix bug that allowed selection of a project directory in ‘Compare’ mode triggering a crash.
• Fix bug in crawls dialog with crawls not being ordered by modified date.
• Fix tool tip alignment issues in Compare mode.
• Fix bug in JavaScript rendering causing a stall when crawling PDFs.
• Fix bug where re-spidered URLs that previously failed rendering appeared in the External tab.
• Fix crash in Locales that use non ASCII numbering systems when crawling sites with multiple instances of HTML elements.
• Fix crash removing URLs from Tree View.

---

Small Update – Version 15.2 Released 18th May 2021

We have just released a small update to version 15.2 of the SEO Spider. This release is mainly bug fixes and small improvements –

• Include ‘Current Indexability Status’ (and previous) columns in ‘Change Detection > Indexability’.
• Added right-click ‘Respider’ to details panel.
• Add status and status code to right click ‘Export > Inlinks’.
• Improve usability of comparison mode crawls dialog.
• Improve usability of Crawl Comparison URL Mapping UI.
• Remove new lines from Link Paths.
• Fix various issues with Forms based authentication.
• Fix bug where cells containing lots of data would cause Google Sheets exports to fail.
• Fix bug reading sitemaps containing a DOCTYPE.
• Fix bug where not all image types were shown in the Image Preview tab.
• Fix crash performing a search in the master view after changing a Custom Search or Extraction.
• Fix bug where doing a respider on loaded in completed crawl with GSC/GA looses columns in master view when saved.
• Fix bug with change detection metrics missing when upgrading crawl to 15.2.
• Fix bug with Server Response Times columns showing incorrect data if ‘server response times category’ is also configured.
• Fix bug with persistent spell check dictionary being cleared on startup.
• Fix bug with Word Cloud Visualisation HTML export not working.
• Fix bug with trailing slashes being stripped from file names in screenshots bulk export.
• Fix bug with class name shown in Gnome menu on Ubuntu 18.04.
• Fix bug with SERP description clipping behind table.
• Fix crash in Image Details preview pane.
• Fix bug with PSI exports showing headings as Secs, but data as MS.
• Fix Duplicate ‘Max Image Size’ label in Spider Preferences.
• Fix crash with using an invalid regex in Regex Replace UI.