Screaming Frog SEO Spider Update – Version 19.0

We’re delighted to announce Screaming Frog SEO Spider version 19.0, codenamed internally as ‘Peel’.

This update contains a number of significant updates, new features and enhancements based upon user feedback and a little internal steer.

Let’s take a look at what’s new.

---

1) Updated Design

While subtle, the GUI appearance has been refreshed in look and feel, with crawl behaviour functions (crawl a subdomain, subfolder, all subdomains etc) moved to the main nav for ease.

These options had previously been within the configuration, so this makes them accessible to free users as well.

There’s now alternate row colours in the main tables, updated icons and even the Twitter icon and link have been removed (!). While the UX, tabs and filters are much the same, the configuration has received an overhaul.

---

2) Unified Config

The configuration has been unified into a single dialog, with links to each section. This makes adjusting the config more efficient than opening and closing each separately. The naming and location of config items should be familiar to existing users, while being easier to navigate for new users.

There’s been a few small adjustments, such as saving and loading configuration profiles now appearing under ‘Configuration’, rather than the ‘File’ menu.

System settings such as user interface, language, storage mode and more are available under ‘File > Settings’ on Windows and Linux or ‘Screaming Frog SEO Spider > Settings’ on macOS, in their own unified configuration.

You can also ‘Cancel’ any changes made by using the cancel button on the configuration dialog.

---

3) Segments

You can now segment a crawl to better identify and monitor issues and opportunities from different templates, page types, or areas of priority.

The segmentation config can be accessed via the config menu or right-hand ‘Segments’ tab, and it allows you to segment based upon any data found in the crawl, including data from APIs such as GA or GSC, or post-crawl analysis.

You can set up a segment at the start, during, or at the end of a crawl. There’s a ‘segments’ column with coloured labels in each tab.

When segments are set up, the right hand ‘Issues’ tab includes a segments bar, so you can quickly see where on the site the issues are at a glance.

You can then use the right-hand segments filter, to drill down to individual segments.

There’s a new right-hand ‘Segments’ tab with an aggregated view, to quickly see where issues are by segment.

You can use the Segments tab ‘view’ filter to better analyse items like crawl depth by segment.

Or which segments have different types of issues.

Once set-up, segments can be saved with the configuration. Segments are fully integrated into various other features in the SEO Spider as well.

In crawl visualisations, you can now choose to colour by segment.

You can also choose to create XML Sitemaps by segment, and the SEO Spider will automatically create a Sitemap Index file referencing each segmented sitemap.

Within the Export for Looker Studio for automated crawl reports, a separate sheet will also be automatically created for each segment created. This means you can monitor issues by segment in a Looker Studio Crawl Report as well.

---

4) Visual Custom Extraction

Custom Extraction is a super powerful feature in the SEO Spider, but it’s also quite an advanced feature and many users couldn’t care less about learning XPath or CSSPath (understandably, so).

To help with this, you’re now able to open a web page in our inbuilt browser and select the elements you wish to extract from either the web page, raw HTML or rendered HTML. We’ll then formulate the correct XPath/CSSPath for you, and provide a range of other options as well.

Just click the web page icon to the side of an extractor to bring up the browser –

Input the URL you wish to scrape, and then select the element on the page. The SEO Spider will then highlight the area you wish to extract, and create an expression for you, with a preview of what will be extracted based upon the raw or rendered HTML.

You can switch to Rendered or Source HTML view and pick a line of HTML as well. For example, if you wish to extract the ‘content’ of an OG tag –

You can then select the attribute you wish to extract from the dropdown, and it will formulate the expression for you.

In this case below, it will scrape the published time, which is shown in the source and rendered HTML previews after selecting the ‘content’ attribute.

For those of you that have mastered XPath, CSSPath and regex, you can continue to input your expressions in the same way as before.

At the moment this new feature doesn’t help with extracting JS, but we plan on extending this functionality to help scrape conceivably anything from the HTML.

---

5) 3D Visualisations

If you’re a fan of our crawl visualisations, then you’ll dig the introduction of a 3D Force-Directed Crawl Diagram, and 3D Force-Directed Directory-Tree Diagram.

They work in the same way as existing crawl and directory-tree visualisations, except (yes, you guessed it) they are 3D and allow you to move around nodes like you’re in space, and ‘within’ the visualisation itself.

These visualisations are restricted to 100k URLs currently, otherwise your machine may explode. At 100k URLs, the visualisation is also slightly bonkers.

Do they help you identify more issues, more effectively? Not really.

But they are fun, and sometimes that’s enough.

---

6) New Filters & Issues

There’s a variety of new filters and issues available across existing tabs that help better filter data, or communicate issues discovered.

The new filters available include the following –

• ‘URL > GA Tracking Parameters’ – URLs that contain Google Analytics tracking parameters. In addition to creating duplicate pages that must be crawled, using tracking parameters on links internally can overwrite the original session data. utm= parameters strip the original source of traffic and starts a new session with the specified attributes. _ga= and _gl= parameters are used for cross-domain linking and identify a specific user, including this on links prevents a unique user ID from being assigned.
• ‘Page Titles > Outside <head>’ – Pages with a title element that is outside of the head element in the HTML. The page title should be within the head element, or search engines may ignore it. Google will often still recognise the page title even outside of the head element, however this should not be relied upon.
• ‘Meta Description > Outside <head>’ – Pages with a meta description that is outside of the head element in the HTML. The meta description should be within the head element, or search engines may ignore it.
• ‘H1 > Alt Text in h1’ – Pages which have image alt text within an h1. This can be because text within the image is considered as the main heading on the page, or due to inappropriate mark-up. Some CMS templates will automatically include an h1 around a logo across a website. While there are strong arguments that text rather than alt text should be used for headings, search engines may understand alt text within an h1 as part of the h1 and score accordingly.
• ‘H1 > Non-sequential’ – Pages with an h1 that is not the first heading on the page. Heading elements should be in a logical sequentially-descending order. The purpose of heading elements is to convey the structure of the page and they should be in logical order from h1 to h6, which helps navigating the page and users that rely on assistive technologies.
• ‘H2 > Non-sequential’ – Pages with an h2 that is not the second heading level after the h1 on the page. Heading elements should be in a logical sequentially-descending order. The purpose of heading elements is to convey the structure of the page and they should be in logical order from h1 to h6, which helps navigating the page and users that rely on assistive technologies.
• ‘Content > Soft 404 Pages’ – Pages that respond with a ‘200’ status code suggesting they are ‘OK’, but appear to be an error page – often referred to as a ‘404’ or ‘page not found’. These typically should respond with a 404 status code if the page is no longer available. These pages are identified by looking for common error text used on pages, such as ‘Page Not Found’, or ‘404 Page Can’t Be Found’. The text used to identify these pages can be configured under ‘Config > Spider > Preferences’.
• ‘Content > Lorem Ipsum Placeholder’ – Pages that contain ‘Lorem ipsum’ text that is commonly used as a placeholder to demonstrate the visual form of a webpage. This can be left on web pages by mistake, particularly during new website builds.
• ‘Images > Missing Size Attributes’ – Image elements without dimensions (width and height size attributes) specified in the HTML. This can cause large layout shifts as the page loads and be frustrating experience for users. It is one of the major reasons that contributes to a high Cumulative Layout Shift (CLS).
• ‘Images > Incorrectly Sized Images’ – Images identified where their real dimensions (WxH) do not match the display dimensions when rendered. If there is an estimated 4kb file size difference or more, the image is flagged for potential optimisation. In particular, this can help identify oversized images, which can contribute to poor page load speed. It can also help identify smaller sized images, that are being stretched when rendered.
• ‘Canonicals > Outside <head>’ – Pages with a canonical link element that is outside of the head element in the HTML. The canonical link element should be within the head element, or search engines will ignore it.
• ‘Directives > Outside <head>’ – Pages with a meta robots that is outside of the head element in the HTML. The meta robots should be within the head element, or search engines may ignore it. Google will typically still recognise meta robots such as a ‘noindex’ directive, even outside of the head element, however this should not be relied upon.
• ‘Hreflang > Outside <head>’ – Pages with an hreflang link element that is outside of the head element in the HTML. The hreflang link element should be within the head element, or search engines will ignore it.
• ‘JavaScript > Pages with JavaScript Errors’ – Pages with JavaScript errors captured in the Chrome DevTools console log during page rendering. While JavaScript errors are common and often have little effect on page rendering, they can be problematic – both in search engine rendering, which can hinder indexing, and for the user when interacting with the page. View console error messages in the lower ‘Chrome Console Log’ tab, view how the page is rendered in the ‘Rendered Page’ tab, and export in bulk via ‘Bulk Export > JavaScript > Pages With JavaScript Issues’.
• ‘Validation > <body> Element Preceding <html>’ – Pages that have a body element preceding the opening html element. Browsers and Googlebot will automatically assume the start of the body and generate an empty head element before it. This means the intended head element below and its metadata will be seen in the body and ignored.

In some cases new data is collected and reported alongside the new filter. For example, we now collect image element dimension attributes, display dimensions and their real dimensions to better identify oversized images, which can be seen in the image details tab and various bulk exports.

---

Other Updates

Version 19.0 also includes a number of smaller updates and bug fixes.

• There’s a new ‘focus’ mode under ‘View’ in the top-level menu, that auto hides tabs that are not in use to reduce clutter.
• You can enable JavaScript error reporting (‘Config > Spider > Rendering’) and view JS errors, warnings and issues using the lower ‘Chrome Console Log’ tab.
• Pre-set user-agents have been updated to include more Google crawler variants, such as Googlebot Image, News, Google-Inspection Tool etc. Thanks to Jaroslav for the suggestion.
• The SERP snippet tab has been updated to match Google’s latest SERP design, including favicon.
• The Windows installer has been updated with more options around saving a shortcut etc.
• Updated Forms Based Authentication to support Local Storage.
• Introduced a configurable trust store to allow users to connect to GA/GSC via outbound proxies.
• Java has been updated to 17.0.7 and all dependencies have been updated to latest versions.
• Updated another million Google rich result features due to changes for structured data validation.

That’s everything for this release!

Thanks to everyone for their continued support, feature requests and feedback. Please let us know if you expericence any issues with version 19.0 of the SEO Spider via our support.

Small Update – Version 19.1 Released 27th July 2023

We have just released a small update to version 19.1 of the SEO Spider. This release is mainly bug fixes and small improvements –

• Added Trusted Certificates UI for users in some corporate environments. This can be found under ‘File > Settings > Trusted Certificates’.
• Updated new Windows installer to support silent uninstallation.
• Adjusted UI layout in all languages to improve appearance.
• Fixed issue with being unable to edit a scheduled task once it has been run on macOS.
• Fixed issue with regex matching for segments and column searching.
• Fixed issue with custom extraction columns not being show when loading a crawl in database mode.
• Fixed various crashes.

Small Update – Version 19.2 Released 30th August 2023

We have just released a small update to version 19.2 of the SEO Spider. This release is mainly bug fixes and small improvements –

• Improve messaging in list mode to make it clear how many URLs have been removed by excludes, includes or being duplicates.
• Improve rendering disabled dialog messaging clearer.
• Add support for Session Storage to Forms Authentication.
• Add more retries when writing to Google Drive to make exports more robust.
• Display JSON-LD Parse issues Structured Data details tab.
• Remove brackets from sheet names in Excel.
• Resolved a false positive in multiple body tag filter.
• Fixed issue with PSI metric vanishing after PSI API update.
• Fixed issue with using long XPath expressions in Custom Extraction.
• Fixed issue where timestamp of crawl was modified but and not modifiying it.
• Fixed issue with negative crawl depth appearing in the Crawl Path Report.
• Fixed issue with ‘issues’ not being populated in comparision mode.
• Fixed issue with being unable to edit tasks on macOS.
• Fixed issue with GA4 not finding data for some URLs for customers with a large number of URLs in GA4.
• Fixed issue with Export in list mode not matching uploading URLs correctly.
• Fix issue when loading in crawls where a needless backup was performed, slowing things down.
• Fixed various crashes.

Small Update – Version 19.3 Released 31th October 2023

We have just released a small update to version 19.3 of the SEO Spider. This release is mainly bug fixes and small improvements –

• Added Subscription, Paywalled Content and Vehicle Listing rich result features to structured data validation.
• Removed Google How-To feature.
• Updated to Schema.org version 23.
• Bring Sitemap only user-agent directive handling in line with Googlebot’s updated behaviour.
• Introduced synced scrolling of duplicate content frames in the Duplicate Details Tab.
• Updated ‘View > Reset Columns for All Tables’ to also reset visibility if disabled.
• Added right-hand Segments Tab & Overview filter to ‘Reports’ to allow exporting in Scheduling / CLI.
• Added indication as to why ‘OK’ button is disabled when there is an error on another tab in system config.
• Fixed issue with page transfer size missing in rendered crawls.
• Fixed issue with image in ‘Missing Alt Attribute’ filter during JavaScript rendering.
• Fixed issue with PageSpeed tab not appearing if you enable PSI mid-crawl with focus mode enabled.
• Fixed issue with Microdata not parsing.
• Fixed issue with unreadable scheduled crawl history errors.
• Fixed issue where Forms Auth was not using the user-agent from the working config.
• Fixed issue with various sites not loading in forms based authentication.
• Fixed various disaply issues with the config windows.
• Fixed various crashes.
• Update Chrome to resolve WebP exploit CVE-2023-4863.

Small Update – Version 19.4 Released 7th November 2023

We have just released a small update to version 19.4 of the SEO Spider. This release is mainly bug fixes –

• Fixed an issue with JavaScript rendering and redirects.
• Fixed an issue causing a crash when seleting the bounce rate metric in GA4.

Small Update – Version 19.5 Released 12th February 2024

We have just released a small update to version 19.5 of the SEO Spider. This release is mainly bug fixes and small improvements –

• In preparation for Google replacing FID with INP as a Core Web Vital metric on the 12th of March, we’ve switched FID to INP in our Core Web Vitals Assessment.
• Reorder our Core Web Vitals to match how they display in Google.
• Remove hyphens from XML Sitemaps causing issues with GSC.
• Update various Structured Data validation defintions.
• Update apache-commons-text to 1.11.0.
• Update to Java 17.0.10.
• Prevent users from choosing incompatiable GA4 metrics.
• Allow Export for Looker Studio items to be copyable.
• Add missing translations.
• Add missing icon for Ubuntu users
• Update Ubunut dependancies.
• Fixed issue with Chrome Console Log Numbers not clearing when Clear button is clicked.
• Fixed issue with dropdowns for Google accounts not being show for Linux users when editied via Scheduling.
• Fixed issue with pausing a large crawl being slow.
• Fixed issue with API related tabs not appearing in Focus Mode.
• Fixed a couple of crashes.

Small Update – Version 19.6 Released 20th February 2024

We have just released a small update to version 19.6 of the SEO Spider. This release is mainly bug fixes and small improvements –

• Update regression in Include behaviour.
• Fixed a crash on macOS when the app looses focus.

Small Update – Version 19.7 Released 11th March 2024

We have just released a small update to 19.7 of the SEO Spider. This release is mainly bug fixes and small improvements –

• Fixed issue not being able to pause a crawl when API data is loaded post crawl.
• Fixed issue with double hyphen not working in some URLs.
• Fixed crash entering file:// prefixed URLs in Visual Custom Extraction, this is no longer allowed.
• Fixed crash when dealing wtih really large robots.txt files.

Small Update – Version 19.8 Released 12th March 2024

This release superseded 19.7 with a single bug fix –

• Fix issue preventing scheduled tasks with ‘Export for Looker Studio’ being loaded.