Screaming Frog SEO Spider Update – Version 23.0

We’re quite pleased to announce Screaming Frog SEO Spider version 23.0, codenamed internally as ‘Rush Hour’.

The SEO Spider has a number of integrations, and the core of this release is keeping these integrations updated for users to avoid breaking changes, as well as smaller feature updates.

So, let’s take a look at what’s new.

---

1) Lighthouse & PSI Updated to Insight Audits

Lighthouse and PSI are being updated with the latest improvements to PageSpeed advice, which has now been reflected in the SEO Spider.

This comes as part of the evolution of Lighthouse performance audits and DevTools performance panel insights into consolidated and consistent audits across tooling.

In Lighthouse 13 some audits have been retired, while others have been consolidated or renamed.

The updates are fairly large and include changes to audits and naming in Lighthouse that users have long been familiar with, including breaking changes to the API. For example, previously separate insights around images have been consolidated into a single ‘Improve Image Delivery’ audit.

There are pros and cons to these changes, but after the initial frustration with having to re-learn some processes, the changes do mostly make sense.

The groupings make providing some recommendations more efficient, and it’s still largely possible to get the granular detail required in bulk from consolidated audits.

To get ahead of the (breaking) changes, we have updated our PSI integration to match those across Lighthouse and PSI for consistency.

The changes to PageSpeed Issues are listed below.

7 New Issues –

• Document Request Latency
• Improve Image Delivery
• LCP Request Discovery
• Forced Reflow
• Avoid Enormous Network Payloads
• Network Dependency Tree
• Duplicated JavaScript

11 Removed or Consolidated Issues –

• Defer Offscreen Images
• Preload Key Requests
• Efficiently Encode Images (now part of ‘Improve Image Delivery‘)
• Properly Size Images (now part of ‘Improve Image Delivery‘)
• Serve Images in Next-Gen Formats (now part of ‘Improve Image Delivery‘)
• Use Video Formats for Animated Content (now part of ‘Improve Image Delivery‘)
• Enable Text Compression (now part of ‘Document Request Latency‘)
• Reduce Server Response Times (TTFB) (now part of ‘Document Request Latency‘)
• Avoid Multiple Page Redirects (now part of ‘Document Request Latency‘)
• Preconnect to Required Origins (now part of ‘Network Dependency Tree‘)
• Image Elements Do Not Have Explicit Width & Height (now part of ‘Layout Shift Culprits‘)

6 Renamed Issues –

• Eliminate Render-Blocking Resources (now ‘Render Blocking Requests‘)
• Avoid Excessive DOM Size (now ‘Optimize DOM Size‘)
• Serve Static Assets with an Efficient Cache Policy (now ‘Use Efficient Cache Lifetimes‘)
• Ensure Text Remains Visible During Webfont Load (now ‘Font Display‘)
• Avoid Large Layout Shifts (now ‘Layout Shift Culprits‘)
• Avoid Serving Legacy JavaScript to Modern Browsers (now ‘Legacy JavaScript‘)

Older metrics, such as first meaningful paint, have also been removed.

If you’re running automated crawl reports in Looker Studio and have selected PageSpeed data, then you will find that some columns will not populate after updating.

The next time you ‘edit’ the scheduled crawl task, you will also be required to make some updates due to these breaking changes.

The SEO Spider provides an in-app warning, and advice in our Looker Studio Export Breaking Changes FAQ on how to solve it quickly.

---

2) Ahrefs v3 API Update

The SEO Spider has been updated to integrate v3 of the Ahrefs API after they announced plans to retire v2 of the API and introduced Ahrefs Connect for apps.

This allows users on any paid plan (not just enterprise) to access data from their latest API via our integration. The format is similar to the previous integration; however, users will be required to re-authenticate using the new OAuth flow introduced by Ahrefs.

You’re able to pull metrics around backlinks, referring domains, URL rating, domain rating, organic traffic, keywords, cost and more.

Metrics have been added, removed and renamed where appropriate. Read more on the Ahrefs integration in our user guide.

---

3) Auto-Deleting Crawls (Crawl Retention)

Crawls are automatically saved and available to be opened or deleted via the ‘File > Crawls’ menu in default database storage mode. To date, the only way to delete database crawls was to delete them manually via this dialog.

However, users are now able to automate deleting crawls via new ‘Crawl Retention‘ settings available in ‘File > Settings > Retention’.

You do not need to worry about disappearing crawls, as by default the crawl retention settings are set to ‘Never’ automatically delete crawls.

The crawl retention functionality allows users to automatically delete crawls after a period of time, which can be useful for anyone who doesn’t want to keep crawls but does want to take advantage of the scale that database storage mode offers (over memory storage).

As part of this feature, we also introduced the ability to ‘Lock’ projects or specific crawls in the ‘File > Crawls’ menu from being deleted. If you wish to lock a single crawl or all crawls in a project, just right click and select ‘Lock’.

For project folders, this will lock all existing and future crawl files, including scheduled crawls, from being automatically deleted via the retention policy settings.

---

4) Semantic Similarity Embedding Rules

You can now set embedding rules via ‘Config > Content > Embeddings’, which allows you to define URL patterns for semantic similarity analysis.

This means if you’re using vector embeddings for redirect mapping, as an example, you can add a rule to only find semantic matches for the staging site on the live website (so pages from the staging website itself are not considered as well).

In the above, the closest semantically similar address can only be the staging site for the live site.

This can also be used in a variety of other ways, such as if you wanted to see the closest matches between two specific areas of a website, or between a page and multiple external pages.

---

5) Display All Links in Visualisations

It’s now possible to see all inlink and outlink relationships in our site visualisations.

You can right-click on a node and select to ‘Show Inlinks’, ‘Show Inlinks to Children’ of a node, or perform the same for ‘Outlinks’.

This will update the visualisation to show all incoming links, which can be useful when analysing internal linking to a page or a section of a website –

Linking nodes are highlighted in green, while other nodes are faded to grey.

This option is available across all force-directed diagrams, including the 3D visualisations.

---

6) Display Links in Semantic Content Cluster Diagram

Similar to site visualisations, you’re now also able to right-click and ‘Show Inlinks’ or outlinks within the semantic Content Cluster Diagram.

As well as viewing all internal links to a page, you can also select to ‘Show Inlinks Within Cluster’, to see if a page is benefiting from links from semantically similar pages.

This can be a useful way to visually identify internal link opportunities or gaps in internal linking based upon semantics.

---

Other Updates

Version 23.0 also includes a number of smaller updates and bug fixes.

• Limit Crawl Total Per Subdomain – Under ‘Config > Spider > Limits’ you can limit the number of URLs crawled per subdomain. This can be useful in a variety of scenarios, such as crawling a sample number of URLs from X number of domains in list mode.
• Improved Heading Counts – The ‘Occurrences’ count in the h1 and h2 tabs was limited to ‘2’, as only two headings are extracted by default. While only the first two h1 and h2 headings will continue to be extracted, the occurrence number will show the total number of each on a page.
• Move Up & Down Buttons for Custom Search, Extraction & JS for Ordering – The ordering (top to bottom) impacts how they are displayed in columns in tabs (from left to right), so this can now be adjusted without having to delete and add.
• Configurable Percent Encoding of URLs – While percent encoding of URLs is uppercase, a small number of servers will redirect to lowercase only and error. This is therefore configurable in ‘Config > URL Rewriting’.
• Irish Language Spelling & Grammar Support – Just for the craic!
• Updated AI Models for System Prompts & JS Snippets – OpenAI system prompts have been updated to use ‘gpt-5-mini’, Gemini to ‘gemini-2.5-flash’ and Anthropic to ‘claude-sonnet-4-5’. As always, we recommend reviewing these models, and costs prior to use.
• New Exports & Reports – There’s a new ‘All Error Inlinks’ bulk exports under ‘Bulk Export > Response Codes > Internal/External’ which combine no response, 4XX and 5XX errors. There is also a new ‘Redirects to Error’ report under ‘Reports > Redirects’, which includes any redirects which end up blocked, no response, 4XX or 5XX error.
• Redirection (HTTP Refresh) Filter – While these were reported and followed, a new filter has been introduced to better report these.

That’s everything for version 23.0!

Thanks to everyone for their continued support, feature requests and feedback. Please let us know if you experience any issues with this latest update via our support.