Screaming Frog SEO Spider Update – Version 24.0

We’re delighted to announce Screaming Frog SEO Spider version 24.0, codenamed internally as ‘bolus’.

In this release we have introduced a number of much demanded new features, as well as smaller quality of life improvements and updates.

Let’s take a look at what’s new.

---

1) Screaming Frog SEO Spider MCP

You can now run crawls, analyse, export and manipulate data using the SEO Spider and node.js within Claude, LM Studio and other AI chat assistants.

The MCP allows you to connect to the SEO Spider and automate countless crawl related activities using natural language. From the basic, such as running and summarising a crawl.

To more fun, and advanced tasks.

Such as manipulating or visualising the data the way you want to view it.

As well as performing the leg work for exporting, combining data, running the analysis pipeline and installing relevant scripts for interactive visuals for greater insight.

Like this pretty link equity analysis we made earlier using the Ahrefs integration in a crawl.

Integrating with Claude Cowork, also opens up lots of possibilities.

There are plenty of use cases, ultimately the idea of the integration is to make daily tasks and workflows that much more efficient. This is not a replacement for an experienced SEO professional, obviously.

To set this up, please refer to our documentation on the MCP Server.

Please note – Not every feature and function is currently supported in the SEO Spider MCP. However, we plan to extend functionality based upon user feedback.

So let us know what else you’d like to see!

---

2) Auto Compare Crawls

You’re now able to select ‘Auto Compare Crawls’ for scheduled (and CLI) crawls.

This can be set up in the scheduled crawl task, which will then auto compare the last two crawls performed in a project.

This allows you to quickly view a crawl comparison of the last two crawls, including all changes that have occurred since the last crawl.

Read more in our crawl comparison tutorial.

---

3) View Crawl Changes in Email Notifications

If you have enabled crawl completion notifications, you will already be familiar with the email summary upon crawl completion.

This has been improved to include all issues found within the crawl in a table below the summary.

If you have selected to ‘Auto Compare Crawls’, you will see a crawl comparison of issues in the email – highlighting any changes in issues found between the last two crawls.

This is a great way to monitor and be alerted to website changes and SEO issues – such as pages being added or removed, a sudden spike in non-indexable pages, or page titles disappearing.

All without having to open up the crawl.

---

4) Send Crawl Export Attachments by Email

You’re now also able to send emails with export attachments automatically upon crawl completion.

This can be set up in the scheduled crawl task under ‘Notifications‘.

For example, if you need to send a weekly broken links export to a developer, or a crawl issues summary to a stakeholder, you can input their email and the report of choice and it will appear in their inbox.

Attachments are zipped and the email will warn the receiver if there is an error, such as attachments being too large in size for the email provider.

---

5) Find Uncrawlable Links

You can now find common uncrawlable link types in the SEO Spider.

These can be viewed in the Links tab and the new ‘Pages With Uncrawlable Internal Outlinks‘ filter.

There is a new ‘Link Crawlability’ column in the lower Outlinks tab, which helps to identify any of these link types as ‘Uncrawlable’. They can be exported in bulk via the ‘Bulk Export > Links > Pages With Uncrawlable Internal Outlinks’ export.

Uncrawlable links are links found in the HTML, but do not conform to link best practices outlined by Google. For example:

<span href="https://example.com">
<div href="https://example.com">
<a onclick="goto('https://example.com')">
<a href="javascript:goTo('products')">


To find uncrawlable link types, please ensure ‘Store’ is enabled in the ‘Config > Spider > Crawl’ menu.

There isn’t an option to crawl these currently, but this might be available in the future as we refine how we parse some of these link types.

Please note – Our research indicates Google will crawl anything in the HTML that looks like a link. They will attempt to parse it and use for discovery. So strictly speaking, many of these link types will likely be ‘crawlable’.

However they are not recommended, and should not be relied upon. They might not be treated in the same way as a recommended hyperlink for passing link signals.

---

6) Usage Stats

If you’ve ever wondered how much time you spend crawling, then you can now see your own usage data via ‘Help > Usage Stats’.

This has limited practical use (!), but is fun and might give you a better idea of how much value you’re getting from the tool each year. This data is stored locally on your machine.

---

Other Updates

Version 24.0 also includes a number of smaller updates and bug fixes.

• New Arm64 Linux Versions – We now have two more packages available for Arm users for both Ubuntu and Fedora. Our three Linux users will rejoice again!
• Improved Reporting of Syntactically Invalid Links – There’s a new ‘Crawl Invalid Links‘ option under ‘Config > Spider > Crawl’. Enabling this setting means URLs that are syntactically invalid such as hppts://example.com will be parsed, crawled and reported upon. These will typically appear under ‘Response Codes > No Response’ as malformed links.
• Skip Empty Reports – You can now skip exporting empty reports in ‘Output Settings’ of scheduled crawl tasks, and ‘Bulk Export > Multi Export’. This means any exports that have been selected that do not contain any data, will not be exported. They will also not be attached in email notifications exports.
• Ahrefs Country Level Metrics – There’s a new ‘Options’ tab in the Ahrefs integration with filters to adjust data for countries and volume (monthly or average).
• Filterable Content Clusters – You’re now able to click on the content cluster legends in the graph to enable and disable their appearance in the diagram.
• Live Model Validation & List In AI Integrations – When you connect to an AI provider, the model selected in the prompt will be checked against the live model list endpoint and alert you if they don’t exist – and allow you to select a live model. With models changing so frequently, both system and user prompts can quickly become deprecated.
• System Wide Prompt for AI Integrations – In the ‘Advanced’ tab of each AI provider integration, you’re able to include a chat system prompt to establish the role, personality, constraints, and overall behaviour, providing foundational instructions that guide all its responses for consistency and relevance.
• AI Provider Token Usage – Token usage is now displayed on the ‘Account Information’ dialog for each AI provider when connected.
• Ollama Image Generation & Request Timeout – The Ollama integration now supports their image generation and has an adjustable ‘Request Timeout’ in the advanced tab.
• Renamed Looker Studio Back to Data Studio Again – For our automated crawl reports in Looker Data Studio. Google is definitely messing with us all at this point.
• Java 25 – One for the purists. Version 24 has been updated to Java 25.

That’s everything for version 24.0!

Thanks to everyone for their continued support, feature requests and feedback. Please let us know if you experience any issues with this latest update via our support.