Screaming Frog SEO Spider Update – Version 17.0

Dan Sharp

Posted 17 August, 2022 by in Screaming Frog SEO Spider

Screaming Frog SEO Spider Update – Version 17.0

We’re pleased to announce Screaming Frog SEO Spider version 17.0, codenamed internally as ‘Lionesses’.

Since releasing the URL Inspection API integration in the SEO Spider and the launch of version 5 of the Log File Analyser, we’ve been busy working on the next round of prioritised features and enhancements.

Here’s what’s new in our latest update.


1) Issues Tab

There’s a new ‘Issues’ right-hand tab, which details issues, warnings and opportunities discovered.

This tab has been introduced to help better direct users to potential problems and areas for improvement, based upon existing filters from the ‘Overview’ tab.

Screaming Frog SEO Spider Issues Tab

An in-app explanation of each issue and potential actions is provided in English, German, Spanish, French and Italian.

Each issue has a ‘type’ and an estimated ‘priority’ based upon the potential impact.

  • Issues are an error or issue that should ideally be fixed.
  • Opportunities are potential areas for optimisation and improvement.
  • Warnings are not necessarily an issue, but should be checked – and potentially fixed.

E.g – An ‘Internal URL Blocked by Robots.txt’ will be classed as a ‘warning’, but with a ‘High’ priority as it could potentially have a big impact if incorrectly disallowed.

For experienced users, the new Issues tab is a useful way to quickly identify top-level problems and dive straight into them as an alternative to the Overview tab. For users with less SEO expertise, it will help provide more direction and guidance on improving their website.

All Issues can be exported in bulk via ‘Bulk Export > Issues > All’. This will export each issue discovered (including their ‘inlinks’ variants for things like broken links) as a separate spreadsheet in a folder (as a CSV, Excel and Sheets).

Bulk Export All Issues

The new Issues tab also works with crawl comparison similar to the Overview tab, to allow users to identify where issues have changed and monitor progress over time.

Crawl comparison for Issues

The SEO Spider has always been built for experts and we have been reticent to ever tell our users how to do SEO, as SEO requires context. The new issues tab does not replace SEO expertise and a professional who has that context of the business, strategy, objectives, resource, website and nuances related to SEO and prioritising what’s important.

The new issues tab should provide helpful hints and support for an SEO who can make sense of the data and interpret it into appropriate prioritised actions.


2) Links Tab

There’s a new ‘Links’ tab which helps better identify link-based issues, such as pages with a high crawl-depth, pages without any internal outlinks, pages using nofollow on internal links, or non-descriptive anchor text.

Screaming Frog SEO Spider Links Tab

Filters such as ‘high’ internal and external outlinks and non-descriptive anchor text can be customised under ‘Config > Spider > Preferences’ to user-preferred limits.

All data can be exported alongside source pages via the ‘Bulk Export > Links’ menu as well.


3) New Limits

Users are now able to control the number of URLs crawled by URL Path for improved crawl control and sampling of template types.

Under ‘Config > Spider > Limits’ there’s now a ‘Limit by URL Path’ configuration to enter a list of URL patterns and the maximum number of pages to crawl for each.

Limit by URL Path

In the example above a maximum of 1,000 product URLs will be crawled, which will be enough of a sample to make smarter decisions.

Users can also now ‘Limit URLs Per Crawl Depth’, which can help with better sampling in some scenarios.


4) ‘Multiple Properties’ Config For URL Inspection API

The URL Inspection API is limited to 2k queries per property a day by Google.

However, it’s possible to have multiple verified properties (subdomains or subfolders) for a website, where each individual property will have a 2k query limit.

Therefore, in the URL Inspection API configuration (‘Config > API Access > GSC > URL Inspection’) users can now select to use ‘multiple properties’ in a single crawl. The SEO Spider will automatically detect all relevant properties in the account, and use the most specific property to request data for the URL.

URL Inspection API Multiple Properties Config

This means it’s now possible to get far more than 2k URLs with URL Inspection API data in a single crawl, if there are multiple properties set up – without having to perform multiple crawls.

Please use responsibly. This feature wasn’t built for the purpose to circumvent Google’s limits and motivate users to create many different properties to get indexing data for every URL on a website. Google may adjust this limit (to domain etc) if it’s abused.


5) Apple Silicon Version & RPM for Fedora

There’s now a native Apple Silicon version available for users on M1/2 macs, and an RPM for Fedora Linux users.

Apple Silicon Version

In limited internal testing of the native Silicon version we found that:

  • Crawling a locally hosted site with very little latency resulted in the crawl running twice as quickly.
  • Loading in a saved crawl was 4 times faster.

The native experience is just so much smoother overall in comparison to using the Rosetta2 emulation layer.


6) Detachable Tabs

All tabs are now detachable. Users can right-click and ‘detach’ any tab from the main UI and move to a preferred position (across multiple screens).


This is pretty cool for users that like to keep one eye on a crawl while doing other things, or analysing all the things at once.

There are options to ‘re-attach’, ‘pin’ – and reset all tabs back to normal when it’s all too much, too.


Other Updates

Version 17.0 also includes a number of smaller updates and bug fixes.

  • The ‘Response Codes’ tab now has an additional ‘Internal’ and ‘External’ filter, which is reflected in the Overview tab and Crawl Overview Report. Thanks to Aleyda for the nudge on that one!
  • Tabs can be re-ordered more efficiently by dragging, or via a new ‘configure tabs’ menu available via a right-click on a tab or the tab down arrow.
  • The main scheduling UI has been updated with details of task name, next run, interval and has improved validation for potential issues that might stop it launching.
  • There’s a new ‘Save & Open’ option for exports, which works for all formats.
  • GA & GSC configs now have date pre-sets (last week, last month etc) which can be saved as part of the configuration and supplied in scheduled crawls.
  • The PSI integration now has Interaction to Next Paint (INP) and Time to First Byte (TTFB) metrics available.
  • The ‘Ignore Non-Indexable URLs for On-Page Filters’ configuration has been updated to ‘Ignore Non-Indexable URLs for Issues‘ and will by default, not flag non-indexable URLs for checks related to issues across a wider range of tabs and filters as detailed in-app and in the user guide.
  • The hreflang tag limit has been increased from 200 to 500, for websites that have lost the plot lots of alternative localised URLs.
  • All new tabs, filters and issues, warnings and opportunities counts are available in the ‘Export for Data Studio’ and Data Studio Crawl Report via scheduling.
  • The SEO Spider has been updated to Java 17.
  • Fixed lots of small bugs that nobody wants to read about, particularly when everyone is on holiday and nobody is reading this.

We hope the new update is useful.

A big thank you goes out to our beta testers who have helped with the different language versions of our new in-app issue descriptions. In particular, MJ Cachón for translation help, a new Spanish Google Data Studio crawl template – and generally being awesome.

As always, thanks to the SEO community for your continued support, feedback and feature requests. Please let us know if you experience any issues with version 17.0 of the SEO Spider via our support.


Small Update – Version 17.1 Released 23rd August 2022

We have just released a small update to version 17.1 of the SEO Spider. This release is mainly bug fixes and small improvements –

  • Fix issue preventing scheduling working on Ubuntu.
  • Fix issue preventing JavaScript crawling working on Ubuntu.
  • Fix cookies issues around forms based authentication.
  • Fix JavaScript crawling failing to render some pages.
  • Fix crash when changing filters in the Response Codes tab.
  • Fix crash crawling URLs with HTTP in the domain name.
  • Fix crash selecting ‘Save & Open’.

Small Update – Version 17.2 Released 12th September 2022

We have just released a small update to version 17.2 of the SEO Spider. This release is mainly bug fixes and small improvements –

  • Reinstate auto scrolling behaviour at the bottom of table view, so users can see URLs appear as they are dicovered. Thanks to Pete Mindenhall, Steve Morgan and a few others that spotted it.
  • Add handling for webp files in bad content filter.
  • Fix regression in cookie handling not showing all cookies.
  • Fix issue showing incorrect values for Total Internal/External URLs when ignoring URLs blocked by robots.txt.
  • Fix issue with last mode used not being persisted after restarts.
  • Fix issue with new Issues tab graphs not updating during a crawl.
  • Fix issue with new task dialog showing single digits for time.
  • Fix issue with selected filter being lost when returning to a tab.
  • Fix issue with old Response Codes 5XX export tab not working on upgrade.
  • Fix issue with Spelling & Grammer crashing when used on Apple Silicon.
  • Fix issue with Response Codes 4XX filtering showing 2XX responses after re-spider & reload.
  • Fix issue with grabled fonts affecting users with their own version of Roboto font installed.
  • Fix crash in view source panel.
  • Fix crash crawling robots.txt with long Sitemap: line.
  • Fix crash around detachable tabs and full screen on macOS.
  • Fix crash showing ‘Visualisations > Force-Directed Crawl Diagram’.
  • Fix crash removing/re-spidering URL.
  • Fix crash viewing scheduling history.


Dan Sharp is founder & Director of Screaming Frog. He has developed search strategies for a variety of clients from international brands to small and medium-sized businesses and designed and managed the build of the innovative SEO Spider software.

53 Comments

  • jules rosen 2 years ago

    I’ve just started using this free tool and I want to thank the Developers for a continuing to maintain and improve it

    Reply
  • Frederik Nørgaard 2 years ago

    Thank you Dan. The Issues Tab looks great. Looking forward to using it.

    Reply
  • Henry 2 years ago

    Great update, again.

    HenryV

    Reply
  • Ruslan 2 years ago

    Alright! Every M1 Max user is in heaven now ;)

    Reply
  • Prakash Humagain 2 years ago

    Awesome! It will be great if we can download straight as a pdf file and email to clients straight away. As most of SEO Agency offers free SEO audit and use random tools to create BS Audit report. Even me I don’t run screaming frog until we get contract signed but now as Screaming Frog itself offering this then would be nice to have these feature as well. Hope I made it clear and not asking more too much. Thanks for the update.

    Reply
  • Towarzysz 2 years ago

    testing new versions all day well done guys

    Reply
  • Rohan Jolly 2 years ago

    A very useful program to detect issues that affect user experience on websites. What I like best is that there is no forceful push towards one size fits all solutions that is so common with other automatic SEO tools.

    Reply
  • Everett Sizemore 2 years ago

    As always, thank you for the update and excellent tool. I think the detachable tabs and separating internal/external response codes will be very helpful for my needs.

    Reply
    • screamingfrog 2 years ago

      Thanks man, awesome to hear!

      Hope all is good with you.

      Dan

      Reply
  • Suhaib Kamran 2 years ago

    Game changer – issues tab was definitely needed. So glad to see it.

    Reply
    • Game changer indeed. We saw it on other SEO tools since years and we are so glad we have it in Screaming Frog already. Thanks guys, it will be in great use.

      Reply
  • Tom 2 years ago

    Very nice Update. The Issue Tab is so helpfull.
    Did you also change the Font of ScreamingFrog?

    Reply
    • screamingfrog 2 years ago

      Hi Tom,

      Good spot! Yes, we switched to use Roboto fairly recently (partly as some system fonts had become unreliable across platforms).

      Cheers.

      Dan

      Reply
  • Darrell Wilson 2 years ago

    The new Issues tab is superb and is incredibly useful. A great addition to an already amazing tool.

    Reply
  • Patrick 2 years ago

    Hey! We are reading this…and we’re not on holiday (yet) Great new stuff, loving it!

    Reply
  • Salman 2 years ago

    I am curious about how did you estimate impact number of each issue in the issues tab?
    Thanks

    Reply
    • screamingfrog 2 years ago

      Hi Salman,

      It’s an estimated impact based upon our expertise and common SEO knowledge / best practice.

      As per the post above, it doesn’t have context specific to each site or business.

      And hence why it’s only a best estimate, to try and identify where you might want to check first.

      An experienced SEO still needs to review and prioritise.

      Cheers,

      Dan

      Reply
  • Bond Cleaning 2 years ago

    It’s amazing Update..Thanks a lot

    Reply
  • Jonathan 2 years ago

    Awesome ! Thanks Dan !

    Best tool ever !

    Reply
  • Sashis 2 years ago

    Been using this tool for a long time and never once I have been disappointed. New updates are fire as well.

    Reply
  • Alex Juel 2 years ago

    Finally the ability to export all of the reports at once! This is awesome, thanks SF team!

    Reply
  • Axel Moreau 2 years ago

    Amazing update guys, long time SF fan. Detachable tabs will be a godsend.

    Reply
  • Axel 2 years ago

    After install. bad font – https://i.imgur.com/GfbuCts.jpeg

    Reply
    • screamingfrog 2 years ago

      Pop us an email to support@screamingfrog.co.uk if you need help :-)

      Cheers.

      Dan

      Reply
    • Mert 2 years ago

      Hii
      Can I ask you, how did you solve this problem?

      Reply
      • screamingfrog 2 years ago

        Hi Mert,

        It’s fixed in a beta, if you want to email in for it.

        Cheers.

        Dan

        Reply
  • Dawid 2 years ago

    Links Tab and Detachable Tabs are one of the most anticipated improvements. Incredibly useful functionalities. It’s great that Screaming Frog is developing so intensively.

    Reply
  • Canberk Balcilar 2 years ago

    Great update, thank you!

    Reply
  • Fabian K. 2 years ago

    Another great and useful update – thanks!

    Reply
  • Kayla moores 2 years ago

    Been using this tool for a long time, I love new update, more stuff I can dig in. Thanks for all you done till today.

    Reply
  • Corrado 2 years ago

    Another amazing update with amazing features!

    If i may, I would love to be able to export all the new “Issues” in a unique Excel file divided by tabs, or at least be able to choose which issues to export!
    This way users won’t be flooded by so many excel files!

    I hope you’ll consider adding this feature in a future update :)

    Thank you very much!

    Reply
  • dominicaparadise 2 years ago

    This update was so handy and eliminate some of bugs that i faced before it.
    keep going and release useful updates like this, thanks.

    Reply
  • Marc 2 years ago

    I’d love to see some easier ways to filter internal links.
    I.e. filter by only showing links that appear in the body of a page.
    Excluding header, footer and sidebar links.
    I understand it can be done already, but making it simpler would be great.

    Reply
    • screamingfrog 2 years ago

      Hi Marc,

      Thanks for the suggestion!

      Yeah, you can do this already – https://www.screamingfrog.co.uk/how-to-analyse-link-position/

      How would you make it simpler? Any thoughts appreciated, welcome to send through via email (support@screamingfrog.co.uk) if easier.

      Always open to any new ideas!

      Cheers.

      Dan

      Reply
  • Greg 2 years ago

    Always new features rolling up. Thank you all team, you do a great job

    Reply
  • ROYALSITE 2 years ago

    Finally the ability to export all of the reports at once! This is awesome, thanks SF team!

    Reply
  • Officina Seo 2 years ago

    I’m using this free tool since version 4 and I want to thank the Developers for a continuing to maintain and improve it

    Reply
  • Karandijital 2 years ago

    I would like to thank everyone who contributed to a great program that eased our workload.

    Reply
  • Joel 2 years ago

    Thank you for the update Dan!

    Regards from Spain

    Reply
  • Joris Has 2 years ago

    well this one made me shout a bit out loud…. “Limit by URL Path”

    Reply
  • Oliver 2 years ago

    Hey
    Is it just me? I can’t install 17.2. There are write permission problems with some .dll . so i need to abort the installation and screaming frog isn’t usable.

    Reply
    • screamingfrog 2 years ago

      Hi Oliver,

      Restart your machine, don’t open the app, and give it another go.

      It should work.

      Cheers.

      Dan

      Reply
      • Oliver 2 years ago

        Sorry my fault.
        An instance of screaming frog was running in the background….

        Reply
  • Ahmad iqbal 2 years ago

    That’s amazing updates. I am happy to see the new updates for beginners to easy way to find issues, warnings, opportunities,
    Thanks, my best wishes to you.

    Best regards
    Ahmad Iqbal SEO

    Reply
  • Roberto 1 year ago

    Epic as usual guys. Great update. Thanks for that.

    Reply
  • Mark 1 year ago

    Screaming Frog is great. Proud to recommend it in my list of the best SEO tools.

    Reply
  • Rafal 1 year ago

    I am really happy about this latest update. Screaming Frog is one of the best SEO audit tools on the market. I’ve been using it for a few years now, and I’m so glad you keep adding new features! Thank you so much for the update!

    Reply
  • Hi to all the SF team,

    We wanted just to express our happiness and satisfaction about this new version of your SEO-Software . As a professional SEO Company working for lot of clients, it helps us a lot running audits and sorting relevant data that help us prioritize our actions.

    Reply
  • Sofia 1 year ago

    Thank you for this update and your explanations.

    Reply
  • Lanny 1 year ago

    Nice, thanks for the updates. Screaming Frig is still one of my favourite tools for OnPage analysis and the issues tab is very helpful.

    Reply
  • asmedio 1 year ago

    great update – as always. Thanks!

    Reply
  • Frank Jackson 1 year ago

    Amazing Update Dan! Thanks a lot to you and your Developers team

    Reply

Leave A Comment.

Back to top