Posted 18 July, 2017 by in Screaming Frog SEO Spider

Screaming Frog SEO Spider Update – Version 8.0

I’m excited to announce the release of Screaming Frog SEO Spider 8.0, codenamed internally as ‘All J’s must go’.

Our team have been busy in development working on some exciting new features, so let’s get straight to it, and talk about what’s new in version 8.0.

1) Updated User Interface

The SEO Spider has ruined many an SEO’s slide deck over the years, with its retro (yet, beautifully functional) user interface, and we felt it was finally deserving of an update. However, please don’t panic – it retains the core usability and data led functionality of what made the interface loved by users. And, it still stays true to its fairly retro styling.

Screaming Frog SEO Spider 8.0

As you can see, it’s a little more modern and has splashes of colour, but now also takes advantage of new technologies in the updated framework, and works with HDPI monitors by default.

2) External Link Metrics Integration

You can now connect to Majestic, Ahrefs and Moz APIs and pull in external link metrics during a crawl. This has been a much-requested feature and is extremely useful for performing a content audit, or quickly bulk checking link metrics against a list of URLs.

When you have connected to an API, link metrics will appear in real time, under the new ‘Link Metrics’ tab and in the ‘Internal’ tab, so they can be combined with all the usual crawl and analytical data.

link metrics integration

We’ve now also introduced an ‘API’ tab into the right-hand window pane, to allow users to keep an eye on progress.

link metrics api tab

You will be required to have an account with the tool providers to pull in data using your own API credentials. Each of the tools offer different functionality and metrics from their APIs, and you’re able to customise what data you want to pull in.

Ahrefs Link Metrics Config

The SEO Spider will calculate the API usage of pulling data based upon your API plan (where possible via the API), and can even combine link counts for HTTP and HTTPS versions of URLs for Majestic and Ahrefs to help you save time.

Moz is the only tool with a free (slower, and limited API), as well as a paid plan, which you can select and allows requests to be super fast.

moz link metrics

So you can pull in Moz metrics such as Page Authority, Domain Authority, or Spam Score and lots more.

moz link metrics

3) Custom Configuration Profiles

You can already adjust and save your configuration to be the default, however, we know users want to be able to switch between multiple set-ups quickly, depending on the crawl type, client or objective. Hence, you are now able to create multiple custom configuration profiles and seamlessly switch between them.

custom configuration profiles

There isn’t a limit to the number of profiles, you can create as many as you like. The custom configuration profiles are saved within your user directory, so you can also copy and share your favourite profiles with colleagues for them to load and use.

4) JavaScript Redirects

The SEO Spider will now discover and report on JavaScript redirects. The SEO Spider was the first commercial crawler with JavaScript rendering, and this functionality has been advanced further to help identify client-side redirects, which is another first.

JavaScript redirects

While not strictly speaking a response code, they can be viewed under the ‘Response Codes’ tab and ‘Redirection (JavaScript)’ filter. Meta Refreshes are now also included within this area and treated in a similar way to regular server-side and client-side redirect reporting.

5) HSTS Support

HTTP Strict Transport Security (HSTS) is a server directive that forces all connections over HTTPS. If any ‘insecure’ links are discovered in a crawl with a Strict-Transport-Security header set, the SEO Spider will show a 307 response with a status message of ‘HSTS Policy’.

HSTS support

The SEO Spider will request the HTTPS version as instructed, but highlight this with a 307 response (inline with browsers, such as Chrome), to help identify when HSTS and insecure links are used (rather than just requesting the secure version, and not highlighting that insecure links actually exist).

The search engines and browsers will only request the HTTPS version, so obviously the 307 response HSTS policy should not be considered as a real temporary redirect and ‘a redirect to fix’. John Mueller discussed this in a Google+ post last year.

6) Hreflang Auditing In XML Sitemaps

The SEO Spider already extracts, crawls and reports on hreflang attributes delivered by HTML link element and HTTP Header, and will now for XML Sitemaps in list mode as well.

XML Sitemap Hreflang auditing

There’s now an extra column under the ‘hreflang’ tab, for ‘Sitemap hreflang’ which allows users to audit for common issues, such as missing confirmation links, incorrect language codes, not using the canonical, and much more.

7) Fetch & Render Screenshots Exporting

You can view the rendered page the SEO Spider crawled in the ‘Rendered Page’ tab at the bottom of the user interface when crawling in JavaScript rendering mode.

JavaScript rendering screenshots

These rendered screenshots are now viewable within the ‘C:\Users\User Name\.ScreamingFrogSEOSpider\screenshots-XXXXXXXXXXXXXXX’ folder, and can be exported via the ‘bulk export > Screenshots’ top level menu, to save navigating, copying and pasting.

bulk export rendered screenshots

The rendered screenshots are stored on a temporary basis to this directory during a crawl, and while it’s still open. They will only be saved if you save the crawl project.

Other Updates

Version 8.0 also includes a number of smaller updates, which include –

  • The ‘Internal’ tab now has new columns for ‘Unique Inlinks’, ‘Unique Outlinks’ and ‘Unique External Outlinks’ numbers. The unique number of ‘inlinks’ was previously only available within the ‘Site Structure’ tab and displays a percentage of the overall number of pages linking to a page.
  • A new ‘Noindex Confirmation Links’ filter is available within the ‘Hreflang’ tab and corresponding export in the ‘Reports > Hreflang > Noindex Confirmation Links’ menu.
  • An ‘Occurences’ column has been added to the Hreflang tab to count the number on each page and identify potential problems.
  • A new ‘Encoded URL’ column has been added to ‘Internal’ and ‘Response Codes’ tab.
  • The ‘Level’ column has been renamed to ‘Crawl Depth’ to avoid confusion & support queries.
  • There’s a new ‘External Links’ export under the ‘Bulk Export’ top level menu, which provides all source pages with external links.
  • The SERP Snippet tool has been updated to refine pixel widths within the SERPs.
  • Java is now bundled with the SEO Spider, so it doesn’t have to be downloaded separately anymore.
  • Added a new preset user-agent for SeznamBot (for a search engine in the Czech Republic). Thanks to Jaroslav for the suggestion.
  • The insecure content report now includes hreflang and rel=“next” and rel=“prev” links.
  • You can highlight multiple rows, right click and open them all in a browser now.
  • List mode now supports Sitemap Index files (alongside usual sitemap .xml files).

We also fixed up some bugs.

  • Fixed a couple of crashes in JavaScript rendering.
  • Fixed parsing of query strings in the canonical HTTP header.
  • Fixed a bug with missing confirmation links of external URLs.
  • Fixed a few crashes in Xpath and in GA integration.
  • Fixed filtering out custom headers in rendering requests, causing some rendering to fail.

If you spot any bugs, or have any issues, then please do get in touch with us via support.

We’re now starting work on version 9.0, which has a couple of very big features we’re already excited about. Thanks to everyone for all their feedback, suggestions and continued support. There’s lots more to come!

Now go and download version 8.0 of the SEO Spider.

I also wanted to say a quick thank you to our friends at Jolt hosting, for working hard to keep the server online yesterday! If you’re experiencing any problems downloading, do let us know via Twitter.

Small Update – Version 8.1 Released 27th July 2017

We have just released a small update to version 8.1 of the SEO Spider. This release is mainly bug fixes and small improvements –

  • Fix a crash when using Forms Based Authentication at the same time, in two instances.
  • Fix a crash selecting a URL in the main window, for certain types of URL string.
  • Menus vanishing on mouse up on Windows, when used on multiple monitors.
  • Trailing space in meta charset causing page to be read with wrong charset.
  • Fix for API tab configuration buttons, which can be unresponsive.
  • Fix for crash showing open/save dialogs when last used directory has been deleted.
  • Fix for a crash when using AHREFs.
  • Debug check box doesn’t stay ticked.
  • Fonts not anti aliased in Ubuntu.
  • ga:socialActivity has been deprecated and removed to match Googles API changes.
  • Fix for a crash switching to Tree View, after loading in a saved project.
  • Sitemap reading doesn’t extract images.
  • Cookies stored against the wrong URL when using Forms Authentication.
  • Pop ups (authentication/memory etc) while minimised in Windows leaves app unresponsive.
  • SERP Snippet fixes.

Small Update – Version 8.2 Released 18th October 2017

  • Allow restoring of Majestic, AHREFs and Moz API data.
  • Sped up remove/re-spider URLs, which were much slower in the update.
  • Restored the ‘copy pretty/ugly url options’ for AJAX URLs.
  • Add ‘Open URL in Browser’, to multi select options in tree view.
  • Ensure file extension is always present when exporting/saving.
  • Show file type drop down on Mac when saving/exporting.
  • Redirect chains report now shows JavaScript redirects!
  • Windows installer now supports hi resolution displays.
  • Insecure content report now includes HTTP-> HTTPS redirects.
  • List mode is now limited to 500 URLs without a licence, like regular Spider mode.
  • Fix a JavaScript crawling crash.
  • Fix bug with duplicate custom extraction names causing incorrect exporting.
  • Fix various bugs in forms based authentication.
  • Fix issue with Windows XP start-up.
  • Fix Ubuntu package warnings.
  • Fix saving/restoring of screenshots.
  • Fix a Majestic API crash.
  • Fix a bug where double clicking an .seospider file on Mac didn’t load the file.
  • Fix an issue with lower window loosing sort, when clicking URLs in the master view.

Small Update – Version 8.3 Released 8th November 2017

  • Fixed issue with JavaScript rendering reporting “Rendering Failed”.
  • Fixed issue preventing Mac version starting on case sensitive file systems.
  • Fixed issue preventing Mac version starting for users with a space in the path to their home directory.
  • Fixed a parse error with Xpath Custom Extraction.
  • Fixed an empty export when right clicking and ‘Export -> URL Info/ InLinks / OutLinks’
  • We stopped reporting empty meta refreshes.

If you spot any other issues, please do get in touch with our support team.