Screaming Frog SEO Spider Update – Version 8.0
Dan Sharp
Posted 18 July, 2017 by Dan Sharp in Screaming Frog SEO Spider
Screaming Frog SEO Spider Update – Version 8.0
I’m excited to announce the release of Screaming Frog SEO Spider 8.0, codenamed internally as ‘All J’s must go’.
Our team have been busy in development working on some exciting new features, so let’s get straight to it, and talk about what’s new in version 8.0.
1) Updated User Interface
The SEO Spider has ruined many an SEO’s slide deck over the years, with its retro (yet, beautifully functional) user interface, and we felt it was finally deserving of an update. However, please don’t panic – it retains the core usability and data led functionality of what made the interface loved by users. And, it still stays true to its fairly retro styling.
As you can see, it’s a little more modern and has splashes of colour, but now also takes advantage of new technologies in the updated framework, and works with HDPI monitors by default.
2) External Link Metrics Integration
You can now connect to Majestic, Ahrefs and Moz APIs and pull in external link metrics during a crawl. This has been a much-requested feature and is extremely useful for performing a content audit, or quickly bulk checking link metrics against a list of URLs.
When you have connected to an API, link metrics will appear in real time, under the new ‘Link Metrics’ tab and in the ‘Internal’ tab, so they can be combined with all the usual crawl and analytical data.
We’ve now also introduced an ‘API’ tab into the right-hand window pane, to allow users to keep an eye on progress.
You will be required to have an account with the tool providers to pull in data using your own API credentials. Each of the tools offer different functionality and metrics from their APIs, and you’re able to customise what data you want to pull in.
The SEO Spider will calculate the API usage of pulling data based upon your API plan (where possible via the API), and can even combine link counts for HTTP and HTTPS versions of URLs for Majestic and Ahrefs to help you save time.
Moz is the only tool with a free (slower, and limited API), as well as a paid plan, which you can select and allows requests to be super fast.
So you can pull in Moz metrics such as Page Authority, Domain Authority, or Spam Score and lots more.
3) Custom Configuration Profiles
You can already adjust and save your configuration to be the default, however, we know users want to be able to switch between multiple set-ups quickly, depending on the crawl type, client or objective. Hence, you are now able to create multiple custom configuration profiles and seamlessly switch between them.
There isn’t a limit to the number of profiles, you can create as many as you like. The custom configuration profiles are saved within your user directory, so you can also copy and share your favourite profiles with colleagues for them to load and use.
4) JavaScript Redirects
The SEO Spider will now discover and report on JavaScript redirects. The SEO Spider was the first commercial crawler with JavaScript rendering, and this functionality has been advanced further to help identify client-side redirects, which is another first.
While not strictly speaking a response code, they can be viewed under the ‘Response Codes’ tab and ‘Redirection (JavaScript)’ filter. Meta Refreshes are now also included within this area and treated in a similar way to regular server-side and client-side redirect reporting.
5) HSTS Support
HTTP Strict Transport Security (HSTS) is a server directive that forces all connections over HTTPS. If any ‘insecure’ links are discovered in a crawl with a Strict-Transport-Security header set, the SEO Spider will show a 307 response with a status message of ‘HSTS Policy’.
The SEO Spider will request the HTTPS version as instructed, but highlight this with a 307 response (inline with browsers, such as Chrome), to help identify when HSTS and insecure links are used (rather than just requesting the secure version, and not highlighting that insecure links actually exist).
The search engines and browsers will only request the HTTPS version, so obviously the 307 response HSTS policy should not be considered as a real temporary redirect and ‘a redirect to fix’. John Mueller discussed this in a Google+ post last year.
6) Hreflang Auditing In XML Sitemaps
The SEO Spider already extracts, crawls and reports on hreflang attributes delivered by HTML link element and HTTP Header, and will now for XML Sitemaps in list mode as well.
There’s now an extra column under the ‘hreflang’ tab, for ‘Sitemap hreflang’ which allows users to audit for common issues, such as missing confirmation links, incorrect language codes, not using the canonical, and much more.
7) Fetch & Render Screenshots Exporting
You can view the rendered page the SEO Spider crawled in the ‘Rendered Page’ tab at the bottom of the user interface when crawling in JavaScript rendering mode.
These rendered screenshots are now viewable within the ‘C:\Users\User Name\.ScreamingFrogSEOSpider\screenshots-XXXXXXXXXXXXXXX’ folder, and can be exported via the ‘bulk export > Screenshots’ top level menu, to save navigating, copying and pasting.
The rendered screenshots are stored on a temporary basis to this directory during a crawl, and while it’s still open. They will only be saved if you save the crawl project.
Other Updates
Version 8.0 also includes a number of smaller updates, which include –
- The ‘Internal’ tab now has new columns for ‘Unique Inlinks’, ‘Unique Outlinks’ and ‘Unique External Outlinks’ numbers. The unique number of ‘inlinks’ was previously only available within the ‘Site Structure’ tab and displays a percentage of the overall number of pages linking to a page.
- A new ‘Noindex Confirmation Links’ filter is available within the ‘Hreflang’ tab and corresponding export in the ‘Reports > Hreflang > Noindex Confirmation Links’ menu.
- An ‘Occurences’ column has been added to the Hreflang tab to count the number on each page and identify potential problems.
- A new ‘Encoded URL’ column has been added to ‘Internal’ and ‘Response Codes’ tab.
- The ‘Level’ column has been renamed to ‘Crawl Depth’ to avoid confusion & support queries.
- There’s a new ‘External Links’ export under the ‘Bulk Export’ top level menu, which provides all source pages with external links.
- The SERP Snippet tool has been updated to refine pixel widths within the SERPs.
- Java is now bundled with the SEO Spider, so it doesn’t have to be downloaded separately anymore.
- Added a new preset user-agent for SeznamBot (for a search engine in the Czech Republic). Thanks to Jaroslav for the suggestion.
- The insecure content report now includes hreflang and rel=“next” and rel=“prev” links.
- You can highlight multiple rows, right click and open them all in a browser now.
- List mode now supports Sitemap Index files (alongside usual sitemap .xml files).
We also fixed up some bugs.
- Fixed a couple of crashes in JavaScript rendering.
- Fixed parsing of query strings in the canonical HTTP header.
- Fixed a bug with missing confirmation links of external URLs.
- Fixed a few crashes in Xpath and in GA integration.
- Fixed filtering out custom headers in rendering requests, causing some rendering to fail.
If you spot any bugs, or have any issues, then please do get in touch with us via support.
We’re now starting work on version 9.0, which has a couple of very big features we’re already excited about. Thanks to everyone for all their feedback, suggestions and continued support. There’s lots more to come!
Now go and download version 8.0 of the SEO Spider.
I also wanted to say a quick thank you to our friends at Jolt hosting, for working hard to keep the server online yesterday! If you’re experiencing any problems downloading, do let us know via Twitter.
Small Update – Version 8.1 Released 27th July 2017
We have just released a small update to version 8.1 of the SEO Spider. This release is mainly bug fixes and small improvements –
- Fix a crash when using Forms Based Authentication at the same time, in two instances.
- Fix a crash selecting a URL in the main window, for certain types of URL string.
- Menus vanishing on mouse up on Windows, when used on multiple monitors.
- Trailing space in meta charset causing page to be read with wrong charset.
- Fix for API tab configuration buttons, which can be unresponsive.
- Fix for crash showing open/save dialogs when last used directory has been deleted.
- Fix for a crash when using AHREFs.
- Debug check box doesn’t stay ticked.
- Fonts not anti aliased in Ubuntu.
- ga:socialActivity has been deprecated and removed to match Googles API changes.
- Fix for a crash switching to Tree View, after loading in a saved project.
- Sitemap reading doesn’t extract images.
- Cookies stored against the wrong URL when using Forms Authentication.
- Pop ups (authentication/memory etc) while minimised in Windows leaves app unresponsive.
- SERP Snippet fixes.
Small Update – Version 8.2 Released 18th October 2017
- Allow restoring of Majestic, AHREFs and Moz API data.
- Sped up remove/re-spider URLs, which were much slower in the update.
- Restored the ‘copy pretty/ugly url options’ for AJAX URLs.
- Add ‘Open URL in Browser’, to multi select options in tree view.
- Ensure file extension is always present when exporting/saving.
- Show file type drop down on Mac when saving/exporting.
- Redirect chains report now shows JavaScript redirects!
- Windows installer now supports hi resolution displays.
- Insecure content report now includes HTTP-> HTTPS redirects.
- List mode is now limited to 500 URLs without a licence, like regular Spider mode.
- Fix a JavaScript crawling crash.
- Fix bug with duplicate custom extraction names causing incorrect exporting.
- Fix various bugs in forms based authentication.
- Fix issue with Windows XP start-up.
- Fix Ubuntu package warnings.
- Fix saving/restoring of screenshots.
- Fix a Majestic API crash.
- Fix a bug where double clicking an .seospider file on Mac didn’t load the file.
- Fix an issue with lower window loosing sort, when clicking URLs in the master view.
Small Update – Version 8.3 Released 8th November 2017
- Fixed issue with JavaScript rendering reporting “Rendering Failed”.
- Fixed issue preventing Mac version starting on case sensitive file systems.
- Fixed issue preventing Mac version starting for users with a space in the path to their home directory.
- Fixed a parse error with Xpath Custom Extraction.
- Fixed an empty export when right clicking and ‘Export -> URL Info/ InLinks / OutLinks’
- We stopped reporting empty meta refreshes.
If you spot any other issues, please do get in touch with our support team.
Oh. Yes. Can’t wait to give this one a whirl :). New GUI!!
This is so great!
Thank you for those all new features. Great job guys.
Please add more themes for layout in the future ;)
Thanks, Lukasz!
We like that idea, and may have more here in the future.
It looks amazing! The new design is quite attractive and I love the integration with Google services as well as Ahrefs.
Thanks, Alberto!
Waow!! Looks really awesome. Can’t wait to checkout the new features especially the external metrics integration.
Thanks for the update.
Stecy
Very cool to hear, I do enjoy watching the various API progress bars during a crawl :-)
Woah woah! Integration of link metrics into one is a killer. The auditing gets way easier now on. The UI looks better, Thanks Dan, and the team for the update.
Cheers, Suresh!
Super Dan
Nice! This is great!
Like the new look. One of the best tools on the market. I use it every day.
This is a great update. Other than Excel, this is the best seo tool going and I’ve loved it for years and years.
Great! Thanks for an awesome product!
I got a 4k monitor last week. Good timing with this update of Screaming Frog :-)
This has been the best seo tool investment we ever made! Thank you for continuing to make it better!
new look was great, but when I tried to run it on my pc it became so slow
Hey Arfel,
Thanks for your comment. This might be because you need to increase memory allocation – https://www.screamingfrog.co.uk/seo-spider/faq/#why-am-i-experiencing-slow-down-or-hanging-upon-exports-saving-crawls
However, pop through the details to us and your log files if that’s not the case. Support is here – https://www.screamingfrog.co.uk/seo-spider/support/
Cheers
Dan
Hi Screaming Frog Team,
Thanks for the new release, looks pretty sweet – however, when exporting crawls to .xlsx all special letters such as äöü are turned into “�” Please fix this bug as soon as you can – it has quite the impact on our work.
Thanks in advance,
Florian
Hey Florian,
Thank you! Are you able to send through an example URL to us via support@screamingfrog.co.uk?
Thanks,
Dan
Hey,
Thanks for the update to Screaming Frog, loving the new interface and features.
I have an issue with loading a crawl. Its taking 30 minutes to load in a 30k URL crawl, my RAM allocation is 1GB.
Any idea if this is normal? Do I need to allocate more RAM?
Thanks.
Stan.
Hey Stan,
No worries at all, hope you find them useful. Yes, that just sounds like memory allocation. Allocate as much as you can, and it shouldn’t be an issue.
Thanks,
Dan
Thats worked a treat, increased to 2GB and its loaded the crawl within 2 minutes.
A bug that I have also picked up (could just be me) is that when moving the application from say my laptop display to my LG 27″ display (got my laptop with a monitor as second screen), the menu items flash straight off when you click them?
So if i click Configuration for example, the drop down menu appears but then disappears straight away.
Thanks Dan.
Hi Stan,
Good to hear! Thanks for mentioning the bug, which is definitely a bug and known now. We’ve had a few users report on this one.
It’s actually a bug in Java, when using Windows 10 high laptop and a 2nd monitor. We’ve just patched it up, and it will be in 8.1.
For now, if you just use the menus on your laptop rather than second screen, they should work! The update will be fairly soon anyway.
Cheers.
Dan
Good job! Thanks for an awesome product!
Often the spider mode is suspended on OSx. The program crawls only the home page and ends.
Hi Marcin,
I am not sure I know what you mean, so if you could provide more detail to support that would be useful – https://www.screamingfrog.co.uk/seo-spider/support/
If the SEO Spider only crawls the homepage and then ends, that’s usually due to the site not responding, being blocked by robots.txt, having an internal nofollow, being JavaScript etc. More here on common reasons –
https://www.screamingfrog.co.uk/seo-spider/faq/#why-wont-the-seo-spider-crawl-my-website
Thanks,
Dan
Yeah. I’ll send more details soon.
You guys are awesome!!!
External link metrics will be really useful when doing detailed analysis.
Already excited for the new version!
Having the same menu issues that you mentioned are being patched – found that if I click-hold and move my mouse to the title bar the menu will stay open. Looking forward to the patch.
Another issue I’m having is when trying to use the Javascript rendering so I can generate a page preview. When that method is selected I can never crawl past the homepage, but if I use text or Old Ajax the sites crawl just fine. I tried playing with the crawl limits to see if there was an effect, but there wasn’t.
Hi Kevin,
If you email us via support (https://www.screamingfrog.co.uk/seo-spider/support/) you can use the beta, with the menu patch.
With regards to JavaScript rendering – We’d be interested in seeing your example, and your full log file. Again, if you can ping through via support please!
Cheers.
Dan
Love the new design! thanks for not ruining the functionality. I am having issues with AHREFS working, anyone else having similar problems? still works fine in URLprofiler so I know the API key isn’t the issue?
Hey Shaheen,
Good to hear we didn’t ruin the functionality!
No issues reported with Ahrefs, but the API key will be different to other tools. Every tool uses their OpenApps API, you have to authorise it and use a specific key they give you.
More here – https://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/#ahrefs
Cheers.
Dan
sounds like I’m the only one then?
You are the only report – Did you read the guide I just linked? :-)
Dan
“You can now connect to Majestic, Ahrefs and Moz APIs” :D Great option! New design is amazing! Big plus for updates…
Cheers, Pawel!
New design is quite attractive. Great job. It would be great if yiu tell me about how much memory is required.
Thanks, Linda! You need at least 1GB of RAM free, but ideally you’d use a machine with 8GB of RAM. You should be able to crawl a few hundred thousand URLs then.
Nice update. Waiting in 9.0 upgrading hardcore features: better parsing, extraction. Increasing speed, memory using optimization :)
Cheers, Pablo. Some of those will be addressed in the next release.
Excellent work!
This is a killer update….have to check all the new features. Now I am leaning towards the paid version. Are you guys using paid or free versions?
Really very informative post. Well done guys. Good effort.
You guys really do a brilliant job! :)
Just one idea: Get also the API of a keyword scraping tool integrated (e.g. keywordtool.io) and pull the data of the keywords of h1/ title (thats a bit tricky to identify the most important automatically) pulled into your reports – that would be a massiv benefit and afterwards you could analyse: What second phrase/ other more relevant keyword might be missing/ cannibalization of topics…
Looking forward to test the feature!
Max
Hi Max,
Thanks for the suggestion! I can see why that would be useful sometimes.
Cheers,
Dan
Excelent! I love the new interface and features. Thanks!
Great changes!
Interface was “old school”, but most of users used to love it like it was. JS reditect + Fetch&Render looks promising as now crawling and auditing different JS frameworks is a issue and even Google can’t make it correct.
Could You share some sneak preview of incoming “big features”?
Hi Pawel,
Agreed, I think we all had a lot of love for the old UI. But we did need to move on, Swing just doesn’t play nice with new high def monitors.
With regards to big features – We don’t usually share in too much detail, but we are working on improved scale.
Thanks,
Dan
Having two rows for the tabs at the top was pretty convenient vs. the single row now. Especially since I use the Internal and Custom tabs a lot which means repeatedly going between the beginning and end of the row.
Hi Marios,
We noted this change in beta testing, but wanted to see what the general reaction was.
It sounds like you prefer the multiple row scaling, to the dropdown and select of a tab.
Thanks for the feedback!
Dan
A feature I would like is being able to faster select and copy data to the clipboard.
I often select data from 2 columns and would like to select all rows. The way to do this now is to select de data with the mouse and drag down until you get to the bottom.
The problem is, that it scrolls really slow (even slower in this new version), so if you have 500 rows, it takes a while to select all.
I know that I would use the export, but I would like to be able just to copy it.
A solution could be making something like Excel, where you can select the data and with your keyboard select more, by holding down shift, and if you then hold cmd/ctrl you’ll get to the end.
I hope it makes sense and maybe something you would consider. It would make me love the tool even more =)
Hi Jakob,
Totally with you on this one. A couple of tips on the way I get around this –
If you highlight a cell or two (which can be in multiple columns), then click shift and use the scroll bar and click the last cell, it will highlight all cells in between :-)
So you can highlight 500 or 5k rows super quick this way. Also, holding shift and the down arrows is slightly quicker than the mouse scroll (not scroll bar).
Another option I like, is control + A (highlight all), then control + C (copy) and control + V (paste) shortcuts (but sounds like you know all these!).
Cheers.
Dan
Hi Dan.
Thank you for your response.
The first solution with selection the first cells and scroll down works perfectly, thanks for the tip =)
The second with the down arrow doesn’t work if you wish to select more than one collumn.
And the select all I’m aware of =) I just wished to be able just to select the columns I need, so I didn’t have to paste it in Excel to copy the right once =)
But as mentioned, the first solution is going to help me a lot – thank you very much.
Bit late to the party, but this URL still comes up for queries for shortcuts in Screaming Frog, so this might be helpful to someone one day:
On Windows, you can also use SHIFT + END to select all lines of first column (or all lines after the one you selected from that list). If you wanted to select two columns next to each other, for instance, works the same way, just make sure you select them in the first line of a set of URLs.
That’s an awesome tip! Thanks, Gianna!
Thanks for sharing your information with us. Great changes! I hope it will be better for us for analyze to the site..
Been using Screamingfrog it for a while, glad it continues to be updated and growing. Much appreciation!
Hi there,
I love the new interface and new features. However, the one feature which I found useful the most isn’t working ‘Authentication’. Every time I add the dev URL in, then I get 302 temporary redirect.
Many thanks,
Veronika
Hey Veronika,
Good to hear! With regards to authentication – Are you able to pop through the specific example to us via support please?
https://www.screamingfrog.co.uk/seo-spider/support/
Cheers!
Dan
Hey Dan,
I have raised a support ticket. This feature is really important to us, as we literally use it on a daily basis and this was one of the reasons, why we have selected Screaming Frog within out tools portfolio.
Many thanks,
Veronika
While the new features are nice, especially HiDPI support, I don’t like the UI, there was nothing wrong with the existing one, except for the limitations of Swing.
* The UI controls no longer look or act native, especially with the green everywhere. While I understand it’s your brand’s color, I detest green, and there’s no obvious config setting to change it.
* The font rendering quality took a huge nose dive. Some fonts are now too thin, other are over-bolded and the kerning is wrong everywhere.
* The wrong font is being used for the UI. Lucidia is the wrong font for Windows UI’s.
* There are bugs with the menu positioning when using the program in a maximized window.
Everything above was too much to handle (especially the terrible font rendering, it was murder on my eyes), I’ve rolled back to 7.x.
Please don’t follow Adobe’s lead where they screw with the interface every year to justify the high cost of of Creative Cloud.
Thanks
Hi Coding Man,
Sorry to hear you don’t like the new UI! We tried very hard to find a balance to modernising and improving the UI, but keeping the core of what was great about the older GUI.
It’s the only time we have updated the UI in 7 years, so I can’t see us following Adobes lead :-)
In more detail – Unfortunately, the limitations of Swing are never going to be addressed by Oracle. JavaFX, which is how the new UI is built, is where all their UI development efforts are being spent. It was a tough decision for us to choose the right time to move. Swing is very robust, having been out for nearly 2 decades, but JavaFX is the future, but of course will come with its own set of teething issues.
Issues such as the menu program you describe have been reported (http://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8184968) to Oracle, and hopefully will be fully resolved soon.
If you could mail through some examples of the how the controls are not acting natively as well as some screenshots of the font issues you’re experiencing, the dev team to take a closer look at these for you.
Cheers.
Dan
The latest version is amazing. The integration of Majestic is being words.
Thank you screaming Frog!
Hi, this new version looks great. Can’t wait to use it and the MOZ integration. That will save me lots of work in the future. Just a quick question:
Is changing memory allocation still only possible in the .ini file or can we do that from the software itself now? That would be great :)
We will test with pleasure :)
Amazing update !!! Good job.
Been using your free tool for years. Just about to jump in with the paid version now. Stoked to make use of the new features.
Hi there. Where is the absolute neccessary “Windows Look and Feel” feature gone? Please help! Cheers, bin
Screaming Frog is aweseome, as always. Number of perks comparing to any free online tool is absolutely great. Also, I have you say that your tool is pretty fast. Just do not refresh that design please, maybe it seems oldschool but it is still legible and friendly.
Very successful update!
Great software
I love your software! Thank you for continuing to improve it. Very excited to check out the new features. Thank you Screaming Frog!
OK i gotta say i’m very impressed with this new update, didn’t even know till i just check the blog, awesome as always, this app is so comprehensive, now with this new update… all i have to say is WOW!
SEO Spider is one amazing tool. Great integration set up!!
That’s awesome. This is very useful. Thanks.
Glad to see the Windows XP issue has finally been resolved, been looking forward to trying out the update since it rolled out.
Hi guys, I’m a bit late to the screaming frog party, but absolutely love it so far. Keen to learn more from the pro’s – do you guys have any online courses?
Wow this new design is very attractive looking to give SEO spider 8.0 a try.
Screaming Frog is an excellent tool for website SEO auditing. love the new external link integration, A real time saver.
What is meant with that?
“We stopped reporting empty meta refreshes.”
Hi Andre,
We were previously reporting meta refreshes that were empty (the content was empty).
If it has no content / value, it’s essentially ignored, so we stopped reporting on it.
Thanks,
Dan
Everbody told me, it’s a good spider tool for SEO auditing. And I tried It.
Woahh…. it’s true.
Nice update. Many useful features here I see. I started looking for some good implements to improve my SEO and Screaming Frog looks really well.
Is v8.3 compatible with Mac OS High Sierra 10.13.3?
Waiting to upgrade to Apple’s latest and greatest,
but I sure don’t want to lose the “Noisy Amphibian” functionality.
Good job! I am asking for more motifs. I love this SEO tool! ;)
SEO Spider is a very useful and amazing tool. Thanks a lot guys!
Very useful stuff. We use SCREAMINGFROG in our Agency!
Hi , i can’t found how to do analyze de logs by screaming frogs , when i watch your video on your youtube i think itsn’ same version, by the way i use Screaming Frog seo spider 9.4, can you help me plz?!
Hi Yassine,
To analyse log files, you need the Log File Analyser – https://www.screamingfrog.co.uk/log-file-analyser/
It’s a different piece of software, with a different licence.
Cheers.
Dan
I was waiting for upgrade few months, great news!
“The ‘Level’ column has been renamed to ‘Crawl Depth’ to avoid confusion & support queries.”
–> Was about to get in touch :-). Always appreciate an excellent documentation when it comes to coding / development.
Thanks a bunch for the frequent work on your site making Screamingfrog even better as it already is :-)
Greetings from Munich,
Peter
Thank you, great news!
I have just been introduced to Screaming Frog – what a great tool – have found it so useful, thanks for creating it guys!