Screaming Frog Log File Analyser Update – Version 4.0

I am pleased to announce the release of the Screaming Frog Log File Analyser 4.0, codenamed ‘Stay At Home’.

If you’re not already familiar with the Log File Analyser tool, it allows you to upload your server log files, verify search engine bots, and get valuable insight into search bot behavior when crawling your website.

While everyone has more important things on their minds at this time, this small release includes limited, but essential updates that will help improve tracking of evergreen Googlebot and Bingbot and more.

Let’s jump straight to them.

1) Wildcard User-Agent Matching

You’re now able to use wildcard matching when configuring user-agents you wish to import into a project. This makes it far more flexible, particularly when user-agents strings change regularly, such as the new evergreen Googlebot and Bingbot.

You can choose from our pre-defined list of common search engine bots, or configure your own.

The default user-agent wildcard matching for Googlebot and Bingbot has also been updated to improve tracking of their ‘evergreen’ version naming.

In reality, Google are still going to be crawling with their old Chrome 41 user-agent as discussed here. Chrome 41 represents the Googlebot that makes the initial HTTP request to fetch the HTML of a page. The updated Chrome versions, are the Googlebot which renders the page.

2) Remove Parameters

You’re now able to supply a list of parameters to strip from URLs and consolidate when you import log files.

This is available in the ‘new’ project configuration and is particularly useful when you have known parameters or issues, and need to consolidate data.

3) New JSON Timestamp Support

In version 3.0, we provided support for log files in JSON format. There isn’t a common standard, so we have utilised customer provided JSON formats, and provided support for as many as possible.

This support has now been extended further to cover some less common JSON timestamp examples we have been provided by users. All you need to do is drag and drop in log files (or folders) as usual, and the Log File Analyser will automatically detect the format and analyse them.

Thanks to everyone for their examples, and keep them coming!

4) Java 11 Update

While this will make little practical difference to many users, behind the scenes we have updated to Java 11. Our SEO Spider will be following soon.

More To Come

As outlined above, this is just a small update for now. However, there’s lots more to come and please do keep sending in your feature requests with what you’d like to see in the Log File Analyser.

If you’re looking for inspiration for log file analysis, then check out our guide on 22 ways to analyse log files for SEO.

As always, thanks to the SEO community for your support, and please let us know if you experience any issues with version 4.0. Keep safe, well and mostly inside for now!

Small Update – Version 4.1 Released 18th May 2020

We have just released a small update to version 4.1 of the Log File Analyser. This release is mainly bug fixes and small improvements –

• Update to Java 11.0.7.
• Add support for cs-user-agent and cs-referrer in W3C log files.
• Fix issue with User Agent filter using bot verification status dropdown value when non-bot User Agent is selected.
• Fix issue where modifying an existing user agent leads to ghost user agents.
• Fix crash when scrolling through table views.
• Fix crash validating workspace.

Small Update – Version 4.2 Released 19th August 2020

We have just released a small update to version 4.2 of the Log File Analyser. This release is mainly bug fixes and small improvements –

• Fix issue not being able to run on macOS Big Sur.
• Fix issue run on multiple monitors with different scaling settings.
• Fix issue with older macOS versions (10.12 and earlier) not being able to mount the installer DMG.
• Fix issue not being able to run when Java 8 is installed locally on Windows.
• Fix issue with loosing data types in some locales when exporting to Excel.

Small Update – Version 4.3 Released 12th January 2021

We have just released a small update to version 4.3 of the Log File Analyser. This release is mainly bug fixes and small improvements –

• Speed up bot verification for large projects.
• Add HTTP protocol support.
• Add Applebot UA to predefined User Agents.
• Highlighting has been switched to SF green within the app.
• Fix crash verifying bots.
• Fix unusable window sizes issue on Linux KDE.
• Fix crash dragging and dropping URL Data.
• Fix crashing opening project.
• Fix crash parsing some variations of Apache logs.

Small Update – Version 4.4 Released 22nd April 2021

We have just released a small update to version 4.4 of the Log File Analyser. This release is mainly bug fixes and small improvements –

• Fix bug preventing the import of some log files with a forward slash in the file name.
• Fix crash when updating from 4.2 or below for some projects.
• Fix crash running bot verification for some projects from 4.2 or below.
• Fix crash showing some alert dialogs.

Small Update – Version 4.5 Released 11th January 2022

We have just released a small update to version 4.5 of the Log File Analyser. This release includes a security patch and bug fix –

• Update to log4j 2.17.1 to fix CVE-2021-45105, CVE-2021-45046 and CVE-2021-44228.
• Fix crash on macOS Monterey.