Screaming Frog Log File Analyser Update – Version 4.0

Dan Sharp

Posted 25 March, 2020 by in Screaming Frog Log File Analyser

Screaming Frog Log File Analyser Update – Version 4.0

I am pleased to announce the release of the Screaming Frog Log File Analyser 4.0, codenamed ‘Stay At Home’.

If you’re not already familiar with the Log File Analyser tool, it allows you to upload your server log files, verify search engine bots, and get valuable insight into search bot behavior when crawling your website.

Screaming Frog Log File Analyser 4.0

While everyone has more important things on their minds at this time, this small release includes limited, but essential updates that will help improve tracking of evergreen Googlebot and Bingbot and more.

Let’s jump straight to them.

1) Wildcard User-Agent Matching

You’re now able to use wildcard matching when configuring user-agents you wish to import into a project. This makes it far more flexible, particularly when user-agents strings change regularly, such as the new evergreen Googlebot and Bingbot.

You can choose from our pre-defined list of common search engine bots, or configure your own.

Regex configurable user-agents

The default user-agent wildcard matching for Googlebot and Bingbot has also been updated to improve tracking of their ‘evergreen’ version naming.

Googlebot events

In reality, Google are still going to be crawling with their old Chrome 41 user-agent as discussed here. Chrome 41 represents the Googlebot that makes the initial HTTP request to fetch the HTML of a page. The updated Chrome versions, are the Googlebot which renders the page.

2) Remove Parameters

You’re now able to supply a list of parameters to strip from URLs and consolidate when you import log files.

Remove Parameters

This is available in the ‘new’ project configuration and is particularly useful when you have known parameters or issues, and need to consolidate data.

3) New JSON Timestamp Support

In version 3.0, we provided support for log files in JSON format. There isn’t a common standard, so we have utilised customer provided JSON formats, and provided support for as many as possible.

This support has now been extended further to cover some less common JSON timestamp examples we have been provided by users. All you need to do is drag and drop in log files (or folders) as usual, and the Log File Analyser will automatically detect the format and analyse them.

Extended JSON Timestamp support

Thanks to everyone for their examples, and keep them coming!

4) Java 11 Update

While this will make little practical difference to many users, behind the scenes we have updated to Java 11. Our SEO Spider will be following soon.

More To Come

As outlined above, this is just a small update for now. However, there’s lots more to come and please do keep sending in your feature requests with what you’d like to see in the Log File Analyser.

If you’re looking for inspiration for log file analysis, then check out our guide on 22 ways to analyse log files for SEO.

As always, thanks to the SEO community for your support, and please let us know if you experience any issues with version 4.0. Keep safe, well and mostly inside for now!

Small Update – Version 4.1 Released 18th May 2020

We have just released a small update to version 4.1 of the Log File Analyser. This release is mainly bug fixes and small improvements –

  • Update to Java 11.0.7.
  • Add support for cs-user-agent and cs-referrer in W3C log files.
  • Fix issue with User Agent filter using bot verification status dropdown value when non-bot User Agent is selected.
  • Fix issue where modifying an existing user agent leads to ghost user agents.
  • Fix crash when scrolling through table views.
  • Fix crash validating workspace.

Small Update – Version 4.2 Released 19th August 2020

We have just released a small update to version 4.2 of the Log File Analyser. This release is mainly bug fixes and small improvements –

  • Fix issue not being able to run on macOS Big Sur.
  • Fix issue run on multiple monitors with different scaling settings.
  • Fix issue with older macOS versions (10.12 and earlier) not being able to mount the installer DMG.
  • Fix issue not being able to run when Java 8 is installed locally on Windows.
  • Fix issue with loosing data types in some locales when exporting to Excel.

Small Update – Version 4.3 Released 12th January 2021

We have just released a small update to version 4.3 of the Log File Analyser. This release is mainly bug fixes and small improvements –

  • Speed up bot verification for large projects.
  • Add HTTP protocol support.
  • Add Applebot UA to predefined User Agents.
  • Highlighting has been switched to SF green within the app.
  • Fix crash verifying bots.
  • Fix unusable window sizes issue on Linux KDE.
  • Fix crash dragging and dropping URL Data.
  • Fix crashing opening project.
  • Fix crash parsing some variations of Apache logs.

Small Update – Version 4.4 Released 22nd April 2021

We have just released a small update to version 4.4 of the Log File Analyser. This release is mainly bug fixes and small improvements –

  • Fix bug preventing the import of some log files with a forward slash in the file name.
  • Fix crash when updating from 4.2 or below for some projects.
  • Fix crash running bot verification for some projects from 4.2 or below.
  • Fix crash showing some alert dialogs.

Small Update – Version 4.5 Released 11th January 2022

We have just released a small update to version 4.5 of the Log File Analyser. This release includes a security patch and bug fix –

  • Update to log4j 2.17.1 to fix CVE-2021-45105, CVE-2021-45046 and CVE-2021-44228.
  • Fix crash on macOS Monterey.

Dan Sharp is founder & Director of Screaming Frog. He has developed search strategies for a variety of clients from international brands to small and medium-sized businesses and designed and managed the build of the innovative SEO Spider software.


  • Alberto Valle 4 years ago

    Great update Screamingfrog team! I have been using the tool since its first version the tool and always helps me to have some other optimization ideas to improve the ranking of our website.

    • screamingfrog 4 years ago

      Thanks, Alberto! Much appreciated. Log files are still such an untapped source of data.

  • Panos Kontogiannis 4 years ago

    One of the best tools for analyzing and finding answers to why this is so. When you have the information you need, it’s easier to see the puzzle.

  • Gnaneshwar Gaddam 4 years ago

    Thanks for this great update!

  • ucorp 4 years ago

    Nice to know that a new version has been release ! I will check it out

  • Alf Stegmeyer 4 years ago

    Screaming Frog is Amazing. It helps me fix mistakes on my site I didn’t know I made. Last week I spent two days fixing a url problem I created that caused a dozen 404 errors. I fixed 11. I spent a day (almost two) trying to find the last one. The Frog said it was there but I could not find it. I emailed Squarespace and Screaming Frog. Dan @ Screaming Frog found the problem and emailed me back. It was this little leftover invisible link thing. I apparently have it fixed.

    The Frog is a must-have, period, full stop. I would be creating a train wreck without it. The information it provides in one place is simply amazing. Apparently the new version does much more. The new version is probably way over my head but I might buy it on general principle – because what these guys do is a must for anyone who wants to put together a site. Genius.

    My next move is to hire one of the team to help me build a punch list of things I must do.

    Canonicals? I’m lucky if I can spell it no less decide what to do about them. 404’s are a disaster if you don’t know they exist and Google maps your site.

    For me, the Frog is the “WHAT I DON’T KNOW – ABOUT WHAT I DON’T KNOW” tool.

    Just buy it…period…

  • Florian 4 years ago

    Great Job!

  • jacob 4 years ago

    Great update Screamingfrog team!

  • Lars 2 years ago

    Thanks so much for the update. So many overlook the potential of log file analysis even though it can be such an insightful tool in evaluating a website.


Leave A Comment.

Back to top