SEO Log File Analyser

Analyse search engine bot crawling with the industry leading Log File Analyser for Windows, macOS and Linux.

Log File Analyser

The Screaming Frog SEO Log File Analyser allows you to upload your log files, verify search engine bots, identify crawled URLs and analyse search bot behaviour for invaluable SEO insight.

Download and analyse 1k log events for free, or buy a Log File Analyser licence for 129 Per Year to remove the limit.

Available On

windows
mac
linux

What can you do with the SEO Log File Analyser?

The Log File Analyser is light, but extremely powerful – able to process, store and analyse millions of lines of log file event data in a smart database. It gathers key log file data to allow SEOs to make informed decisions. Some of the common uses include –

Discover Crawl Frequency

Get insight to which search bots crawl most frequently, how many URLs are crawled each day and the total number of bot events.

Audit Redirects

Find temporary and permanent redirects encountered by search bots, that might be different to those in a browser or simulated crawl.

Improve Crawl Budget

Analyse your most and least crawled URLs & directories of the site, to identify waste and improve crawl efficiency.

Identify Large & Slow Pages

Review the average bytes downloaded & time taken to identify large pages or performance issues.

Find Uncrawled & Orphan Pages

Import a list of URLs and match against log file data, to identify orphan or unknown pages or URLs which Googlebot hasn't crawled.

Combine & Compare Any Data

Import and match any data with a 'URLs' column against log file data. So import crawls, directives, or external link data for advanced analysis.

Features

  • Auto Verify Search Engine Bots
  • Find Broken Links, Errors & Redirects
  • View Most & Least Crawled URLs & Sections
  • View Bot Crawl Frequency
  • View Bot IP Crawl Activity
  • Identify Large & Slow URLs
  • Find Uncrawled & Orphan URLs
  • Combine & Compare Crawl Data
  • Log Event Limit
  • Project Limit
  • Free Technical Support

Price per licence

Licences last 1 year. After that you will be required to renew your licence.

Free Version

  • Auto Verify Search Engine Bots
  • Find Broken Links, Errors & Redirects
  • View Most & Least Crawled URLs & Sections
  • View Bot Crawl Frequency
  • View Bot IP Crawl Activity
  • Identify Large & Slow URLs
  • Find Uncrawled & Orphan URLs
  • Combine & Compare Crawl Data
  • Log Event Limit - 1,000 Lines
  • Project Limit - 1
  • Free Technical Support

Paid Version

  • Auto Verify Search Engine Bots
  • Find Broken Links, Errors & Redirects
  • View Most & Least Crawled URLs & Sections
  • View Bot Crawl Frequency
  • View Bot IP Crawl Activity
  • Identify Large & Slow URLs
  • Find Uncrawled & Orphan URLs
  • Combine & Compare Crawl Data
  • Log Event Limit - Unlimited*
  • Project Limit - Unlimited
  • Free Technical Support

129 Per Year

Purchase licence

* The maximum number of log events you can analyse is dependent on hard drive capacity. Please see our FAQ.

The Log File Analyser Tool Reports On...

The Screaming Frog Log File Analyser is an SEO auditing tool, built by real SEOs with thousands of users worldwide. A quick summary of some of the data collected and analysed includes -

  • Crawled URLs – View and analyse exactly which URLs have been crawled by search bots, such as Googlebot and Bingbot.
  • Crawl Frequency – Analyse the most & least frequently crawled URLs by search bot user-agents.
  • Full Event Data – Access full log file event data for every URL discovered in the logs by timestamp.
  • Errors – Identify client side errors, such as broken links and server errors (4XX, 5XX response codes).
  • Redirects – View permanent & temporary redirects (302, or 301 responses).
  • Inconsistent Response Codes – Quickly view URLs with inconsistent response codes over a period of time.
  • Time Of Last Response – View exactly when a search bot last crawled a URL (and the first, as well as every other event!).
  • Average Bytes – Analyse the average bytes of every crawled URL direct from log file event data.
  • Average Response Time (ms) – Discover the average response time of every URL.
  • Referers – View the number of referer events for every URL discovered.
  • Directories – Analyse the most and least frequently crawled directories and sections of the site.
  • Uncrawled URLs – Import a list of URLs & discover URLs which have not been crawled.
  • Orphan URLs – Import a list of URLs & discover which are in log data, but not known by you.
  • Analyse Bots Over Time – Upload multiple log files over at once or over time, to analyse and measure bot activity.
  • Compare Any Data – Upload any data with a ‘URLs’ header to automatically match against log file data and analyse.
  • Verify Search Bots – Automatically verify search bots such as Googlebot, and view IPs spoofing requests.

About The Tool

The Screaming Frog Log File Analyser allows you to quickly upload a log file and analyse it from an SEO perspective. Log file data is extremely valuable as it shows exactly what has happened when a search engine bot visits your website. However, log files can be extremely large in size and difficult to analyse, without programming experience or the right tool. The Log File Analyser is specifically designed for SEOs to make this process less painful.

The Log File Analyser supports Apache and W3C Extended Log File Format which covers Apache, IIS and NGINX servers. This also covers Amazon Elastic Load Balancing custom log file format.

Simply drag and drop your log file, or log files directly onto the user interface and the tool will automatically process the data to allow you to analyse and filter for common SEO issues in the program’s user interface, or export to CSV and Excel.

 

Analyse 1k Log Events For Free

The free version of the Log File Analyser tool is free to download and use. However, this version is restricted to analysing 1k log events and it does not allow you to save more than one project at a time.

For just £99 per year you can purchase a licence, which removes the 1k log events limit and allows you to save multiple projects.

Alternatively hit the ‘buy a licence’ button in the Log File Analyser to buy a licence after downloading and trialling the software.

FAQ & User Guide

For guidance and tips on using the Screaming Frog SEO Log File Analyser –

  • The Log File Analyser supports Apache, W3C Extended and Amazon Elastic Load Balancing Log File formats.
  • You can test to see whether your log format is supported by trialing the free version of the tool. If you have any issues, then send through your Log File Analyser logs (‘Help > Debug > Save Logs’) to our support and we can help.
  • Please read our user guide, regularly updated FAQ and guide on ways to perform log file analysis for inspiration.
  • Please also watch the demo video embedded above. We plan on expanding our video tutorials over time.

 

Updates

Keep updated with future releases of the by subscribing to the our RSS feed or following us on Twitter @screamingfrog.

 

Support & Feedback

If you have any technical problems, feedback or feature requests for the Log File Analyser, then please just contact us via our support. We plan to regularly update the Log File Analyser and currently have lots of new features in development!

Join the mailing list for updates, tips & giveaways

Back to top