The Screaming Frog SEO Spider is a small desktop program you can install on your PC or Mac which spiders websites’ links, images, CSS, script and apps from an SEO perspective. It fetches key onsite page elements for SEO, presents them in tabs by type and allows you to filter for common SEO issues, or slice and dice the data how you see fit by exporting into Excel. You can view, analyse and filter the crawl data as it’s gathered and updated continuously in the program’s user interface.
The Screaming Frog SEO Spider allows you to quickly analyse, audit and review a site from an onsite SEO perspective. It’s particulary good for analysing medium to large sites where manually checking every page would be extremely labour intensive (or impossible!) and where you can easily miss a redirect, meta refresh or duplicate page issue.
The spider allows you to export key onsite SEO elements (url, page title, meta description, headings etc) to Excel so it can easily be used as a base to make SEO recommendations from. Our video below provides a demonstration of what the tool can do -
What Information Does The Screaming Frog SEO Spider Tool Report?
A quick summary of some of the data collected -
It is a SEO tool built by real SEOs. Check out what SEOs are saying about the SEO Spider!
By downloading, installing and using the Screaming Frog SEO Spider, you agree to the terms and conditions.
The standard ‘Lite’ version of the tool is completely free, however this version is limited to crawling a maximum of 500 URI and it does not give you full access to the configuration options of the spider or the custom source code feature.
For just £99 per annum you can purchase an individual licence which removes restrictions on the 500 URI maximum crawl and opens up the spider’s configuration options and custom source code search feature . Alternatively hit the ‘buy a licence’ button in the spider to buy a licence after downloading and testing the software.
How Does The Screaming Frog SEO Spider Work?
As default the spider crawls sites like Googlebot does (it obeys allow, disallow directives and wildcard support like Googlebot) but presents its own user-agent ‘Screaming Frog SEO Spider’ which it will obey specific directives for in robots.txt. If there are no directives for the spider, it will crawl your site like Googlebot while still presenting its own UA.
Tips On Using The SEO Spider Tool
Bugs, Feedback & Liability
If you have any problems with the spider please see our support page.
Using the Screaming Frog SEO spider is entirely at your own risk, we will not held responsible or be liable for any costs, damages or actions from its use – that is entirely down to the user. Please read our terms and conditions before use.
Keep updated with future release of the tool by checking for updates in the interface, subscribing to the Screaming Frog Blog RSS, signing up to our e-mail above or following us on Twitter @screamingfrog.