How To Find Broken Links Using The SEO Spider
You can use the Screaming Frog SEO Spider for free (and paid) to check for broken links (the http response ‘404 not found error’) on your website.
Below is a very quick and easy tutorial on how to use the tool as a broken link checker. First of all, you’ll need to download the SEO Spider which is free in lite form, for crawling up to 500 URLs.
You can crawl more than 500 URLs with the paid version. The next steps to find broken links are –
1) Crawl The Website
Open up the SEO Spider, type or copy in the website you wish to crawl in the ‘enter url to spider’ box and hit ‘Start’.
2) Click The ‘Response Codes’ tab & ‘Client Error (4XX)’ Filter To View Broken Links
You can wait until the crawl finishes and reaches 100%, or you can just view 404 broken links while crawling by navigating to the ‘Response Codes’ tab and using the filter for ‘Client Error 4XX’.
There are two ways to do this, you can simply click on the ‘tab’ at the top and use the drop down filter –
Alternatively you can use the right-hand window crawl overview pane and just click directly on ‘Client Error (4xx)’ tree view under the ‘Response Codes’ folder. They both show the same results, regardless of which way you navigate.
This crawl overview pane updates while crawling, so you can see there number of client error 4XX links you have at a glance. In the instance above, there are 4 client errors which is 0.58% of the links discovered in the crawl.
3) View The Source Of The Broken Links By Clicking The ‘Inlinks’ Tab
Obviously you’ll want to know the source of the broken links discovered (which URLs on the website link to these broken links), so they can be fixed. To do this, simply click on a URL in the top window pane and then click on the ‘Inlinks’ tab at the bottom to populate the lower window pane.
You can click on the above to view a larger image. As you can see in this example, there is a broken link to the BrightonSEO website (http://www.brightonseo.com/category/videos/), which is linked to from this page – https://www.screamingfrog.co.uk/15-top-quotes-and-takeaways-from-brighton-seo/.
Here’s a closer view of the lower window pane which details the ‘inlinks’ data –
‘From’ is the source where the 404 broken link can be found, while ‘To’ is the broken link. You can also see the anchor text, alt text (if it’s an image which is hyperlinked) and whether the link is followed (true) or nofollow (false).
It looks like the only broken links on our website are external links (sites we link out to), but obviously the SEO Spider will discover any internal broken links if you have any.
4) Use The ‘Bulk Export > Response Codes > Client Error (4XX) Inlinks’ Export
If you’d rather view the data in a spreadsheet you can export both the ‘source’ URLs and ‘broken links’ by using the ‘Bulk Export’, ‘Response Codes’ and ‘Client Error (4XX) Inlinks’ option in the top level menu.
There’s a number of ways you can export data from the Screaming Frog SEO spider, so I recommend reading our user guide on exporting.
If the above does not return all the broken links found in a crawl of the website it is likely that the broken links are referenced in canonical link elements. To find these pages the Canonical Errors report must be used under ‘Reports > Canonical Errors’ in the top level menu of the SEO Spider.
This export will also include redirects (3XX responses), server errors (5XX responses) or any other pages that do not resolve (a 2XX response). Hence, the status column in the export should be filtered to 4XX responses. This can then be used in the same way as the ‘Client Error (4XX) Inlinks’ export where the ‘Source’ column details the page that the broken canonical can be found on and the ‘Destination’ column details the URL of the page giving a 4XX response.
Crawling A List Of URLs For Broken Links
Finally, if you have a list of URLs you’d like to check for broken links instead of crawling a website, then you can simply upload them in list mode.
To switch to ‘list’ mode, simply click on ‘mode > list’ in the top level navigation and you’ll then be able to choose to paste in the URLs or upload via a file.
Hopefully the above guide helps illustrate how to use the SEO Spider tool to check for broken links efficiently.