How To Use The Screaming Frog SEO Spider Tool To Audit Backlinks
For those of you that like to keep an eye on your backlinks and check to see if they are still where they should be, it can be a laborious task. Perhaps you have a list of links from directories, articles or sources which need to be checked that they are still in place from time to time. How can you be sure they are where they should be?
Well, it’s easy to use the SEO Spider to help do this! (Please note, the custom search feature outlined below is only available to licenced users). Here is a quick guide to show how this can be done:
- Step 1 – List Your URLs
Get the full list of URLs you want to check in a single txt or CSV file. Make sure you use the full URL (including http://) of the page that should contains the backlink to your site. Save this list as a .txt or .csv.
- Step 2 – Open The SEO Spider!
Go ahead and open up the SEO Spider tool!
- Step 3 – Upload The URL List
On the top toolbar, click ‘mode’ and select ‘list’. Next, click ‘select file’ and browse to where you saved your txt or csv of URLs in Step 1. Click open. The file reader will tell you how many URLs it found in the file you uploaded. If it was “0”, go back and take another look at the list of URLs and check they are all formatted OK.
 |
Check out our user guide for more information on ‘list mode‘.
- Step 4 – Configure The Custom Source Code Filter
Back up on the navigation, click on “configuration” and then “custom”. In the custom filter configuration window you have several options available and I’m sure you’ll figure out what works best for you, however, my preference would be to select “Does Not Contain” in the filter 1 dropdown, then enter your website URL in the text input field.
 |
Please see our user guide for more information about the custom source code search feature.
- Step 5 – Prepare & Crawl
Click the Custom Tab. Select your chosen filter from the Filter drop down on the left. Click Start.
- Step 6 – Review
In the example I set out above, hopefully the output will be blank which means that all the backlink locations do contain my site link, however, if some are listed, such as in my example below, you can can chase up why your link is no longer present.
 |
I hope this is useful. Now it’s your turn, what different ways do you use the SEO spider?
Graeme has been in the digital marketing industry for more than 12 years, having previously founded, grown and sold one of Europe’s largest digital marketing agencies. Graeme believes that ‘search’ is both fascinating and rewarding to work on and similarly rewarding for clients.
It is not really useful when you leave off a huge detail !!
“Note: These options are only available to licensed users.”
Just wasted 10 minutes. THANK YOU!
Hi Matt,
Apologies for being such a big waste of your time.
I have included a note in the blog post to make this clear.
Thanks,
Dan
Very helpful guys, thanks. I’ve been using Scrapebox to do the same thing, but have always questioned it’s accuracy, so having a 2nd tool to run the query just means I’ll hopefully get the most accurate info. Also opens up a lot more possibilities that I hadn’t thought of on what I can be using Screaming Frog for, so thanks for that!
Hey Jeff,
Good to hear it’s helping out! We use this method internally for clients and it saves a ton of time!
Cheers.
Dan
Just wondering if there was a way to view the pages that are linking to the discovered 404 errors? I can see the number of inlinks and outlinks but haven’t figured out how to find the specific source linking to the broken page…
Thanks so much!
Brit
Hey Britney,
We just spoke on Twitter, but thought I’d link to the answer in our FAQs just incase anyone else has the same query –
http://www.screamingfrog.co.uk/seo-spider/faq/#24
Cheers.
Dan
Great post Graeme.
One thing to remember is to check the response codes tab too!
The custom tab will return 200 response codes, which hopefully most will be. However it is well worth checking for other response codes in the other tabs.
Cheers,
Jack
Good shout Jack.
Obviously the SEO Spider will only be able to determine whether the URL being crawled ‘contains’ or ‘does not contain’ the query string ‘domain.com’, if the URL returns a 200 response.
If a ‘no response’, 3XX, 4XX & 5XX are returned, the URL will not appear under the custom tab, because the SEO spider does not know if the URL contains or does not contain. Hence, these should all be checked & either recrawled again or essentially included as a ‘does not contain’.
Thanks for this post Graeme.
I knew Screaming Frog was a fantastic tool for checking your own site URLs but hadn’t appreciated it could handle this requirement too. What a brilliant ‘bonus’! :-)
Graeme, thanks for this useful blog post.
I was wondering if there ae plans to enhance SF to crawl a site or part of a site and get information on Backlinks too, in order to produce a list of urls ordered by No. of referring domains or backlinks… I currently do this with ahrefs and does the job well, but i then find myself bringing that data into SF so it’d be great to everything done with one single tool. Perhaps I am asking for too much :)
cheers
Hi David,
It’s certainly been considered internally by us for a longtime and I agree the functionality would be really useful.
I can’t tell you when it might be available though unfortunately! :-) But it’s on the ‘todo’.
Thanks for the feedback, appreciated.
Dan