I’ve been meaning to write about how to use the Screaming Frog SEO Spider to audit redirects in a site migration for a little while. It’s a really important feature, something we use all the time internally and love, and one that I realised perhaps not all are completely aware of.

Obviously when you’re changing URL structure or moving to a new domain, 301 permanent redirects should be used to pass indexing and link signals from the old URLs to the new URLs. Let’s not go too deep into how you set them up, but fairly regularly things can go wrong; redirects can be set-up incorrectly to the wrong targets, have multiple hops, or even error and 404 etc.

Hence, it’s always wise to audit the old URLs, check they redirect correctly & spot any errors that might need to be corrected quickly. This can be a challenge, particularly at scale, but I hope the following process will really help.

1) Upload The Old URLs

Switch the SEO Spider to list mode, select the file with all the old URLs to audit & upload.

seo spider list mode

We always recommend uploading URLs from a crawl pre-migration, but it’s often wise to combine and de-dupe this with other sources, such as landing page URLs from analytics, Google WMT top pages, data from Majestic SEO / OSE, scraped URLs from site: queries, log files, sitemaps etc.

2) Tick The ‘Always Follow Redirects’ Box

Navigate to the ‘advanced’ tab inside the ‘spider configuration’ and tick the ‘always follow redirects‘ option.

always-follow-redirects

As default ‘list mode’ works at a 0 crawl depth, meaning it just crawls the URLs included in the upload. With this feature ticked, it means it ignores depth & will follow redirects until the final destination (a no response, 2XX, 4XX or 5XX etc).

3) Start The Crawl

Now hit the ‘start’ button, let the SEO Spider crawl the site, reach 100% and come to a stop.

4) Click On ‘Reports’ & ‘Redirect Chains’

This is where the magic happens.

redirect chains report

This report does not just include URLs which have redirect chains, it includes every URL in the original upload & the response in a single export. Click on the tiny incomprehensible image below to view a larger version which might make more sense (yes, I set-up some silly redirects to show how it works!) -

redirect chains report export

As the image highlights, if a URL has multiple redirects in a chain, this export will map out each hop along the way, redirect 1, to redirect 2, to redirect 3 etc and all the responses along the way until the final target. It will also show how many hops there are (remember 5+ Google might give up and treat as a 404) & identify any pesky redirect loops.

So this report, although intended for ‘redirect chains’, was also built to provide a comprehensive method to audit all redirects in a site migration in a nice, easy, single export so we don’t all go mad.

If you have any queries with the above feature, please do just let us know in the comments. Hopefully this helps ease the pain of site migrations a little.

screamingfrog (31 Posts)

Dan Sharp is founder & Director of Screaming Frog. He has developed search strategies for a variety of clients from international brands to small and medium sized businesses and designed and managed the build of the innovative SEO Spider software.