Posted 24 October, 2013 by in Screaming Frog SEO Spider

How To Audit Redirects In A Site Migration Using The SEO Spider

I’ve been meaning to write about how to use the Screaming Frog SEO Spider to audit redirects in a site migration for a little while. It’s a really important feature, something we use all the time internally and love, and one that I realised perhaps not all are completely aware of.

Obviously when you’re changing URL structure or moving to a new domain, 301 permanent redirects should be used to pass indexing and link signals from the old URLs to the new URLs. Let’s not go too deep into how you set them up, but fairly regularly things can go wrong; redirects can be set-up incorrectly to the wrong targets, have multiple hops, or even error and 404 etc.

Hence, it’s always wise to audit the old URLs, check they redirect correctly & spot any errors that might need to be corrected quickly. This can be a challenge, particularly at scale, but I hope the following process will really help.

1) Upload The Old URLs

Switch the SEO Spider to list mode, select the file with all the old URLs to audit & upload.

SEO Spider List Mode

We always recommend uploading URLs from a crawl pre-migration, but it’s often wise to combine and de-dupe this with other sources, such as landing page URLs from analytics, Google Search Console top pages, data from Majestic, Ahrefs, Moz Link Explorer, scraped URLs from site: queries, log files, sitemaps etc.

2) Tick The ‘Always Follow Redirects’ Box

Navigate to the ‘advanced’ tab inside the ‘spider configuration’ and tick the ‘always follow redirects‘ option.

Always follow redirects

As default ‘list mode’ works at a 0 crawl depth, meaning it just crawls the URLs included in the upload. With this feature ticked, it means it ignores depth & will follow redirects until the final destination (a no response, 2XX, 4XX or 5XX etc).

3) Start The Crawl

Now hit the ‘start’ button, let the SEO Spider crawl the site, reach 100% and come to a stop.

Start the crawl

4) Export The ‘All Redirects’ Report

This can be found under ‘Reports’ in the top level menu.

All Redirects Report

This is where the magic happens.

This report does not just include URLs which have redirect chains, it includes every URL in the original upload & the response in a single export. It then records the start and final URLs and statuses in fixed columns, and maps out the full hops for more information.

Click on the tiny incomprehensible image below to view a larger version which might make more sense.

redirect chains report

Fixed Columns

As the image highlights, the report includes fixed columns for the start URL and final URL, and other vital information. The full list of fixed columns from left to right includes –

  • Chain Type – Whether it’s an HTTP Redirect, JavaScript Redirect or Meta Refresh for example.
  • Number of Redirects – The number of hops in the chain.
  • Redirect Loop – True of False to whether the redirect goes back to a previous URL and loops.
  • Temp Redirect in Chain – True or false to whether there is a temporary redirect in the chain.
  • Address – The start URL uploaded and crawled first.
  • Final Address – The final destination URL.
  • Indexability – Whether the final address URL is Indexable or Non-Indexable at a glance.
  • Indexability Status – The reason why a URL is Non-Indexable. For example, if it’s ‘noindex’.
  • Final Content – The content type of the final address.
  • Final Status Code – The HTTP status code of the final address (this will be a no response, 1XX, 2XX, 4XX, or 5XX).
  • Final Status – The HTTP status of the final address.

    • These columns allow you to quickly diagnose any issues. Then if required, you can also dig further into the full redirect chains.

    Variable Columns

    The report will map out the full details of the redirect chain in variable columns to the right of the fixed columns. So if there are 3 redirects in a chain, it will contain details of the exact content, status code, status and redirect target URL of each hop along the way. These columns are named as follows.

  • Content 1 – The content type of the start address.
  • Status Code 1 – The HTTP status code of the start address.
  • Status 1 – The HTTP status of the start address.
  • Redirect Type 1 – The redirect type (HTTP Redirect, JavaScript Redirect, Meta Refresh etc) of the start address.
  • Redirect URI 1 – The redirect target URL of the start URL.
  • Content 2 – The content type of the second URL.
  • Status Code 2 – The HTTP status code of the second URL.
  • Status 2 – The HTTP status of the second URL.
  • Redirect Type 2 – The redirect type (HTTP Redirect, JavaScript Redirect, Meta Refresh etc) of the second URL.
  • Redirect URI 2 – The redirect target URL of the second URL.
  • etc!

So this report, although intended for ‘redirect chains’, was also built to provide a comprehensive method to audit all redirects in a site migration in a nice, easy, single export so we don’t all go mad.

If you have any queries with the above feature, please do just let us know in the comments. Hopefully this helps ease the pain of site migrations a little.