SEO Spider Tutorials

Getting Started Guide

Our beginners guide to getting started using the SEO Spider. Including initial set-up, crawling, and viewing reports and issues discovered.

How To Find Broken Links Using The SEO Spider

Learn how to crawl your website and find broken links (404 errors), view which pages link to them, the link text, and export in bulk.

Site Architecture & Crawl Visualisations Guide

Visualise site architecture and internal linking to help communicate ideas and reveal underlying patterns that are harder to uncover in data, and spreadsheets.

How To Compare Crawls

Compare crawls to see how data, issues and opportunities have changed over time to track progress and monitor site health.

How To Crawl JavaScript Websites

Render web pages using the integrated Chromium WRS to crawl dynamic, JavaScript rich websites and frameworks, such as Angular, React and Vue.js.

How To Crawl Large Websites

Crawl large websites by switching to database storage mode, increasing memory and by configuring the crawl to extract the data you need.

How To Crawl A Staging Website

Find out how to crawl a staging or development website, considering robots.txt, authentication, and the SEO Spider configuration.

How To Automate Crawl Reports In Looker Studio

Learn how to set-up fully automated Google Looker Studio crawl reports to monitor site health, detect issues, & track performance.

How To Audit Core Web Vitals

Analyse which pages pass or fail Google's Core Web Vitals assessment using field data from CrUX to improve page speed.

How to Run the Screaming Frog SEO Spider in the Cloud

Learn how to run the Screaming Frog SEO Spider in the cloud, using Google Compute Engine.

How To Check For Duplicate Content

Minimise duplicate content by identifying exact duplicate pages, and near-duplicate content where some text matches between pages on a website.

Internal Linking Audit With the SEO Spider

An introduction to the various ways the SEO Spider can be used to improve internal linking.

How to Work in Teams Using the SEO Spider

Find out the best ways to work collaboratively when using the SEO Spider.

How To Test & Validate Structured Data

Audit and validate structured data against Schema.org specifications, and Google’s search feature requirements to enable special search result features.

How To Find Missing Image Alt Text & Attributes

Find pages that are missing image alt text and alt attributes, preview the images and bulk export the source pages and images to be corrected.

How To Use List Mode

Upload a list of URLs using 'list mode', and control what other elements are crawled, such as external links, or images for laser-focused crawling.

Web Scraping & Custom Extraction

Scrape any data from the HTML of a page using CSS Path, XPath and regex to enhance a crawl, such as author name, comments, shares or more.

Spell & Grammar Check Your Website

A guide on how to find spelling and grammar errors at scale across your website, ignoring words, customising the dictionary and the content checked.

How To Audit Redirects In A Site Migration Using The SEO Spider

Learn how to crawl a list of URLs, follow redirect chains, identify the final target & identify errors & non-indexable pages.

How To Debug Missing Pages In A Crawl

Find out why pages might be missing in a crawl, and how to debug and analyse why they are not being crawled.

How To Audit Canonicals

A step-by-step tutorial on how to audit canonical link elements found in the HTML and HTTP Headers for common errors and issues.

How To Audit & Validate Accelerated Mobile Pages (AMP)

This tutorial walks you through how to audit Accelerated Mobile Pages (AMP). The SEO Spider uses the official AMP validator to allow bulk URL validation.

How To Audit Hreflang

Hreflang is a simple concept, but incredibly difficult to get right at scale, and often implemented incorrectly. Find out how to audit them!

How To Audit rel="next" and rel="prev" Pagination Attributes

While Google no longer use rel=”next” and rel=”prev” pagination attributes, Bing use them as a hint. Learn how to audit your pagination set-up.

How To Audit XML Sitemaps

XML Sitemaps should be up to date, error free, and include indexable, canonical URLs only. Learn how to audit them for common issues.

How To Bulk Check Redirects

Check redirects by crawling your site, or by uploading a list of URLs in bulk. Discover if they are 301s or 302s, loops and chains.

How To Find Orphan Pages

Connect to Google Analytics and Search Console, integrate XML Sitemaps and discover orphan pages, that are not linked to internally from the website.

How To Use Custom Search

Find anything you want in the HTML or text of a website, such as old brand names, keywords for internal linking, or verifying tracking tags.

HTTP Status Codes - Why Won't My Website Crawl?

If the SEO Spider only crawls one page, or not as expected, the 'Status' and 'Status Code' are the first things to check. Learn more in our guide.

Robots.txt Testing In The SEO Spider

View URLs blocked by robots.txt, the disallow lines & use the custom robots.txt to check & validate a site's robots.txt thoroughly, and at scale.

What Is Link Score?

Find out more about our Link Score algorithm which calculates the relative importance of URLs based on the internal linking of a site.

XML Sitemap Generator

Learn how to generate an XML Sitemap (and image sitemap), including the URLs, last modified, priority and change frequency you want.

How To Use The SEO Spider In A Site Migration

Learn how to use the SEO Spider's crawling and features in a site migration to work more efficiently, and improve chances of success.

How To Bypass Geo IP Redirection In A Crawl

Learn how to set your preferred country using web form authentication to crawl a site when it uses automatic GEO IP redirection.

How To Perform A Cookie Audit

Perform a cookie audit for compliance to identify all cookies that are being automatically set without explicit user consent at scale.

How To Find Broken Bookmarks

Find and fix broken bookmarks also referred to as ‘jump links’, ‘named anchors’, and ‘skip links'.

How To Analyse Link Position

Analyse the position of links found in a crawl to identify and fix links faster, and improve internal linking for users and search engines.

How To Perform A Parity Audit

Learn how to perform a JavaScript and mobile vs desktop parity audit to uncover differences and potential SEO issues.

How To Automate The URL Inspection API

Collect URL Inspection API data in bulk, work with the 2k limit, & automate URL Inspection data reporting to monitor indexing.

How To Test Readability

Learn about the Flesch reading-ease score and how to assess readability across a website at scale.

How To Audit PDFs

Crawl, parse and audit PDFs for common issues and use-cases, such as broken links, or document titles.

How To Use The SEO Spider For Broken Link Building

Scale broken link building by using the SEO Spider to crawl a list of seed URLs, and discovering broken external links on them for outreach.

How To Debug Invalid HTML Elements In The Head

Identify invalid HTML Elements in the <head>, view metadata affected, and debug what's actually causing the issue.

An SEOs Guide To Crawling HSTS

HSTS declares to a client that it should only be accessed via HTTPS & can show as a 307 redirect. Find out why & how to see the real status code.

Configure X Virtual Framebuffer

When running headless on Linux, a display is needed if you want to use JS rendering. You can use a Virtual Frame Buffer to simulate having a display attached.

Resolving Google Analytics / Google Search Console Connection Issues

If you've experienced a 'Failed to Connect To Google Analytics / Search Console' security message, this guide will help you debug and resolve the issue.