Tutorials

For the SEO Spider

Getting Started Guide

Our beginners guide to getting started using the SEO Spider. Including initial set-up, crawling, and viewing reports and issues discovered.

Continue Reading

How To Find Broken Links Using The SEO Spider

Learn how to crawl your website and find broken links (404 errors), view which pages link to them, the link text, and export in bulk.

Continue Reading

Site Architecture & Crawl Visualisations Guide

Visualise site architecture and internal linking to help communicate ideas and reveal underlying patterns that are harder to uncover in data, and spreadsheets.

Continue Reading

How To Compare Crawls

Compare crawls to see how data, issues and opportunities have changed over time to track progress and monitor site health.

Continue Reading

How To Crawl JavaScript Websites

Render web pages using the integrated Chromium WRS to crawl dynamic, JavaScript rich websites and frameworks, such as Angular, React and Vue.js.

Continue Reading

How To Crawl Large Websites

Crawl large websites by switching to database storage mode, increasing memory and by configuring the crawl to extract the data you need.

Continue Reading

How To Crawl A Staging Website

Find out how to crawl a staging or development website, considering robots.txt, authentication, and the SEO Spider configuration.

Continue Reading

How To Automate Crawl Reports In Looker Studio

Learn how to set-up fully automated Google Looker Studio crawl reports to monitor site health, detect issues, & track performance.

Continue Reading

How To Audit Core Web Vitals

Analyse which pages pass or fail Google's Core Web Vitals assessment using field data from CrUX to improve page speed.

Continue Reading

How to Run the Screaming Frog SEO Spider in the Cloud

Learn how to run the Screaming Frog SEO Spider in the cloud, using Google Compute Engine.

Continue Reading

How To Check For Duplicate Content

Minimise duplicate content by identifying exact duplicate pages, and near-duplicate content where some text matches between pages on a website.

Continue Reading

How to Work in Teams Using the SEO Spider

Find out the best ways to work collaboratively when using the SEO Spider.

Continue Reading

How To Test & Validate Structured Data

Audit and validate structured data against Schema.org specifications, and Google’s search feature requirements to enable special search result features.

Continue Reading

How To Find Missing Image Alt Text & Attributes

Find pages that are missing image alt text and alt attributes, preview the images and bulk export the source pages and images to be corrected.

Continue Reading

How To Use List Mode

Upload a list of URLs using 'list mode', and control what other elements are crawled, such as external links, or images for laser-focused crawling.

Continue Reading

Web Scraping & Custom Extraction

Scrape any data from the HTML of a page using CSS Path, XPath and regex to enhance a crawl, such as author name, comments, shares or more.

Continue Reading

Spell & Grammar Check Your Website

A guide on how to find spelling and grammar errors at scale across your website, ignoring words, customising the dictionary and the content checked.

Continue Reading

How To Audit Redirects In A Site Migration Using The SEO Spider

Learn how to crawl a list of URLs, follow redirect chains, identify the final target & identify errors & non-indexable pages.

Continue Reading

How To Debug Missing Pages In A Crawl

Find out why pages might be missing in a crawl, and how to debug and analyse why they are not being crawled.

Continue Reading

How To Audit Canonicals

A step-by-step tutorial on how to audit canonical link elements found in the HTML and HTTP Headers for common errors and issues.

Continue Reading

How To Audit & Validate Accelerated Mobile Pages (AMP)

This tutorial walks you through how to audit Accelerated Mobile Pages (AMP). The SEO Spider uses the official AMP validator to allow bulk URL validation.

Continue Reading

How To Audit Hreflang

Hreflang is a simple concept, but incredibly difficult to get right at scale, and often implemented incorrectly. Find out how to audit them!

Continue Reading

How To Audit rel="next" and rel="prev" Pagination Attributes

While Google no longer use rel=”next” and rel=”prev” pagination attributes, Bing use them as a hint. Learn how to audit your pagination set-up.

Continue Reading

How To Audit XML Sitemaps

XML Sitemaps should be up to date, error free, and include indexable, canonical URLs only. Learn how to audit them for common issues.

Continue Reading

How To Bulk Check Redirects

Check redirects by crawling your site, or by uploading a list of URLs in bulk. Discover if they are 301s or 302s, loops and chains.

Continue Reading

How To Find Orphan Pages

Connect to Google Analytics and Search Console, integrate XML Sitemaps and discover orphan pages, that are not linked to internally from the website.

Continue Reading

How To Use Custom Search

Find anything you want in the HTML or text of a website, such as old brand names, keywords for internal linking, or verifying tracking tags.

Continue Reading

HTTP Status Codes - Why Won't My Website Crawl?

If the SEO Spider only crawls one page, or not as expected, the 'Status' and 'Status Code' are the first things to check. Learn more in our guide.

Continue Reading

Robots.txt Testing In The SEO Spider

View URLs blocked by robots.txt, the disallow lines & use the custom robots.txt to check & validate a site's robots.txt thoroughly, and at scale.

Continue Reading

What Is Link Score?

Find out more about our Link Score algorithm which calculates the relative importance of URLs based on the internal linking of a site.

Continue Reading

XML Sitemap Generator

Learn how to generate an XML Sitemap (and image sitemap), including the URLs, last modified, priority and change frequency you want.

Continue Reading

How To Use The SEO Spider In A Site Migration

Learn how to use the SEO Spider's crawling and features in a site migration to work more efficiently, and improve chances of success.

Continue Reading

How To Bypass Geo IP Redirection In A Crawl

Learn how to set your preferred country using web form authentication to crawl a site when it uses automatic GEO IP redirection.

Continue Reading

How To Perform A Cookie Audit

Perform a cookie audit for compliance to identify all cookies that are being automatically set without explicit user consent at scale.

Continue Reading

How To Find Broken Bookmarks

Find and fix broken bookmarks also referred to as ‘jump links’, ‘named anchors’, and ‘skip links'.

Continue Reading

How To Analyse Link Position

Analyse the position of links found in a crawl to identify and fix links faster, and improve internal linking for users and search engines.

Continue Reading

How To Perform A Parity Audit

Learn how to perform a JavaScript and mobile vs desktop parity audit to uncover differences and potential SEO issues.

Continue Reading

How To Automate The URL Inspection API

Collect URL Inspection API data in bulk, work with the 2k limit, & automate URL Inspection data reporting to monitor indexing.

Continue Reading

How To Test Readability

Learn about the Flesch reading-ease score and how to assess readability across a website at scale.

Continue Reading

How To Audit PDFs

Crawl, parse and audit PDFs for common issues and use-cases, such as broken links, or document titles.

Continue Reading

How To Use The SEO Spider For Broken Link Building

Scale broken link building by using the SEO Spider to crawl a list of seed URLs, and discovering broken external links on them for outreach.

Continue Reading

How To Debug Invalid HTML Elements In The Head

Identify invalid HTML Elements in the <head>, view metadata affected, and debug what's actually causing the issue.

Continue Reading

An SEOs Guide To Crawling HSTS

HSTS declares to a client that it should only be accessed via HTTPS & can show as a 307 redirect. Find out why & how to see the real status code.

Continue Reading

Configure X Virtual Framebuffer

When running headless on Linux, a display is needed if you want to use JS rendering. You can use a Virtual Frame Buffer to simulate having a display attached.

Continue Reading

Resolving Google Analytics / Google Search Console Connection Issues

If you've experienced a 'Failed to Connect To Google Analytics / Search Console' security message, this guide will help you debug and resolve the issue.

Continue Reading

Crawling Password Protected Websites

Crawl websites that require a login, using web forms authentication using our inbuilt Chrome browser.

Continue Reading

Join the mailing list for updates, tips & giveaways

Back to top