User Guide

For the SEO Spider

Configuration Options

Spider Crawl Tab






Internal hyperlinks

External links


Pagination (rel next/prev)



Meta refresh


Mobile alternate

Check links outside of start folder

Crawl outside of start folder

Crawl all subdomains

Follow internal or external ‘nofollow’

Crawl linked XML sitemaps

Spider Extraction Tab

Page details

URL details


Structured data



Spider Limits Tab

Limit crawl total

Limit crawl depth

Limit URLs per crawl depth

Limit max folder depth

Limit number of query strings

Limit max URL length to crawl

Max redirects to follow

Limit by URL path

Spider Rendering Tab


Rendered page screen shots

JavaScript error reporting

Flatten Shadow DOM

Flatten iframes

Enable website archive

AJAX timeout

Window size

Spider Advanced Tab

Cookie storage

Ignore non-indexable URLs for Issues

Ignore paginated URLs for duplicate filters

Always follow redirects

Always follow canonicals

Respect noindex

Respect canonical

Respect next/prev

Respect HSTS policy

Respect self referencing meta refresh

Extract images from img srcset attribute

Crawl fragment identifiers

Perform HTML validation

Green hosting carbon calculation

Assume pages are HTML

Response timeout

5XX response retries

Spider Preferences Tab

Page title & meta description width

Links preferences

Other character preferences

Other Configuration Options

Content area


Spelling & grammar


URL rewriting





User agent

HTTP header

Custom search

Custom extraction

Custom link positions

Custom JavaScript

Google Analytics integration

Google Search Console integration

PageSpeed Insights integration






Crawl analysis

User Interface



Storage mode

Memory allocation

Trusted Certificates



Getting Started Guide

How To Find Broken Links Using The SEO Spider

Site Architecture & Crawl Visualisations Guide

How To Compare Crawls

How To Crawl JavaScript Websites

How To Crawl Large Websites

How To Crawl A Staging Website

How To Automate Crawl Reports In Looker Studio

How To Audit Core Web Vitals

How to Run the Screaming Frog SEO Spider in the Cloud

How To Check For Duplicate Content

How to Crawl with ChatGPT

Internal Linking Audit With the SEO Spider

How to Work in Teams Using the SEO Spider

How To Test & Validate Structured Data

How To Find Missing Image Alt Text & Attributes

How To Use List Mode

Web Scraping & Custom Extraction

Spell & Grammar Check Your Website

How To Audit Redirects In A Site Migration Using The SEO Spider

How To Debug Missing Pages In A Crawl

How To Audit Canonicals

How To Audit Mobile Usability

How To Audit & Validate Accelerated Mobile Pages (AMP)

How To Audit Hreflang

How To Audit rel=”next” and rel=”prev” Pagination Attributes

How To Audit XML Sitemaps

How To Bulk Check Redirects

How To Find Orphan Pages

How To Use Custom Search

HTTP Status Codes – Why Won’t My Website Crawl?

Robots.txt Testing In The SEO Spider

What Is Link Score?

XML Sitemap Generator

How To Use The SEO Spider In A Site Migration

How To Bypass Geo IP Redirection In A Crawl

How To Perform A Cookie Audit

How To Find Broken Bookmarks

How To Analyse Link Position

How To Perform A Parity Audit

How To Automate The URL Inspection API

How To Test Readability

How To Audit PDFs

How To Use The SEO Spider For Broken Link Building

How To Debug Invalid HTML Elements In The Head

An SEOs Guide To Crawling HSTS

Configure X Virtual Framebuffer

Resolving Google Analytics / Google Search Console Connection Issues

Crawling Password Protected Websites

Join the mailing list for updates, tips & giveaways

Back to top