SEO Spider User Guide

General

Installation Installation on Windows Installation on macOS Installation on Ubuntu Installation on Fedora Crawling Saving, opening, exporting & importing crawls Configuration Scheduling Exporting Robots.txt User agent Memory Checking memory allocation Cookies XML sitemap creation Visualisations Reports Command line interface set-up Command line interface User Interface Search function Auto Updates

"How To" Guides

An SEOs Guide To Crawling HSTS Configure X Virtual Framebuffer Crawling Password Protected Websites Getting Started Guide How To Analyse Link Position How To Audit & Validate Accelerated Mobile Pages (AMP) How To Audit Canonicals How To Audit Core Web Vitals How To Audit Hreflang How To Audit PDFs How To Audit rel="next" and rel="prev" Pagination Attributes How To Audit XML Sitemaps How To Automate Crawl Reports In Looker Studio How To Automate The URL Inspection API How To Bulk Check Redirects How To Check For Duplicate Content How To Compare Crawls How To Crawl A Staging Website How To Crawl JavaScript Websites How To Crawl Large Websites How To Debug Invalid HTML Elements In The Head How To Debug Missing Pages In A Crawl How To Find Broken Bookmarks How To Find Missing Image Alt Text & Attributes How To Find Orphan Pages How To Perform A Cookie Audit How To Perform A Parity Audit How to Run the Screaming Frog SEO Spider in the Cloud How To Test & Validate Structured Data How To Test Readability How To Use Custom Search How To Use List Mode How To Use The SEO Spider In A Site Migration How to Work in Teams Using the SEO Spider HTTP Status Codes - Why Won't My Website Crawl? Resolving Google Analytics / Google Search Console Connection Issues Robots.txt Testing In The SEO Spider Site Architecture & Crawl Visualisations Guide Spell & Grammar Check Your Website Web Scraping & Custom Extraction What Is Link Score? XML Sitemap Generator How To Find Broken Links

Configuration Options

Spider Crawl Tab

Images CSS JavaScript SWF Internal hyperlinks External links Canonicals Pagination (rel next/prev) Hreflang AMP Meta refresh iframes Check links outside of start folder Crawl outside of start folder Crawl all subdomains Follow internal or external ‘nofollow’ Crawl linked XML sitemaps

Spider Extraction Tab

Page details URL details Directives Structured data HTML PDF

Spider Limits Tab

Limit crawl total Limit crawl depth Limit URLs per crawl depth Limit max folder depth Limit number of query strings Limit max URL length to crawl Max redirects to follow Limit by URL path

Spider Rendering Tab

Rendering Rendered page screen shots JavaScript error reporting Flatten Shadow DOM Flatten iframes AJAX timeout Window size

Spider Advanced Tab

Cookie storage Ignore non-indexable URLs for Issues Ignore paginated URLs for duplicate filters Always follow redirects Always follow canonicals Respect noindex Respect canonical Respect next/prev Respect HSTS policy Respect self referencing meta refresh Extract images from img srcset attribute Crawl fragment identifiers Response timeout 5XX response retries

Spider Preferences Tab

Page title & meta description width Links preferences Other character preferences

Other Configuration Options

Content area Duplicates Spelling & grammar Robots.txt URL rewriting CDNs Include Exclude Speed User agent HTTP header Custom search Custom extraction Custom link positions Google Analytics integration Google Search Console integration PageSpeed Insights integration Majestic Ahrefs Moz Authentication Segments Crawl analysis User Interface Language Proxy Storage mode Memory allocation Trusted Certificates Mode

Tabs

Top Tabs

Internal External Security Response Codes URL Page titles Meta description Meta keywords h1 h2 Content Images Canonicals Pagination Directives hreflang JavaScript Links AMP Structured data Sitemaps PageSpeed Custom search Custom extraction Analytics Search Console Validation Link Metrics Change Detection

Lower Window Tabs

URL details Inlinks Outlinks Image details Duplicate Details Resources SERP snippet Rendered page View Source HTTP Headers Cookies Structured Data Details PageSpeed Details Spelling & Grammar Details