SEO Spider User Guide

Download | User Guide | FAQ | Support | Terms | Purchase
General | Configuration | Tabs

This user guide provides information on the Screaming Frog SEO spider software. Please follow the relevant links in the menu below for more information.

General

Installation
Crawling
Saving & Uploading Crawls
Exporting
Robots.txt
User-Agent
Memory
Cookies
XML Sitemap Creation
Reports
Command Line & Scheduling

Configuration Options

Spider Basic Tab

Check Images
Check CSS
Check JavaScript
Check SWF
Check External Links
Check Links Outside Of Start Folder
Follow Internal’ or External ‘nofollow’
Crawl All Subdomains
Crawl Outside Of Start Folder
Crawl Canonicals
Ignore Robots.txt

Spider Limits Tab

Limit Search Total
Limit Search Depth
Limit Max URL Length To Crawl
Limit Max Folder Depth
Limit Number of Query Strings

Spider Advanced Tab

Allow Cookies
Request Authentication
Pause On High Memory Usage
Always Follow Redirects
Respect noindex
Respect Canonical
Response Timeout
5XX Response Retries
Max Redirects To Follow

Spider Preferences Tab

Page Title & Meta Description Character Length & Pixel Width
Other Character Preferences

Other Configuration Options

URL Rewriting
Include
Exclude
Speed
User Agent
Custom
Proxy
Mode

Tabs

Internal
External
Response Codes
URI
Page Titles
Meta Description
Meta Keywords
H1
H2
Images
Directives (Older name, ‘Meta & Canonical’)
AJAX
Custom

(Lower Window)
URL Info
In Links
Out Links
Image Info
SERP Snippet

Follow Us!

Why Purchase A Licence?

  • The 500 URI crawl limit is removed
  • You can access ALL the configuration options
  • You can save and re-upload crawls
  • You can search for anything in the source code of a website with the custom source code search feature
  • You get support for any technical issues with the software
Buy a Screaming Frog SEO Spider Licence

Contact Us

Screaming Frog Ltd
Market Chambers,
33 Market Place,
Henley-on-Thames,
Oxfordshire,
RG9 2AA

Tel: +44 (0)1491 415070
Fax: +44 (0)1491 415071
info@screamingfrog.co.uk

Latest From Twitter

Twitter cannot show tweets right now. Please try again.

Looking For Something?