SEO Spider User Guide

Download | User Guide | FAQ | Support | Terms | Purchase
General | Configuration | Tabs

This user guide provides information on the Screaming Frog SEO spider software. Please follow the relevant links in the menu below for more information.

General

Installation
Crawling
Saving & Uploading Crawls
Exporting
Robots.txt
User-Agent
Memory
Cookies
XML Sitemap Creation
Reports

Configuration Options

Spider Basic Tab

Check Images
Check CSS
Check JavaScript
Check SWF
Check External Links
Check Links Outside Of Start Folder
Follow Internal’ or External ‘nofollow’
Crawl All Subdomains
Crawl Outside Of Start Folder
Crawl Canonicals
Ignore Robots.txt

Spider Limits Tab

Limit Search Total
Limit Search Depth
Limit Max URL Length To Crawl
Limit Max Folder Depth
Limit Number of Query Strings

Spider Advanced Tab

Allow Cookies
Request Authentication
Pause On High Memory Usage
Always Follow Redirects
Respect noindex
Respect Canonical
Response Timeout
5XX Response Retries
Max Redirects To Follow

Spider Preferences Tab

Page Title & Meta Description Character Length & Pixel Width
Other Character Preferences

Other Configuration Options

URL Rewriting
Include
Exclude
Speed
User Agent
Custom
Proxy
Mode

Tabs

Internal
External
Response Codes
URI
Page Titles
Meta Description
Meta Keywords
H1
H2
Images
Directives (Older name, ‘Meta & Canonical’)
AJAX
Custom

(Lower Window)
URL Info
In Links
Out Links
Image Info
SERP Snippet

Follow Us!

Get In Touch

Call us today on 01491 415070, or for an in-depth proposal on our services, complete our contact form to request a proposal from Screaming Frog. Contact Us

Contact Us

Screaming Frog Ltd
Market Chambers,
33 Market Place,
Henley-on-Thames,
Oxfordshire,
RG9 2AA

Tel: +44 (0)1491 415070
Fax: +44 (0)1491 415071
info@screamingfrog.co.uk

Latest From Twitter

  • @vincenteillet @maximecoutant Cheers guys, appreciated.

    Retweet Reply Favorite
  • @Seo2nuit @RaphSEO We use whatever is topical/meaningful during the build :-) I am an Arsenal fan, so a French codename might just happen..!

    Retweet Reply Favorite

Looking For Something?