Cookie storage

Configuration > Spider > Advanced > Cookie Storage

Google crawls the web stateless without cookies, but will accept them for the duration of a page load. Some websites can only be viewed when cookies are accepted, and fail when accepting them is disabled.

By default the SEO Spider will accept cookies for a ‘session only’. This means they are accepted for the page load, where they are then cleared and not used for additional requests in the same way as Googlebot.

You can choose to switch cookie storage to ‘Persistent’, which will remember cookies across sessions or ‘Do Not Store’, which means they will not be accepted at all.

For ‘Persistent’, cookies are stored per crawl and shared between crawler threads. Cookies are not stored when a crawl is saved, so resuming crawls from a saved .seospider file will not maintain the cookies used previously.

Cookies are reset at the start of new crawl.