HTTP header
Dan Sharp
Posted 27 February, 2018 by Dan Sharp in
HTTP header
Configuration > HTTP Header
The HTTP Header configuration allows you to supply completely custom header requests during a crawl.
This means you’re able to set anything from accept-language, cookie, referer, or just supplying any unique header name. For example, there are scenarios where you may wish to supply an Accept-Language HTTP header in the SEO Spider’s request to crawl locale-adaptive content.
You can choose to supply any language and region pair that you require within the header value field.
User-agent is configured separately from other headers via ‘Configuration > User-Agent’.