HTTP Headers

You can view the full HTTP response and request headers of any highlighted URL providing your crawl is set to extract HTTP Headers. This only populates the lower windowpane when selecting URLs in the top window.

To enable HTTP Header extraction, click ‘Configuration > Spider > Extraction > HTTP Headers’.

The left-hand side of the tab shows the HTTP Request Headers. The right-hand side of the tab lists the HTTP Response Headers. The columns listed in this right-hand side window include:

• Header Name – The name of the response header from the server.
• Header Value – The value of the response header from the server.

When extracted, HTTP headers are appended in separate unique columns in the Internal tab, where they can be queried alongside crawl data.

They can also be exported in bulk via ‘Bulk Export > Web > All HTTP Headers’ or in aggregate via ‘Reports > HTTP Headers > HTTP Header Summary.