User agent

Configuration > User-Agent

The user-agent configuration allows you to switch the user-agent of the HTTP requests made by the SEO Spider and which robots.txt directives are followed. By default the SEO Spider makes requests using its own ‘Screaming Frog SEO Spider user-agent string.

However, it has inbuilt preset user-agents for Googlebot, Bingbot, various browsers and more. This allows you to switch between them quickly when required. This feature also has a custom user-agent settings which allows you to specify your own user-agent:

• HTTP Request User-Agent: sets the user-agent field of the HTTP request header.
• Robots User-Agent: defines the user-agent used for following robots.txt directives.

Details on how the SEO Spider handles robots.txt can be found here.