Assume pages are HTML

Dan Sharp

Posted 7 May, 2024 by in

Assume pages are HTML

Configuration > Spider > Advanced > Assume Pages are HTML

When enabled, any URL without a content-type will be assumed to be HTML and crawled.


Dan Sharp is founder & Director of Screaming Frog. He has developed search strategies for a variety of clients from international brands to small and medium-sized businesses and designed and managed the build of the innovative SEO Spider software.

Comments are closed.

Back to top