Posted 5 April, 2017 by in SEO

How To Fetch & Render (Almost) Any Site

From time to time, we’ve struggled to get access to Google Search Console quickly from a client to perform a fetch and render. Or, we’ve just wanted to perform a fetch and render on a third party site that isn’t verified to us, to understand how it renders, or test if they are cloaking (by more than just user-agent).

We developed a fetch and render feature in the Screaming Frog SEO Spider, and Max Prin of Merkle has released a very useful web based fetch and render tool.

While these tools emulate how Google render, only Google’s own fetch and render will show you exactly how they see and respond to a page. So, internally we also use a little hack to render pages from unverified sites in Google Search Console.

The Simple iframe Hack

Essentially, all you need to do is create a page on a site you already have verified in Google Search Console, which contains an iframe of the page you wish to render using the desktop (1024) or mobile (411) width dimensions that Google use within fetch and render. Then set the height at 10,000 pixels, which means the scroll bars will be on the browser, rather than the iframe when you look at the results in Search Console.

A basic page, like the below, will do.

<html>
<head><title>Fetch &amp; Render Proxy</title>
</head>
<body style="margin:0px;padding:0px">
<iframe src="http://www.example.com/" frameborder="0" style="overflow:hidden;height:10000;width:1024" height="10000" width="1024"></iframe>
</script>
</body>
</html>

Then you can just fetch and render that page within Google Search Console.

Making It Easier

Rather than having to edit HTML and upload the file to your server each time, you can use PHP to read the website you wish to perform a fetch and render on from a query string parameter. This is then used as the src for the iframe. You can also choose mobile or desktop rendering using the type parameter.

For example, you can see a desktop and mobile fetch and render of Techcrunch , on our screamingprojects.com domain. Here’s how it looks in Google Search Console after you perform a fetch and render there.

Fetch and Render hack

You can download our fetch and render code on Github. This means you can simply change the ‘example.com’ domain to perform a fetch and render with the following URL queries on yourverifieddomain.com.

http://www.yourverifieddomain.com/fetch-and-render/?url=http://www.example.com/&type=mobile

http://www.yourverifieddomain.com/fetch-and-render/?url=http://www.example.com/&type=desktop

A couple of things to note –

  • You can’t load insecure (HTTP) iframes on a secure (HTTPS) domain. So, it’s best to set this up on an insecure domain.
  • Not every website allows you to use it within an iframe in this way. On that note, John Mueller suggested it would be wise to remind everyone of the X-Frame-Options HTTP header which can be used to indicate whether or not a browser should be allowed to render a page in a frame, iframe or object.

That’s it! We hope this little hack is helpful. Enjoy.