How To Fetch & Render (Almost) Any Site

Dan Sharp

Posted 5 April, 2017 by in SEO

How To Fetch & Render (Almost) Any Site

From time to time, we’ve struggled to get access to Google Search Console quickly from a client to perform a fetch and render. Or, we’ve just wanted to perform a fetch and render on a third party site that isn’t verified to us, to understand how it renders, or test if they are cloaking (by more than just user-agent).

We developed a fetch and render feature in the Screaming Frog SEO Spider, and Max Prin of Merkle has released a very useful web based fetch and render tool.

While these tools emulate how Google render, only Google’s own fetch and render will show you exactly how they see and respond to a page. So, internally we also use a little hack to render pages from unverified sites in Google Search Console.

The Simple iframe Hack

Essentially, all you need to do is create a page on a site you already have verified in Google Search Console, which contains an iframe of the page you wish to render using the desktop (1024) or mobile (411) width dimensions that Google use within fetch and render. Then set the height at 10,000 pixels, which means the scroll bars will be on the browser, rather than the iframe when you look at the results in Search Console.

A basic page, like the below, will do.

<html>
<head><title>Fetch &amp; Render Proxy</title>
</head>
<body style="margin:0px;padding:0px">
<iframe src="http://www.example.com/" frameborder="0" style="overflow:hidden;height:10000;width:1024" height="10000" width="1024"></iframe>
</script>
</body>
</html>

Then you can just fetch and render that page within Google Search Console.

Making It Easier

Rather than having to edit HTML and upload the file to your server each time, you can use PHP to read the website you wish to perform a fetch and render on from a query string parameter. This is then used as the src for the iframe. You can also choose mobile or desktop rendering using the type parameter.

For example, you can see a desktop and mobile fetch and render of Techcrunch , on our screamingprojects.com domain. Here’s how it looks in Google Search Console after you perform a fetch and render there.

Fetch and Render hack

You can download our fetch and render code on Github. This means you can simply change the ‘example.com’ domain to perform a fetch and render with the following URL queries on yourverifieddomain.com.

http://www.yourverifieddomain.com/fetch-and-render/?url=http://www.example.com/&type=mobile

http://www.yourverifieddomain.com/fetch-and-render/?url=http://www.example.com/&type=desktop

A couple of things to note –

  • You can’t load insecure (HTTP) iframes on a secure (HTTPS) domain. So, it’s best to set this up on an insecure domain.
  • Not every website allows you to use it within an iframe in this way. On that note, John Mueller suggested it would be wise to remind everyone of the X-Frame-Options HTTP header which can be used to indicate whether or not a browser should be allowed to render a page in a frame, iframe or object.

That’s it! We hope this little hack is helpful. Enjoy.

Dan Sharp is founder & Director of Screaming Frog. He has developed search strategies for a variety of clients from international brands to small and medium-sized businesses and designed and managed the build of the innovative SEO Spider software.

33 Comments

  • Edward Sturm 7 years ago

    Will be using this today on some single page apps with good rankings to see if Fetch and Render is irrelevant for SPAs.

    Reply
    • Marcelo Farjalla 7 years ago

      What was your finding?

      Reply
  • Stefan 7 years ago

    Brilliant, just brilliant! A simple solutions for a complicated problem!

    Reply
  • WpSEO.it 7 years ago

    Hi,
    very useful tool!! Congratulations!
    I tried it for some sites and for one of this i receive a blank page.
    How can i check what is the problem?
    Thanks

    Reply
  • David 7 years ago

    I’m also getting the blank page wpseoit noted – I copy/pasted the HTML code example into a textpad, FTP uploaded, tried a couple different URLs, got blanks in GSC in both cases, any ideas to fix? Thanks!

    Reply
  • Hans 7 years ago

    Awesome! Thank you for providing the code on Github.

    Reply
    • David 7 years ago

      Thanks, Dan – just sent from my @3qdigital email. Cheers.

      Reply
  • Chris Lever 7 years ago

    What an ingenuous idea, although is there any real practical uses other than testing stuff out?

    Reply
  • Angelo 7 years ago

    How do you automate this? Does google provide a search console API to run “fetch as google”? If not, what’s the point running inside Google search console?

    Reply
    • screamingfrog 7 years ago

      Hey Angelo,

      They don’t for fetch as Gbot. The point of fetching through Search Console, is to test how Google respond and render the content.

      Cheers.

      Dan

      Reply
  • russell 7 years ago

    I was reading this post yesterday ( April 5, 2017) https://www.screamingfrog.co.uk/crawl-javascript-seo/ and the last paragraph http://nimb.ws/omevn4 got me thinking how can I use google’s fetch and render tool to test a third party site, and today I am testing this feature, it’s a good feeling, Thanks Dan!

    Reply
  • Travis Causey 7 years ago

    Gold. Thanks!

    Reply
  • Michael 7 years ago

    Brilliant. Now Google is going to fix this. Thank you. Thank you very much.

    Reply
  • Yasin Aberra 7 years ago

    Thanks for this post!

    Reply
  • Per 7 years ago

    Thank you for this – good stuff :)

    Reply
  • Ivan Panchev 7 years ago

    Hello,
    This is a useful hack to fetch and render pages, especially on your competitor’s pages. Everything works properly.
    Thanks for sharing!
    Best regards,
    Ivan

    Reply
  • Gordon 7 years ago

    Thank you for sharing this.

    Reply
  • Luca Spinelli 7 years ago

    One of the best. Amazing work guys.

    Reply
  • Gis 7 years ago

    A simple solutions for a complicated problem. Thanks for this solution.

    Reply
  • Ahad Arzi 6 years ago

    It’s a nice feature. Let me try!

    Reply
  • Wes Foster 6 years ago

    This is a very interesting approach. I guess iframes still have their place in society. Thanks!

    Reply
  • Fred Harrington 6 years ago

    Never would have thought it was this easy! Never would have thought of this solution myself either, haha.

    Reply
  • Brilliant! Thank you!

    Reply
  • ary 6 years ago

    It’s a nice feature. Let me try!

    Reply
  • Rupesh Kumar 6 years ago

    I am facing rendering issue where Googlebot cannot see even half of my website page, like this one https://www.stemjar.com/watch-free-movies-online/. You can see the snapshot here – https://www.screencast.com/t/lJXReHf8 . I tried to implement your solution but that does not improve my fetch and render result. I am getting the same result as before.

    Can you help me in this?

    Reply
  • Prashant Ojha 5 years ago

    Recently google has changed the search console tool and the new feature of Testing live URL is amazing. It can give you brief overview of Fetch & rendered results. Just put the URL in inspection box, click on request indexing (if not indexed in case of already indexed URL’s test live url option is available automatically) and See the HTML, Screenshot and can even check the page resource code, HTTP response and Java Script Console messages.

    Reply
  • Roberto 5 years ago

    Thanks for this post, it’s a nice feature. Let me try!

    Reply

Leave A Comment.

Back to top