PageSpeed Insights integration
Table of Contents
- Installation on Windows
- Installation on macOS
- Installation on Ubuntu
- Saving, opening, exporting & importing crawls
- User agent
- Checking memory allocation
- XML sitemap creation
- Crawl analysis
- Command line interface set-up
- Command line interface
- Search function
- User Interface
Spider Crawl Tab
Spider Extraction Tab
Spider Limits Tab
Spider Rendering Tab
Spider Advanced Tab
- Cookie storage
- Ignore non-indexable URLs for on-page filters
- Ignore paginated URLs for duplicate filters
- Always follow redirects
- Always follow canonicals
- Respect noindex
- Respect canonical
- Respect next/prev
- Respect HSTS policy
- Respect self referencing meta refresh
- Extract images from img srcset attribute
- Crawl fragment identifiers
- Response timeout
- 5XX response retries
Spider Preferences Tab
Other Configuration Options
- Content area
- Spelling & grammar
- Robots.txt settings
- Custom robots.txt
- URL rewriting
- User agent
- HTTP header
- Custom search
- Custom extraction
- Custom link positions
- User Interface
- Google Analytics integration
- Google Search Console integration
- PageSpeed Insights integration
- Memory allocation
- Storage mode
Lower Window Tabs
Right Side Window Tabs
PageSpeed Insights integration
Configuration > API Access > PageSpeed Insights
You can connect to the Google PageSpeed Insights API and pull in data directly during a crawl.
PageSpeed Insights uses Lighthouse, so the SEO Spider is able to display Lighthouse speed metrics, analyse speed opportunities and diagnostics at scale and gather real-world data from the Chrome User Experience Report (CrUX) which contains Core Web Vitals from real-user monitoring (RUM).
To set this up, start the SEO Spider and go to ‘Configuration > API Access > PageSpeed Insights’, enter a free PageSpeed Insights API key, choose your metrics, connect and crawl.
Setting Up A PageSpeed Insights API Key
To set-up a free PageSpeed Insights API key, login to your Google account and then visit the PageSpeed Insights getting started page.
Once you’re on the page, scroll down a paragraph and click on the ‘Get a Key’ button.
Then follow the process of creating a key – by submitting a project name, agreeing to the terms and conditions and clicking ‘next’.
It will then enable the key for PSI and provide an API key which can be copied.
Copy the key, and click ‘Done’.
Then simply paste this in the SEO Spider ‘Secret Key:’ field under ‘Configuration > API Access > PageSpeed Insights’ and press ‘connect’. This key is used when making calls to the API at https://www.googleapis.com/pagespeedonline/v5/runPagespeed.
That’s it, you’re now connected! The SEO Spider will remember your secret key, so you can ‘connect’ quickly upon starting the application each time.
If you find that your API key is saying it’s ‘failed to connect’, it can take a couple of minutes to activate. You can also check that the PSI API has been enabled in the API library as per our FAQ. If it isn’t enabled, enable it – and it should then allow you to connect.
Once you have connected, you can choose metrics and device to query under the ‘metrics’ tab.
The following speed metrics, opportunities and diagnostics data can be configured to be collected via the PageSpeed Insights API integration.
- Total Size Savings
- Total Time Savings
- Total Requests
- Total Page Size
- HTML Size
- HTML Count
- Image Size
- Image Count
- CSS Size
- CSS Count
- Font Size
- Font Count
- Media Size
- Media Count
- Other Size
- Other Count
- Third Party Size
- Third Party Count
CrUX Metrics (‘Field Data’ in PageSpeed Insights)
- CrUX Performance
- CrUX First Contentful Paint Time (sec)
- CrUX First Contentful Paint Category
- CrUX First Input Delay Time (sec)
- CrUX First Input Delay Category
- CrUX Largest Contentful Paint Time (sec)
- CrUX Largest Contentful Paint Category
- CrUX Cumulative Layout Shift
- CrUX Cumulative Layout Shift Category
- CrUX Origin Performance
- CrUX Origin First Contentful Paint Time (sec)
- CrUX Origin First Contentful Paint Category
- CrUX Origin First Input Delay Time (sec)
- CrUX Origin First Input Delay Category
- CrUX Origin Largest Contentful Paint Time (sec)
- CrUX Origin Largest Contentful Paint Category
- CrUX Origin Cumulative Layout Shift
- CrUX Origin Cumulative Layout Shift Category
Lighthouse Metrics (‘Lab Data’ in PageSpeed Insights)
- Performance Score
- Time to First Byte (ms)
- First Contentful Paint Time (sec)
- First Contentful Paint Score
- Speed Index Time (sec)
- Speed Index Score
- Time to Interactive (sec)
- Time to Interactive Score
- First Meaningful Paint Time (sec)
- First Meaningful Paint Score
- Estimated Input Latency (ms)
- Estimated Input Latency Score
- First CPU Idle (sec)
- First CPU Idle Score
- Max Potential First Input Delay (ms)
- Max Potential First Input Delay Score
- Total Blocking Time (ms)
- Total Blocking Time Score
- Cumulative Layout Shift
- Cumulative Layout Shift Score
- Eliminate Render-Blocking Resources Savings (ms)
- Defer Offscreen Images Savings (ms)
- Defer Offscreen Images Savings
- Efficiently Encode Images Savings (ms)
- Efficiently Encode Images Savings
- Properly Size Images Savings (ms)
- Properly Size Images Savings
- Minify CSS Savings (ms)
- Minify CSS Savings
- Remove Unused CSS Savings (ms)
- Remove Unused CSS Savings
- Serve Images in Next-Gen Formats Savings (ms)
- Serve Images in Next-Gen Formats Savings
- Enable Text Compression Savings (ms)
- Enable Text Compression Savings
- Preconnect to Required Origin Savings
- Server Response Times (TTFB) (ms)
- Server Response Times (TTFB) Category
- Multiple Redirects Savings (ms)
- Preload Key Requests Savings (ms)
- Use Video Format for Animated Images Savings (ms)
- Use Video Format for Animated Images Savings
- Total Image Optimization Savings (ms)
- DOM Element Count
- Efficient Cache Policy Savings
- Minimize Main-Thread Work (sec)
- Minimize Main-Thread Work Category
- Text Remains Visible During Webfont Load
You can read more about the definition of each metric, opportunity or diagnostic according to Lighthouse.
Filter by –
- Eliminate Render-Blocking Resources – This highlights all pages with resources that are blocking the first paint of the page, along with the potential savings.
- Properly Size Images – This highlights all pages with images that are not properly sized, along with the potential savings when they are resized appropriately.
- Defer Offscreen Images – This highlights all pages with images that are hidden or offscreen, along with the potential savings if they were lazy-loaded.
- Minify CSS – This highlights all pages with unminified CSS files, along with the potential savings when they are correctly minified.
- Remove Unused CSS – This highlights all pages with unused CSS, along with the potential savings when they are removed of unnecessary bytes.
- Efficiently Encode Images – This highlights all pages with unoptimised images, along with the potential savings.
- Serve Images in Next-Gen Formats – This highlights all pages with images that are in older image formats, along with the potential savings.
- Enable Text Compression – This highlights all pages with text based resources that are not compressed, along with the potential savings.
- Preconnect to Required Origin – This highlights all pages with key requests that aren’t yet prioritizing fetch requests with link rel=preconnect, along with the potential savings.
- Reduce Server Response Times (TTFB) – This highlights all pages where the browser has had to wait for over 600ms for the server to respond to the main document request.
- Avoid Multiple Redirects – This highlights all pages which have resources that redirect, and the potential saving by using the direct URL.
- Preload Key Requests – This highlights all pages with resources that are third level of requests in your critical request chain as preload candidates.
- Use Video Format for Animated Images – This highlights all pages with animated GIFs, along with the potential savings of converting them into videos.
- Avoid Excessive DOM Size – This highlights all pages with a large DOM size over the recommended 1,500 total nodes.
- Serve Static Assets With An Efficient Cache Policy – This highlights all pages with resources that are not cached, along with the potential savings.
- Minimize Main-Thread Work – This highlights all pages with average or slow execution timing on the main thread.
- Ensure Text Remains Visible During Webfont Load – This highlights all pages with fonts that may flash or become invisible during page load.
- Request Errors – This highlights any URLs which returned an error or redirect response from the PageSpeed Insights API.
Please read the Lighthouse performance audits guide for more definitions and explanations of each of the opportunities and diagnostics described above.
The speed opportunities, source pages and resource URLs that have potential savings can be exported in bulk via the ‘Reports > PageSpeed’ menu.
PageSpeed Insights API Limits
The API is limited to 25,000 queries a day at 60 queries per 100 seconds per user. The SEO Spider automatically controls the rate of requests to remain within these limits.
Please consult the ‘quotas’ section of the API dashboard to view your API usage quota.
PageSpeed Insights API Errors
The PSI Status column shows whether an API request for a URL has been a success, or there has been an error. An ‘error’ usually reflects the web interface, where you would see the same error and message.
There two most common error messages are –
- “500: Unable to process request. Please wait a while and try again” – This error is generally replicable in the web interface and our testing suggests that from time to time the PSI API is unable to process requests, possibly due to overall load capacity. If this occurs, we recommend pausing the crawl for 10mins until it’s available again and working in the web interface, and then right click and ‘re-spider’ URLs. This will re-request the PSI data for those URLs selected and continue crawling and requesting API data for other URLs.
- “500: Lighthouse returned error: ERRORED_DOCUMENT_REQUEST. Lighthouse was unable to reliably load the page you requested.” – This error is again typically replicable in the web interface and is not an issue with the SEO Spider, or the API, it is directly related to the Lighthouse audit conducted by PSI. Unfortunately ‘re-spidering’ these URLs to re-request API data generally does not help. You can provide Google with feedback about any errors you experience directly on their mailing list or ask questions via Stack Overflow.
Please read our FAQ on PageSpeed Insights API Errors for more information.
Join the mailing list for updates, tips & giveawaysHow we use the data in this form
Back to top