Semantic Search

The ‘Semantic Search’ tab allows you to enter a search query to return the most relevant pages in a crawl. It requires connecting to an AI provider to generate embeddings, before it can be used.

The semantic search functionality vectorises the search query used, and calculates the cosine similarity between the query and pages in a crawl using vector embeddings rather than keywords.

It can help quantify the relevance of content to a query for all pages in a crawl, and is more akin to how modern search engines and LLMs return content today, rather than more simplistic keyword presence and matching within text.

This functionality can be used to find relevant pages for –

• Keyword Mapping – Identify the most relevant page for a query, and optimise pages to improve relevance.
• Internal Linking – Identifying semantically similar pages for potential internal linking.
• Competitor Analysis – Analysing the scores of competitors in a SERP for a given query, to understand the most relevant.

The ‘Embedding Display’ filter can be adjusted to ‘Centroid’, to see more details about outliers found on the website and the ‘most representative page’, that is closest to the average embedding across the whole site.

If you’ve pulled embeddings from a variety of LLMs you can adjust the filter at the top to view the different results.