MCP Server

You can run crawls, analyse, export and manipulate data using the SEO Spider and node.js inside Claude, LM Studio and other AI assistants using the Screaming Frog SEO Spider MCP.

Quick Start

If you want to set up quickly and test, then read on.

• Ensure you have the latest version (24) of SEO Spider installed. The MCP integration requires it to be in database storage mode (‘File > Settings > Storage Mode’), which is the default.
• The easiest way is to use Claude Desktop and follow the Claude Desktop Setup instructions below for STDIO mode using the MCP Extension to install it.
• Connect, then try some examples prompts suggested below, such as starting a crawl or exporting data from an existing crawl.

Introduction

The SEO Spider can operate as an MCP Server, exposing a wide range of crawl data to large language models (LLMs). This includes reports, bulk exports, and other crawl-related datasets.

In addition to data access tools, the server provides two powerful groups of tools:

Node Tools

Note: the Node tools are disabled by default, you will need to enable it in the MCP Server config (‘File > Settings > MCP Server’).

Filesystem Access Tools

Because these tools are built in, there is no need to configure additional MCP servers for scripting or filesystem access. This makes the SEO Spider MCP server largely self-contained and easier to manage.

All scripts, tool outputs, and installed packages are stored in a configurable directory, which you can define under ‘Settings > MCP Server’.

The full list of tools available to the LLM is detailed in our MCP API list below.

Running the Server

There are a number of MCP Clients that exist that can use the SEO Spider as an MCP Server. We have setups for two of these:

• Claude Desktop
• LM Studio

Claude Desktop works well but can run out of tokens quickly unless you are on the paid plan. LM Studio combined with local models is a good free alternative. We have setup guides for both below.

Claude Desktop Setup

The SEO Spider MCP server can be run in two different modes.

1. STDIO Server

This is the only mode that Claude desktop supports natively. However, you can get it to accept Streamable HTTP Servers which is documented below.

In STDIO mode:

• The client (i.e. Claude Desktop) launches the Spider in CLI mode. The UI will not be visible to the user, it will run headless.
• Communication happens via standard input/output streams.
• The user must explicitly prompt the client to load a crawl.
• You can get the client to list the most recent crawls or reference an existing database crawl by using its crawl ID, which can be found by right-clicking the crawl in the ‘File > Crawls’ dialog.

2. Streamable HTTP Server

• The SEO Spider is started in UI mode and all data is visible to the user. The MCP Server can be started and stopped via the MCP top level menu.
• Actions such as opening a crawl via the MCP will be visible in the UI to the user.
• The SEO Spider exposes the MCP Server URL.

You can select which method you wish to use based upon the pros and cons above.

If you like seeing the UI and reviewing the data being analysed at the same time as Claude, then Streamable is recommended.

Licencing & Privacy: By configuring an MCP server, you must ensure the software is not accessed or interacted with by anyone other than the licensed user. You are responsible for the privacy of all data sent to, or accessed via the MCP server.

How to Add the SEO Spider as an MCP Server to Claude Desktop

There are two ways to add the SEO Spider as an MCP Server to Claude desktop, depending on whether you wish to use STDIO or Streamable mode.

STDIO Mode Setup

Please ensure you are using version 24 before installing the MCP Extension below – or you will receive an ‘Unexpected non-whitespace character’ error.

To add the SEO Spider as an MCP server to Claude Desktop, you can download our MCP Extension below.

Download SEO Spider MCP Extension

When you have downloaded the MCP extension, open up Claude Desktop, navigate to ‘Settings > Extensions’ and click ‘Advanced Settings’.

Then click on ‘Install Extension’, and select the downloaded ‘spider-mcp.mcpb’ file.

When selected, click ‘Install’.

The SEO Spider MCP will then be installed.

In Claude desktop under ‘Settings > Developer’, the extension should be visible with a message to show it is running.

Please read our FAQ if you have an ‘Unexpected non-whitespace character error‘ after installing the MCP Extension. You have installed it before the latest version with the MCP!

After configuring, please close Claude. Ensure it’s closed either via the tray icon, or via Task Manager – or it will continue to run in the background.

Then open Claude desktop to use with the SEO Spider MCP.

In this mode, you’re not required to activate the MCP Server in the SEO Spider via ‘File > Settings > MCP Server’.

However, if you wish to enable the Node.js runtime environment you will need to accept it.

This essentially allows the LLM to install any required Node packages on demand and execute custom node.js scripts to manage and manipulate data. As they are built in and self contained, additional servers are not required for scripting or file access.

Node.js Runtime Environment: Enabling this allows the AI to write and run specialised scripts to handle complex data exports and reporting. This uses node.js to excecute JavaScript code and npm to fetch and install libraries. Enabling this allows the execution of arbitrary code on your system. We have attempted to mitigate as many risks as we can, however, only grant this permission if you fully trust the AI client, as malicious scripts could access local data, network resources, or persist beyond the current session.

Streamable HTTP Server Mode Setup

Please ensure you are using version 24 before editing the config file directly – or it will error.

There isn’t an MCP extension for Streamable mode, it requires editing the config file directly.

Open Claude desktop, select ‘Settings > Developer’ and click ‘Edit Config’.

Open the selected file ‘claude_desktop_config.json’ in a text editor.

Copy and paste the streamable code below into the ‘claude_desktop_config.json’ file and save it:

{
  "mcpServers": {
    "seospider": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "http://localhost:11435/mcp"
      ]
    }
  }
}

You can replace all the existing code in the claude_desktop_config.json file, if you don’t wish to keep existing MCP, or preferences.

If you wish to add above preferences for example, you will need to add a comma, and delete the start bracket { proceeding preferences. Here is an example:

After configuring, please close Claude.

Ensure it’s closed either via the tray icon, or via Task Manager – or it will continue to run in the background. Then open to use with the SEO Spider MCP.

With this mode you must enable the MCP server functionality by going to ‘File > Settings > MCP Server’ and clicking ‘Start MCP Server’.

You can also select to enable the Node.js runtime environment.

This essentially allows the LLM to install any required Node packages on demand and execute custom node.js scripts to manage and manipulate data. As they are built in and self contained, additional servers are not required for scripting or file access.

Node.js Runtime Environment: Enabling this allows the AI to write and run specialised scripts to handle complex data exports and reporting. This uses node.js to excecute JavaScript code and npm to fetch and install libraries. Enabling this allows the execution of arbitrary code on your system. We have attempted to mitigate as many risks as we can, however, only grant this permission if you fully trust the AI client, as malicious scripts could access local data, network resources, or persist beyond the current session.

When ‘Start MCP Server’ has been activated, a ‘MCP Server Active’ message will appear in the bottom left hand corner of the UI.

Windows note: To terminate the Claude app on Windows, you need to close it from the system tray (bottom right corner of your screen) and not just by closing the application window. Simply closing the window will leave the app running in the background and also the Spider MCP server. To properly close it, right-click the Claude icon in the system tray and select “Quit”.

When you next open Claude, you should be able to interact directly with the SEO Spider via the MCP.

LM Studio Setup

To add the SEO Spider as an MCP server to LM Studio do the following:

• Select the settings cog bottom right of the LM Studio Window.
• Select Program tab.
• Press ‘Install > Edit mcp.json’

Add the “screaming-frog-mcp-server” section to the “mcpServers”:

{
  "mcpServers": {
    "screaming-frog-mcp-server": {
      "url": "http://localhost:11435/mcp"
    }
  }
}

Hardware Considerations For Local Models

Local model performance depends heavily on your hardware. For example:

• A MacBook Pro with 64 GB unified memory can run models up to around 40GB.
• Systems with smaller GPUs or less memory may require experimentation to find a suitable model.

In general:

• Claude will perform better than a local model.
• Larger local models perform better than smaller local models, especially for complex prompts.
• Thinking/reasoning models tend to give higher-quality results.
• Models must support tool calling to work with the MCP server.

Security Considerations

Allowing LLMs to run arbitrary scripts introduces inherent security risks but at the same time unlocks great power.

We have attempted to mitigate as many risks as we can, however, only grant permission if you fully trust the LLM.

Context Window and Data Volume

It is easy for an LLM to exceed its context window when generating large reports or working with extensive crawl data. This was a key reason for adding the ability of the LLM to:

• Save reports directly to files.
• Generate Node.js scripts to process data externally.

In general, it is best to start a new chat for every job you want to do to save accumulating redundant conversations in the context window.

Evolving Ecosystem and Feedback

The MCP ecosystem is evolving rapidly. The current set of tools represents an initial attempt to identify what might be useful, but real-world testing will show if there are limitations.

As more users begin integrating the SEO Spider MCP Server into agentic workflows, we expect that:

• New requirements will emerge.
• Existing tools will need refinement.
• Additional capabilities will likely be requested.

We look forward to user suggestions and feedback as to how we can improve the SEO Spider MCP.

Example Prompts

At the outset, it is worth reiterating our stance on AI – We do not recommend using AI to replace an experienced SEO professional.

We recommend using AI to improve efficiency of tasks and workflows like any other tool.

An experienced SEO professional should always analyse, review and interpret crawl data into appropriate prioritised actions relevant to each unique business, website and objectives.

Here are some examples for prompts.

API

This is how the LLM interacts with the SEO Spider. Some AI software, such as LM Studio, allows you to see the tool calls as they happen. Thinking models will show the thinking output allowing you to see the reasoning about which tool was called to achieve the prompt’s goal. Monitoring this is a good way to see if the Spider is supplying expected descriptions and error messages to the LLM.