SEO Spider

How To Connect The SEO Spider MCP To Claude.ai & ChatGPT Using Cloudflare Tunnel

Introduction

The Screaming Frog SEO Spider MCP Server lets AI assistants run crawls, analyse data and export reports directly from the SEO Spider. Claude and ChatGPT connect to custom MCP servers from their own cloud infrastructure, so they can’t reach the SEO Spider running on your machine at localhost.

Cloudflare Tunnel makes the SEO Spider reachable at a public hostname without opening any ports. Cloudflare Access then locks that hostname down with OAuth, so only people you approve can sign in and use it. The same set-up works for Claude (web, desktop and mobile) and ChatGPT.

This tutorial shows how to connect the SEO Spider MCP to Claude and ChatGPT using Cloudflare Tunnel, secured with Cloudflare Access and Managed OAuth.

Requirements

  • The SEO Spider with the MCP Server enabled.
  • A Cloudflare account with a domain on Cloudflare, and Cloudflare Zero Trust (the free plan is fine).
  • Claude web interface with custom connectors, and/or ChatGPT with developer mode.
  • Administrator access on the machine running the SEO Spider, to install cloudflared as a service.

1) Start the SEO Spider MCP Server

Click ‘File > Settings > MCP Server’, enable the Node.js runtime environment, and click ‘Start MCP Server’.

Start MCP Server and node.js

The MCP Server will run in Streamable HTTP mode at http://localhost:11435/mcp. The SEO Spider must stay open with the MCP Server running whenever you use it from Claude or ChatGPT.


2) Create a Cloudflare Tunnel

In the Cloudflare dashboard, go to ‘Zero Trust > Networking > Tunnels’ and create a new Cloudflared tunnel. Give it a name, such as ‘SEO Spider’.

Cloudflare tunnel for the SEO Spider

Cloudflare will show install instructions for your operating system. On Windows, download and run the cloudflared installer, then open Command Prompt as Administrator (right-click Command Prompt and select ‘Run as administrator’) and run the command shown in the dashboard. It looks like –

cloudflared.exe service install <your-token>

This installs cloudflared as a Windows service, so the tunnel starts automatically with your machine. The token is a secret, so don’t share it. If it’s ever exposed, delete the tunnel and create a new one.

Cloudflare Tunnel Success

The tunnel should show as ‘Healthy’ in the dashboard within a minute and then you can click ‘Continue’.


3) Add a Public Hostname

Click the three dots to the right of the tunnel, then ‘Configure > Add Route’ and select ‘Published Application’. In the tunnel settings, add a public hostname that points to the SEO Spider –

  • Subdomain – For example, sf-mcp.
  • Domain – A domain you manage in Cloudflare.
  • Service URL – localhost:11435
  • Path – /mcp
Cloudflare Tunnel public hostname settings

The MCP endpoint will be https://sf-mcp.yourdomain.com/mcp. It’s now publicly reachable without a login, so stop the MCP Server in the SEO Spider until you’ve completed steps 4 and 5, then start it again.


4) Protect the Hostname with Cloudflare Access

Go to ‘Zero Trust > Access controls > Applications’, click ‘Add an application’, choose ‘Self-hosted and private’ and click ‘Continue with Self-hosted and private’. Configure as outlined below –

Cloudflare Access Controls Applications
  • Destinations – Add your public hostname (‘subdomain’ and ‘domain’ fields at the top). Leave the path empty, so the whole hostname is protected, including the OAuth discovery URLs.
  • Access policy – Click ‘Create new policy’, set the action to ‘Allow’ and add an Include rule for ‘Emails’ with your email address.
  • Authentication – Leave ‘Accept all available identity providers’ on.
  • Name – For example, ‘sf-mcp’.

Click ‘Create’. Visiting the endpoint in a private browser window should now show a Cloudflare sign-in page.

Use the ‘Self-hosted’ application type, not the ‘MCP server’ type, which requires the server to validate Cloudflare’s access tokens itself.


5) Enable Managed OAuth

Managed OAuth turns Cloudflare Access into an OAuth server, which lets Claude and ChatGPT sign in through Access. The SEO Spider doesn’t need any changes.

Edit the application by clicking the 3 dots and ‘Edit’, then click ‘Additional settings’ and turn on ‘Managed OAuth’ via the toggle. Add the following to the allowed redirect URIs –

https://claude.ai/api/mcp/auth_callback
https://claude.com/api/mcp/auth_callback
https://chatgpt.com/connector/oauth/*

The first two are for Claude, and the third is for ChatGPT. The wildcard covers the callback URL ChatGPT creates for each connector, so it doesn’t need to be copied from ChatGPT separately.

Cloudflare Managed OAuth

Leave ‘Allow localhost clients’ and ‘Allow loopback clients’ off, then save.


6) Connect Claude

In Claude, go to ‘Plugins > Add > Add Custom Connector’ and enter your endpoint, including /mcp –

https://sf-mcp.yourdomain.com/mcp

Claude Add Custom Connector

Claude should detect the OAuth settings automatically. Leave ‘Sign in now’ and ‘Register automatically’ selected, then click ‘Add’.

Claude custom connector options with OAuth

Click ‘Allow’ and sign in on the Cloudflare page that opens.

Cloudflare Access OAuth

Once connected, the connector is available in new chats, and in the Claude desktop and mobile apps.


7) Connect ChatGPT

In ChatGPT, go to ‘Plugins’, click ‘Add’ and select ‘Add custom MCP server’.

  • Connection – Select ‘Server URL’ (not ‘Tunnel’, which is for OpenAI’s own tunnels) and enter your endpoint, including /mcp.
  • Authentication – OAuth. Under ‘Advanced OAuth settings’, ChatGPT should discover the Cloudflare OAuth endpoints, including a Registration URL.
  • OIDC enabled – Leave unticked.
ChatGPT Custom Connector

Create the plugin, then click ‘Connect’.

ChatGPT Custom Connector Connect

Click ‘Allow’ and sign in on the Cloudflare page that opens.

Cloudflare Access OAuth

Once connected, the connector is available in new chats for ChatGPT.


8) Allow Other Users (Optional)

This step is optional. Skip it if you are the only person using the connection.

New Zero Trust accounts sign users in with their Cloudflare account by default, and this is often restricted to members of your Cloudflare account. To let someone else connect without a Cloudflare account, add the One-time PIN login method, which emails them a sign-in code –

  • Go to ‘Zero Trust > Integrations > Identity providers’, click ‘Add new’ and select ‘One-time PIN’.
  • Check the application’s ‘Authentication’ settings allow it. ‘Accept all available identity providers’ includes it automatically.
  • Add their email address to the application’s ‘Allow’ policy.
Cloudflare One Time Password

They then add the connector in their own Claude or ChatGPT account and sign in with their email and the code they receive. Google sign-in can be added in the same way, via ‘Integrations > Identity providers’.

Everyone shares the same SEO Spider instance, so changes one person makes, such as switching crawl mode or clearing a crawl, affect everyone. Check you have an SEO Spider licence for each user.

Only add people you’d trust with access to your machine. Their AI assistant can run scripts on it with your Windows permissions.


9) Test the Connection

Start a new chat, enable the SEO Spider plugin and try a simple request, such as ‘Crawl www.example.com using the SEO Spider’.

ChatGPT crawl with the SEO Spider

You should receive a positive response that ChatGPT or Claude.ai has connected, and started the crawl. If you’re having problems, read on.


10) Start the Connection Next Time

The set-up only needs to be completed once. As cloudflared runs as a Windows service, the tunnel starts automatically with your machine.

After restarting, open the SEO Spider, click ‘File > Settings > MCP Server’ and click ‘Start MCP Server’. If you enable ‘Auto-start MCP Server on application launch’, this step also isn’t required.

That’s it. There’s no need to add the connector in Claude or ChatGPT again. Start a new chat and enable the SEO Spider connector.

Claude or ChatGPT may occasionally ask you to sign in again when your Cloudflare session expires. Click ‘Connect’ and sign in as before.

If the connection doesn’t work, check the tunnel shows as ‘Healthy’ under ‘Zero Trust > Networking > Tunnels’. If it doesn’t, open Windows ‘Services’, find ‘Cloudflared agent’ and click ‘Start’.


Security

  • Who can connect – Only people in your Access policy can sign in. Requests without a valid token are rejected by Cloudflare before they reach your machine.
  • Only approve your own sign-ins – Only click ‘Allow’ on a Cloudflare prompt you started yourself by connecting Claude or ChatGPT. If someone sends you a link asking you to authorise access, decline it, as approving would connect their AI assistant to your SEO Spider.
  • When it’s reachable – The tunnel runs whenever your machine is on. Stop the MCP Server or close the SEO Spider when you’re not using it. To take it offline completely, stop the ‘Cloudflared agent’ service in Windows ‘Services’.
  • What the AI can do – MCP tools can write files and run scripts on your machine. Review tool calls before approving them, particularly after crawling unfamiliar websites, which may contain text designed to manipulate AI assistants.
  • Where your data goes – Anything the AI reads is sent to Anthropic or OpenAI and passes through Cloudflare, including crawl data, page content and data from connected APIs such as Search Console or Analytics. Check this fits your client agreements before crawling client websites.
  • Keep secrets private – Never share the cloudflared token or paste it into chats.
  • Licensing – You must ensure the SEO Spider is only accessed by licensed users. Licences are per person. Each person connecting to the software requires their own licence.
  • Removing access – To remove someone, delete their email from the Access policy and click ‘Revoke existing tokens’ in the application’s settings. To disconnect completely, delete the connector in Claude or ChatGPT, delete the Access application and tunnel in Cloudflare, and run cloudflared.exe service uninstall.

Support

This tutorial should help users connect the SEO Spider MCP to Claude and ChatGPT using Cloudflare Tunnel. If you run into problems, check the common issues below –

  • ‘Cannot establish a connection to the service control manager: Access is denied’ – Command Prompt isn’t running as Administrator. Reopen it with ‘Run as administrator’.
  • ‘Couldn’t register with sign-in service’ in Claude – Claude’s callback URLs are missing from the Managed OAuth allowed redirect URIs (step 5).
  • ‘Some app settings were rejected’ in ChatGPT – Check ‘Server URL’ is selected rather than ‘Tunnel’, and that ChatGPT’s callback URLs are allowed in step 5.
  • Sign-in restricted to account members – The user is signing in with the default Cloudflare login. Add the One-time PIN method (step 8).
  • Cloudflare error after signing in – Sign-in happens at Cloudflare, so it can succeed even when the SEO Spider isn’t reachable. Check the SEO Spider is open with the MCP Server started, and the tunnel is ‘Healthy’.

For more detail, check out the following guides –

Alternatively please contact us via support and we can help.


This documentation is subject to our Terms of Use.

Join the mailing list for updates, tips & giveaways

Back to top