MCP Server

sf_list_crawls

Retrieves a list of recent web crawl jobs, ordered chronologically with the most recent first. By default, it returns the 10 latest crawls, providing a snapshot of recent activity and status.

sf_load_crawl

Loads a crawl into the SEO Spider application. The load runs in the background: this tool returns as soon as the load has started. Call sf_task_status to check its progress and to retrieve the result once it has completed, or sf_cancel_task to stop it before it finishes.

sf_export_crawl

Export the currently loaded crawl. Only the following modes are supported: [Spider, List, API]. The export is written in the background: this tool returns as soon as the write has started. Call sf_task_status to check its progress and to retrieve the result once it has completed, or sf_cancel_task to stop it before it finishes.

sf_set_crawl_mode

Switches the crawl mode: SPIDER for sf_crawl, LIST for sf_crawl_from_list. Each crawl tool is refused in the other mode. A few settings are kept per mode (the reply lists them): switching saves their values for the mode being left and loads the new mode’s own, so set them after switching. Refused while a crawl is loaded: call sf_clear_crawl first.

sf_crawl

Starts a crawl with optional config. This tool returns as soon as the crawl has started, not when it has finished, so its result says nothing about what was found. Call sf_crawl_progress and wait for ‘crawlRunning’ to be false before reading any results, or they will be incomplete.

sf_crawl_from_list

Starts a List mode crawl with URLs from a file or inline.

sf_pause_crawl

Pauses a running crawl

sf_resume_crawl

Resumes a paused crawl

sf_clear_crawl

Clear a paused crawl

sf_crawl_progress

Get the progress of a running crawl. ‘crawlRunning’ is false once the crawl has stopped, whether it finished on its own, was paused or has not started yet – poll this to find out when a crawl launched by sf_crawl is over. If the crawl stopped with only the start URL completed, it most likely failed: call sf_check_faq_before_answering before telling the user why. ‘crawlProgress.percentComplete’ reaching 100 does not mean the crawl is over: it is the share of queued URLs already done, and more can still be queued. Post-crawl work continues after that, so wait for ‘postCrawlAnalysisProgress.percentComplete’ to reach 100 as well before reading results that crawl analysis populates. While ‘crawlRunning’ is true the tools that export or report, read a crawled URL, change the configuration, load a crawl or start one are refused as busy, so wait for it to be false before calling them – crawl analysis and a crawl being saved count as running too. ‘crawlId’ identifies the crawl the SEO Spider has open.

sf_generate_report

Generates a report for the requested category and writes it to the file path you supply. Returns export metadata not the report contents. Use the sf_list_reports tool to get a list of all available reports. The export is written in the background: this tool returns as soon as the write has started. Call sf_task_status to check its progress and to retrieve the result once it has completed, or sf_cancel_task to stop it before it finishes.

sf_task_status

Reports the status of the background task that is running (in progress, succeeded or failed) and returns the task result details once it has completed.

sf_cancel_task

Cancels the background task currently running, freeing the SEO Spider to serve other tools. Use this when a task is taking too long or was started by mistake. Any partly completed work is discarded.

sf_generate_bulk_export

Generates a bulk export for the requested category and writes it to the output path you supply. Returns export metadata, not the exported data. Use the sf_list_bulk_exports tool to get a list of all available bulk exports. The export is written in the background: this tool returns as soon as the write has started. Call sf_task_status to check its progress and to retrieve the result once it has completed, or sf_cancel_task to stop it before it finishes.

sf_bulk_export_page_content

Bulk-exports page content for all crawled HTML URLs in NDJSON format to the file path you supply. Returns export metadata, not the page content. The export is written in the background: this tool returns as soon as the write has started. Call sf_task_status to check its progress and to retrieve the result once it has completed, or sf_cancel_task to stop it before it finishes.

sf_export_seo_element_urls

Export the URLs in one SEO element filter, with their data fields, as NDJSON to the file path you supply. Call sf_list_filters_for_seo_element for the filter names. A null value means the data is unavailable; do not guess or infer it. The ‘urlContent’ field is only exported when named in data_fields. It holds the visible text of HTML pages and base64 encoded images, so it can make the file very large: limit rows with max_rows and read the file line by line, not whole. The result is export metadata with one sample row, not the rows. The export is written in the background: this tool returns as soon as the write has started. Call sf_task_status to check its progress and to retrieve the result once it has completed, or sf_cancel_task to stop it before it finishes.

sf_export_embeddings

Generates a CSV file with the URL and embeddings. The export is written in the background: this tool returns as soon as the write has started. Call sf_task_status to check its progress and to retrieve the result once it has completed, or sf_cancel_task to stop it before it finishes.

sf_get_url_screenshot

Get the stored screenshot of the web page for the URL. If a file path is supplied will save the image to that file. If no file path is supplied will return the base64 encoded image string.

sf_url_info

Generates a JSON report with information on the supplied URL

sf_url_content

Get the content of a specific URL in the loaded crawl

• If the URL is an image, generates base64 encoded image content.
• If the URL has HTML content type, generates the page content in the form asked for.

sf_url_links

Lists all of the URL inlinks or outlinks depending on the links_direction argument.

sf_open_url_in_browser

Opens the URL or absolute file path in an external browser

sf_list_reports

Lists all available reports

sf_list_bulk_exports

Lists all available bulk exports

sf_list_filters_for_seo_element

List the filters the supplied seo element currently has. For most elements that is a fixed set and the list never changes. For AI, Custom Extraction, Custom Search and Custom JavaScript the filters come from configuration, so only the configured prompts, extractors and searches appear, each under its own name, alongside ‘All’. One with nothing configured lists just ‘All’, and that is not the whole set it could have: add some with sf_set_config_spider, crawl, then list again.

sf_list_fields_for_seo_element_filter

List the available data fields for the supplied seo element name and filter type. The filter must be one this crawl actually filled, which for AI, Custom Extraction, Custom Search and Custom JavaScript means one configured before the crawl ran; call sf_list_filters_for_seo_element for the names and sf_get_config_spider for what is configured.

sf_list_segment_query_columns

Lists the columns a segment query may reference, for the configuration currently loaded. Each entry gives the ‘column’ name to put in square brackets in the query, the ‘label’ the UI shows for it, and the ‘group’ it is filed under. Call this before writing a segment query: column names are internal identifiers rather than UI column headings, and an unrecognised name makes the whole query invalid so the segment matches nothing.

sf_list_ai_prompt_targets

Lists the prompt targets an AI prompt may use, for the configuration currently loaded. A prompt target is the crawled data sent to the model: it goes in a prompt’s ‘promptTarget’ field when its promptType is SIMPLE, or inside curly braces in the prompt text when its promptType is ADVANCED. Each entry gives the ‘promptTarget’ name to write, the ‘label’ the UI shows for it, the ‘group’ it is filed under, and the ‘contentType’ of prompt it belongs to. Call this before writing a prompt: the names are internal identifiers rather than UI column headings, and an unrecognised one is rejected when the config is applied. Custom extractor and custom JavaScript targets are only listed once the extractor or snippet they read has itself been configured.

sf_run_node_script

Runs the node.js script with the supplied optional arguments

sf_npm_install

Executes npm install <dependency> to install a specific package.

sf_list_config_categories

Lists every top-level configuration category in SEO Spider. Each entry gives the category’s ‘name’ – what sf_get_config_sub_schema and sf_get_config_spider take as their category, and what a sf_set_config_spider payload keys that category’s contents under – and a one-line ‘description’ of what that category covers. Call sf_get_config_sub_schema for a category’s full description and the fields it accepts.

sf_get_config_sub_schema

Returns the full JSON Schema for a configuration sub-category. It describes the object that goes under the category name in a sf_set_config_spider payload, not the payload itself. Use sf_list_config_categories first to discover available category names.

sf_get_config_spider

Returns the current JSON configuration for SEO Spider. If a category is specified, only that category’s configuration is returned, still keyed by its category name; otherwise the full configuration is returned. Either result can be passed straight back to sf_set_config_spider unchanged. Use sf_list_config_categories and sf_get_config_sub_schema to discover valid category names and their schemas.

sf_set_config_spider

Applies a partial JSON configuration to SEO Spider. Only the fields present in the JSON are updated; all other settings retain their current values. A field holding a list is replaced by the list you send rather than merged into it, so send every entry you want to keep, not just the ones you are adding. Use sf_list_config_categories and sf_get_config_sub_schema to discover valid category names and their schemas.

sf_list_issue_filter_keys

Lists every issue the Spider can report, for the configuration currently loaded. Each entry gives the ‘filterKey’ to use as a key in issueOverridesConfig.overrides, the ‘name’ and ‘tab’ the UI shows it under, and the ‘defaultType’ and ‘defaultPriority’ it is classified as when nothing overrides it. Call this before writing issueOverridesConfig: filter keys are internal identifiers rather than UI filter names, and only the ones listed here can be overridden.

sf_list_api_connections

Lists every external API SEO Spider can use and its live connection state, as ‘googleApis’ and ‘otherApis’. An API’s data reaches a crawl only when it is connected here; a crawl using a disconnected provider still succeeds, its columns just come back empty, so check here before trusting empty columns. Each entry’s ‘api’ is the provider argument for sf_connect_api, and ‘configCategory’ is the category sf_set_config_spider takes for its options, or null where none of them can be set over MCP. ‘hasStoredCredentials: false’ means the user must supply the key or authorise the account in the desktop app first – retrying will not help. ‘googleApis’ also report ‘googleAccount’: ‘savedAccounts’, the only valid account arguments, plus ‘defaultAccount’, ‘connectedAccount’ and ‘selection’ – what that account connects, such as a Search Console site or a GA4 account, property and data stream. A selection cannot be chosen over MCP; where there is none, name a saved account or ask the user to select one in the desktop app.

sf_connect_api

Connects SEO Spider to an external API using credentials already saved on this machine. It never accepts an API key: keys stay in the desktop app’s config and must never be sent over MCP. Where sf_list_api_connections reports no stored credentials for the provider, the user must enter the key or authorise the account in the desktop app first. The account argument is the only choice on offer: the Search Console site, and the Google Analytics 4 account, property and data stream, are whatever the desktop app last saved. The reply names them, so read it before trusting that provider’s data and ask the user to change the selection in the desktop app if it is wrong. Connecting an AI provider means prompts in a later crawl consume quota, so connect only when the user has asked for API analysis. Refused while a crawl is running: connect before starting one.

sf_disconnect_api

Disconnects SEO Spider from an external API. Saved credentials are untouched: the desktop app keeps the key or authorised account, so sf_connect_api can reconnect without the user doing anything. A later crawl then collects no data for that API, leaving its columns empty, so disconnect an API the user no longer wants requested – an AI provider, whose prompts consume quota on every crawl, most of all. A provider that is not connected is reported as such rather than failing. Refused while a crawl is running: disconnect between crawls.

sf_read_text_file

Read the contents of a text file. The whole file is returned only if it fits within the server’s maximum response size; a file over that limit is rejected outright rather than truncated, so a large export has to be read a page at a time with the ‘skip’ and ‘limit’ arguments.

sf_write_text_file

Writes UTF-8 content to a text file. Caution, will overwrite existing files without warning

sf_allowed_base_directory

Returns the allowed base directory that this server can access for reading and writing files. Subdirectories within this directory are also accessible. Use this to understand which directory and its nested paths are available before trying to access files.

sf_list_directories

Get a detailed listing of all files and directories in a specified relative path. Results clearly distinguish between files and directories with [FILE] and [DIR] prefixes. This tool is essential for understanding directory structure and finding specific files within a directory. Only works within allowed directories.

sf_create_directory

Create a new directory or ensure a directory exists. Can create multiple nested directories in one operation. If the directory already exists, this operation will succeed silently. Perfect for setting up directory structures for projects or ensuring required paths exist. Only works within allowed directories.

sf_check_faq_before_answering

Official SEO Spider FAQ. General knowledge of the SEO Spider is often wrong or out of date, so check this before answering any question about its behaviour, settings or results, or a failed crawl. Call without ids for the question list, then with the relevant ids for answers.