Screaming Frog SEO Spider 24.0: AI integration and increased automation

An MCP server to control SEO Spider from an AI

This is the key new feature of this version. Screaming Frog introduces an MCP server (Model Context Protocol) which allows you to connect the SEO Spider to AI assistants like Claude or LM Studio. Concretely, it becomes possible to launch crawls, analyze data, export reports and manipulate the results directly from a chat interface, in natural language.

The use cases cover a wide spectrum:

• Summarize a crawl in a few words, visualize the data in a personalized form,
• Combine exports,
• Run the full analysis pipeline
• Install scripts to produce interactive visuals.

Screaming Frog gives the example of an internal mesh analysis performed with Ahrefs integration and rendered as a graph via AI.

Integration with Claude Cowork is also mentioned as a promising area, particularly for automating repetitive tasks on files and daily workflows.

It should be noted that not all SEO Spider features are yet supported in MCP. Screaming Frog plans to expand capabilities based on user feedback.

Automatic crawl comparison

Until now, comparing two crawls required manual manipulation. Version 24.0 introduces a “Auto Compare Crawls” option Available in scheduled crawl tasks and command line (CLI) crawls. Activated, it automatically compares the last two crawls of a project and generates a difference report.

This report lists all the changes that occurred between the two crawls: pages added or deleted, evolution of SEO problems detected, variations on tags, response codes, etc.

Changes visible directly in email notifications

End of crawl notifications already existed, but their content is enriched in this version. The email now includes a summary table of all the problems detected during the crawl.

If automatic comparison is activated, the email also displays a comparison table which highlights the changes between the last two crawls: indexable pages become non-indexable, title tags disappeared, peaks of error pages, etc. The idea is to allow continuous monitoring of the site without having to open the software each time.

Sending exports as attachments by email

Still in the automation logic, it is now possible to configure the automatic sending of exports as attachments at the end of a crawl. This option is set in the “Notifications” section of a scheduled task.

The principle is simple: you choose a recipient, you select the desired report (broken links, summary of problems, etc.) and the export automatically arrives in your mailbox after each crawl. Files are sent compressed. If the attachment exceeds the email host’s limit, the email notifies the recipient of the error.

Detection of non-crawable links

The SEO Spider can now identify types of links present in HTML but which do not comply with good linking practices as defined by Google. These links are qualified as non-crawlable in the interface.

They appear in the Links tab via the new “Pages With Uncrawlable Internal Outlinks” filter, as well as in a new “Link Crawlability” column in the Outlinks tab. A bulk export is available via Bulk Export > Links > Pages With Uncrawlable Internal Outlinks.

For this detection to work, the “Store” option must be enabled in Config > Spider > Crawl.

An important clarification from Screaming Frog: their research indicates that Google actually attempts to crawl anything that looks like a link in HTML, including these unconventional formats. These links are therefore often “crawlable” in practice. However, they are still not recommended because they are not treated in the same way as a standard HTML link for transmitting link signals.

Usage statistics

A small, but fun, new feature: it is now possible to consult your own software usage statistics via Help > Usage Stats. There we find the time spent crawling, the number of crawls launched, etc.

This data is stored locally on the machine. Screaming Frog themselves recognize that the practical usefulness is limited, but that it can give a concrete idea of ​​the value derived from the tool on a daily basis.

Other improvements in version 24.0

The update also includes a series of more targeted changes:

• New Linux Arm64 versions : two new packages are available for Arm architectures, under Ubuntu and Fedora.
• Syntactically invalid links : a new “Crawl Invalid Links” option under Config > Spider > Crawl allows you to process and report malformed URLs (like hppts://example.com). These URLs generally appear under Response Codes > No Response.
• Ignore empty exports : in the output settings of scheduled crawls and in Bulk Export > Multi Export, it is now possible not to export reports that do not contain any data. They will also not appear as attachments in emails.
• Ahrefs metrics by country : an Options tab appears in the Ahrefs integration, with filters to adjust the data by country and by type of volume (monthly or average).
• Filterable content clusters : in the content clusters graph, it is now possible to click on the legends to show or hide categories.
• Validation of AI models in real time : When a model is specified in an AI integration, the SEO Spider now checks whether it exists in the provider’s list of available models. If this is not the case, an alert is displayed and it is possible to choose an active one. A necessity, given how quickly AI models are deprecated or renamed.
• Global system prompt for AI integrations : in the “Advanced” tab of each AI integration, it is possible to define a system prompt to regulate the behavior of the model (role, constraints, personality, etc.) in a consistent manner across all interactions.
• Display of consumed tokens : token consumption is now visible in the “Account Information” dialog for each connected AI provider.
• Ollama Integration : The Ollama integration now supports image generation and has an adjustable “Request Timeout” setting in the advanced tab.
• Return to the name “Data Studio” : Screaming Frog renamed “Looker Studio” to “Data Studio” in its automated reports, in accordance with the latest name changes made by Google.
• Update to Java 25 : SEO Spider now runs on Java 25.

Do you want to entrust the SEO audit of your site to experts?