Google is testing a new protocol to distinguish good bots from impostors

Website owners have struggled for years to definitively identify the robots that crawl them. Google is experimenting with a new cryptography-based approach called Web Bot Auth that could permanently change the way bots prove their identity online.

The problem Web Bot Auth seeks to solve

Today, any malicious bot can impersonate Googlebot or any other legitimate crawler. Just copy the correct user-agent. Site owners then have few reliable ways to tell the difference between a trusted robot and an imposter.

Current methods rely primarily on three signals: the user-agent (a simple declarative text string), the IP address, and reverse DNS verification. These approaches have their limits. They can be bypassed, they require manual configurations between each site and each service, and they become increasingly difficult to maintain at scale.

This is precisely the problem that Google seeks to address with Web Bot Auth.

What Web Bot Auth actually is

Web Bot Auth, whose full technical name is HTTP Message Signatures Directoryis a protocol proposed within the framework of the IETF (Internet Engineering Task Force). Its objective is to automate the trust relationship between websites and the automated services that contact them.

The principle is simple to understand : rather than simply declaring its identity via a user-agent, a bot using Web Bot Auth must cryptographically prove that it is who it claims to be. A malicious bot that copies the name of a legitimate crawler will not be able to reproduce this evidence.

Concretely, the protocol works in three stages.

• Standardized key files. Public keys are stored in a universal format, the JSON Web Key Set (JWKS), readable by any server.
• Addresses known in advance. These keys are hosted in a predictable location on the service domain, in the /.well-known/ directory, which avoids any prior negotiation between the site and the bot.
• Self-identifying queries. Each HTTP request sent by a participating bot includes a new header, Signature-Agent, which acts as a digital business card and points directly to the key directory of the relevant service.

For Google Agent, this directory is accessible at https://agent.bot.goog/.well-known/http-message-signatures-directory.

Why crypto is changing things

The real novelty of Web Bot Auth is to replace a declarative identity with a verifiable identity. Today, a bot’s recognition is based on what it says about itself. Tomorrow, with this generalized protocol, it will be able to rely on what it can demonstrate.

A spoofing bot can copy a name, a user-agent or even an IP address. It cannot reproduce the cryptographic signature associated with the private keys of a legitimate service. It is this asymmetry that makes the protocol potentially much more robust than current mechanisms.

The protocol is not, however, an automatic whitelisting or blacklisting system. It does not decide alone whether a bot should be allowed or blocked. It provides a reliable signal that administrators can rely on to make this decision.

Benefits beyond security

Web Bot Auth is about more than just blocking bad actors. It also provides better observability: site owners can get a clearer view of how automated agents interact with their content.

It also simplifies management at scale. Today, establishing trust with an automated service often requires a manual exchange of keys or configurations. Web Bot Auth makes this automatic and standardized processwhich reduces friction for everyone.

As Google mentions, as AI agents proliferate on the web, having a universal verification mechanism becomes a matter of critical infrastructure, not just security.

What this changes for site administrators now

Google specifies that the protocol is currently in experimental phase. This means that not all Google user agents use it yet, and that Google does not yet sign all of the requests sent by the agents who participate in it. Only a subset of traffic is affected, notably that of the Google Agent.

It is therefore imperative to do not rely exclusively on Web Bot Auth. A missing signature does not necessarily mean a bot is malicious. Traditional methods must remain in place in parallel for the duration of the progressive deployment.

For developers and system administrators who want to implement verification themselves, the steps are as follows:

1. Get the set of public keys from https://agent.bot.goog/.well-known/http-message-signatures-directory and cache them according to the Cache-Control header. Old keys missing from the file should be removed from the cache because they may be expired or revoked.
2. Verify that participating requests contain the Signature-Agent header with the value g=”https://agent.bot.goog”.
3. Validate the Signature header according to the specifications of the HTTP Message Signatures standard (RFC 9421), using the Signature and Signature-Input headers bearing the label g.
4. Keep the IP verification in parallel, because all requests are not yet signed.

Google also indicates that for latency-sensitive queriesit is possible to resend the response before signature validation is complete, and then apply possible sanctions to future requests from the caller.

A standard still under construction

Web Bot Auth is a proposal currently being developed within the IETF WBA working group. The protocol is subject to evolution, and Google emphasizes that it remains at an early stage, with an assessment still underway of the technical implications and impact on the ecosystem.

Actors wishing to follow the evolution of the standard can consult the work of the Web Bot Auth Working Group, contact their host to find out if they plan to support it (certain CDNs, WAFs and major bot detection services already do so), and provide feedback to Google via the official dedicated form.