What I learned from the Agent Web Compatibility Test
A website can be crawlable, indexable, server-rendered, and beautifully marked up; and still be unusable to an AI agent.
That is the finding I keep returning to after the first completed round of our Agent Web Compatibility Test, a controlled benchmark that measures what deployed AI runtimes can actually read, understand, and do on the web.
“AI-ready” is usually discussed as a property of a page. Add structured data, publish an llms.txt, render on the server, declare victory. But readiness does not live entirely on the website. An agent’s effective web capability depends on three things at once: the surface the publisher provides, the tools and policies the runtime exposes, and the retrieval path used in that specific session.
AI readability is a compatibility relationship, not a page attribute.
That is why we built a benchmark instead of another checklist. I’ve been investigating this topic for several years, and while many valuable auditing tools now exist, including Lighthouse’s new Agentic Browsing score: we found nothing comparable to our small benchmark, which focuses primarily on evaluating what AI agents can actually do at runtime.
Start with the scoreboard
As of July 27, 2026, the versioned dataset holds 17 runtime records and 159 observed runtime-capability pairs. Below are only the configurations that completed three attempts across all eight core checks. Each cell is passes out of three; the total is out of 24 opportunities.
| Runtime configuratio n | S01 | W01 | W02 | W03 | W04 | W05 | W06 | W07 | Total |
|---|---|---|---|---|---|---|---|---|---|
| Agent WordLift, Slack/NanoClaw | 3 | 3 | 2 | 3 | 3 | 3 | 3 | 3 | 23/24 |
| Antigravity agent client, Gemini 3.6 | 3 | 3 | 0 | 3 | 3 | 3 | 3 | 3 | 21/24 |
| ChatGPT Web, GPT 5.6 Sol Medium | 0 | 1 | 0 | 0 | 3 | 0 | 0 | 0 | 4/24 |
| Claude Web, Sonnet 5 Medium | 0 | 1 | 0 | 0 | 3 | 0 | 0 | 0 | 4/24 |
| Claude Web, Opus 5 High | 0 | 0 | 0 | 0 | 3 | 0 | 0 | 0 | 3/24 |
| Meta AI Web, Thinking/Free | 0 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 3/24 |
| Copilot Web, Smart | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0/24 |
| Gemini Web, 3.1 Pro | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0/24 |
| Perplexity Web, Auto/Free | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0/24 |
Read this carefully, because it is easy to misread. These are not intelligence scores and not a brand ranking. They describe what these exact configurations could prove, on this fixture, from Europe, on the recorded dates and plans.
Several low-scoring systems produced fluent, technically plausible, partially correct explanations. They failed because the deployed surface never exposed the required content or action path — or because the evidence never reached our origin.
The two strongest results came from tool-enabled agent runtimes, not from consumer chat surfaces. The lesson is not that those models are smarter.
It is that model quality and web capability are separate variables.
What we tested
Fixture 0.3.1 runs eight core capabilities. Two WebMCP checks are tracked separately because they require independent browser-host evidence.
| ID | Capability | What counts as a pass |
|---|---|---|
| S01 | JSON-LD structured data | Extract four exact values from a Product block, including a one-time identifier |
| W01 | Static HTML | Read an exact per-run token present in the initial HTML |
| W02 | JavaScript content | Execute the page script in the same-origin browser context and report the dynamic token |
| W03 | Cookie/session | Return an HttpOnly run cookie to the session endpoint |
| W04 | GET JSON | Call an availability endpoint for a synthetic blue-widget |
| W05 | POST JSON | Call the equivalent endpoint using POST |
| W06 | Semantic action | Place a reversible synthetic hold on the blue-widget |
| W07 | Enhanced entity page | Identify, interpret, and synthesize from a live enhanced entity page |
| M02 | Declarative WebMCP | Use a declaratively exposed web tool, with browser-host proof |
| M03 | Imperative WebMCP | Use an imperatively registered web tool, with browser-host proof |
The sequence is deliberate: read → render → maintain state → call → act → understand. A system that can quote visible prose cannot necessarily process structured data. A system that can issue a GET cannot necessarily hold a session or change state. And a system that can name an entity cannot necessarily connect its attributes, relationships, and provenance.
W07 exists because an earlier version of the test was too focused on access. Using a live Merkur enhanced entity page, the runtime must identify the NewsArticle, extract headline and date, identify a graph relationship, recover the external provenance URL, find an RDF alternate declared in the HTML, and produce a deterministic synthesis of those fields. Agents need stable identities, not isolated strings.
Eight things we learned
1. “Browser mode” may be a text fetcher
Several exports were labeled browser. The observed behaviour was closer to a GET-only text extractor: elements stripped, no JavaScript execution, no cookie persistence, no POST, long URLs rejected, no browser-native tool surface.
Product labels describe the user experience. They do not prove the model has a real browser.
2. Valid JSON-LD can be invisible to the model
Our S01 values sat in a valid Product JSON-LD block. Several extraction layers removed every element — including application/ld+json — before the page reached the active model. The structured data existed at the origin and was still unreachable.
Schema.org remains essential; search engines, enterprise retrieval systems, KG pipelines, and capable agents all use it. But for high-value entities, the same identifiers, attributes, relationships, and provenance should also have a coherent HTML representation.
3. GET access is not web agency
ChatGPT Web and both tested Claude configurations reached the GET availability endpoint in all three attempts. None of them preserved the session, called the POST equivalent, or placed the hold.
So if you expect a consumer assistant to check availability, status, price, inventory, or eligibility:
Do not require POST for a read-only check.
Expose it through a short, idempotent GET route. Keep POST for operations that genuinely change state, and give those a visible HTML form or an explicit agent tool.
There is a material difference between retrieving a document, calling a read-only endpoint, holding state, invoking a state-changing operation, and completing an action with verifiable origin confirmation. A single “web access” badge collapses all five.
4. Keep agent-critical URLs under 256 characters
One tested fetch path rejected our 277-character Merkur target URL as too long. A 244-character version was retrieved successfully.
The experiment does not reveal the fetcher’s documented maximum — only that the observed cutoff sits between 244 and 277. We use under 256 characters as a practical interoperability target, not as a standard. Short stable paths, no large opaque identifiers duplicated across slug and query string, concise resolver URLs for long entity IDs, compact run markers and signatures.
A URL that works in your browser can be rejected by an intermediary before the model ever sees it.
5. Signed-in does not mean session-capable
An account can be signed in to the AI product while its page fetcher stays stateless. Our HttpOnly cookie was not returned and the session endpoint answered 401. Website session continuity and AI-product authentication are separate capabilities.
6. A correct answer is not proof of a successful action
Assistants sometimes returned the right token or plausible entity facts while the origin recorded no accepted attestation, no marked retrieval, no POST, no state change. We saw it in both directions — including an assistant that reported a network block while our server had already logged PAGE_OPEN.
This is not deception. Their visibility into their own toolchain is incomplete. But for audits it means one thing: origin telemetry outranks retrospective narration.
7. Free-tier AI may inhabit a thinner web
This round did not include matched free-versus-paid pairs, so it cannot attribute any failure to account tier. (Perplexity Pro was unavailable and correctly remains not_run.)
It belongs beside a separate study of 56 ChatGPT enterprise SSE traces. Using model paths as a behavioural proxy, free-like traces searched the web in 10.8% of cases against 47.4% for paid-like traces, and produced parametric-only answers 32.4% of the time against 5.3%.
The lesson is not “free AI is worse.” It is that tier can change the evidence path. A cheaper experience may search less, verify less, and lean harder on compressed model memory. That gives publishers two simultaneous jobs: make the entity easy to remember through a clean, consistent public footprint, and make the evidence easy to verify through accessible HTML, structured data, entity pages, provenance, sitemaps, and a knowledge graph.
8. Capability and accountability move together
The Antigravity client scored 21/24 — and it is a fundamentally different environment from a chat page. Antigravity is locally installed and gives agents working environments, including browser and terminal access.
That is not simply “better.” It is a trade:
more capability in exchange for a larger operator-managed trust boundary.
Consumer products keep a restrictive policy boundary between model and open web because they must serve millions of users safely across unknown sites. Local and enterprise-managed agents push that boundary outward, and the operator inherits decisions about reachable files and credentials, execution location, approval gates, reversibility, and retained logs.
So the right question is not “which agent can do the most?” It is: which agent can complete the task with the right permissions, confirmations, isolation, and evidence?
9. MCP is the enterprise middle path
Almost no enterprise wants to replace Copilot or Claude to gain one domain capability. They want the assistant employees already use to reach governed knowledge and run approved workflows.
Model Context Protocol provides that bridge; an MCP server exposing resources, tools, and prompts to a compatible host. Microsoft documents the pattern in Copilot Studio; Anthropic introduced MCP as a standard for connecting assistants to external systems.
| Deployment pattern | Effective access | Primary trust owner | Main trade-off |
|---|---|---|---|
| Free consumer assistant | Search or sanitized fetch, often heavily constrained | AI provider | Broad accessibility, potentially thinner grounding |
| Hosted paid assistant | More retrieval or reasoning, still policy-constrained | Provider and customer | Better grounding ≠ agency |
| Local or managed agent | Browser, files, terminal, network, installed tools | User or enterprise | High capability, higher governance burden |
| Existing assistant plus MCP | Explicit domain resources and approved tools | Shared | Controlled specialization without replacing the assistant |
This is the practical case for Agent WordLift over MCP: authoritative entity and Knowledge Graph context, typed AI-visibility tools, deterministic inputs and outputs, bounded permissions, and origin-side telemetry — added to the assistant the organisation already approved. MCP does not grant Copilot or Claude unrestricted browser access, and it does not override host policy. It creates an explicit, governed path around the weakest parts of generic web retrieval.
One caveat I want on the record: the 23/24 Agent WordLift result was observed in its tested Slack/NanoClaw configuration. It is evidence for the execution-layer architecture, not a transferable score. Copilot-plus-Agent-WordLift and Claude-plus-Agent-WordLift each need to become named runtimes in the next round.
Model Context Protocol provides that bridge. An MCP server can expose resources, tools, and prompts to a compatible host. Microsoft documents this model directly in Copilot Studio, and Anthropic introduced MCP as a standard for connecting assistants such as Claude to external systems.
This explains the practical value of Agent WordLift through MCP. It can add a domain-specific execution layer to an existing assistant:
- authoritative entity and Knowledge Graph context;
- typed SEO and AI-visibility tools;
- deterministic inputs and outputs;
- authentication and bounded permissions;
- origin-side telemetry and reusable workflows.
MCP does not magically give Copilot or Claude unrestricted browser access, nor does it override the host product’s policies. It creates an explicit, governed path around the weakest parts of generic web retrieval.
The model remains the reasoning and conversation surface. Agent WordLift supplies the domain memory, tools, and execution contracts.
| Deployment pattern | Effective access | Primary trust owner | Main trade-off |
|---|---|---|---|
| Free consumer assistant | Search or sanitized fetch, often heavily constrained | AI provider | Broad accessibility, potentially thinner grounding |
| Hosted paid assistant | More retrieval or reasoning, still product-policy constrained | AI provider and customer | Better grounding does not guarantee agency |
| Local or managed agent | Browser, files, terminal, network, installed tools | User or enterprise | High capability with higher security and governance burden |
| Existing assistant plus MCP | Explicit domain resources and approved tools | Shared between host, enterprise, and tool provider | Controlled specialization without replacing the assistant |
How we made the benchmark hard to game
The core design decision was to score evidence, not prose claims.
Define the runtime precisely. Every observation records product, model or visible mode, interaction surface, plan, sign-in and connector state, region, date, runner, and attempt number. “Claude,” “Gemini,” or “ChatGPT” is not a unit of analysis; the same model behaves very differently across a chat app, a browser agent, a coding environment, and a tool-enabled client. Plan belongs in the label too.
Fresh run for every attempt. New run ID, nonce, cookie, static token, dynamic token, and verification values, so nothing passes by replaying an earlier answer. Three attempts per runtime, reported as pass counts rather than a best-of anecdote.
Origin as the authority. The server records page opens, session requests, GET and POST calls, hold state changes, exact-value attestations, browser-render events, and diagnostic WebMCP claims. The final JSON export is canonical. This caught several convincing false positives.
False positives cost more than passes. Client-supplied WebMCP labels turned out to be replayable, so they became diagnostic only. Direct hits on the dynamic-token endpoint could imitate rendering, so W02 now requires a same-origin render event. A long URL could satisfy an identity check by substring accident, so attestation became boundary-aware. The agent uses awct_run while the grader uses a separate awct_gold marker.
Separate failure from absence of evidence. We keep four states apart: not_run (no valid attempt), blocked (a constraint prevented it), failed (attempted, wrong result), unverified (plausible result, insufficient evidence). One pre-execution safety refusal was recorded as an access barrier, not a zero.
On caching: it is a plausible source of variability, but our data does not show it caused any specific failure. Repeated identical answers can equally come from a stable extractor, a search index, a blocked origin, or reused session context. “Cache-sensitive” is a valid hypothesis; “failed because of caching” is not a supported conclusion. A dedicated freshness experiment — controlled mutations between attempts, cold-versus-warm sequences, matched tiers, Age/ETag/Cache-Control/Vary evidence — is on the roadmap.
You do not need our fixture to reuse the method. Build one deterministic check per layer, bind every run to a fresh nonce with the answer key server-side, instrument the origin, and repeat instead of demoing.
What web teams should do now
Make the initial HTML meaningful. Critical names, descriptions, identifiers, prices, availability, dates, and relationships belong in server-rendered or statically generated HTML. Meta AI read static HTML in all three attempts while failing every other check — static HTML is still the safest shared interface. Progressive enhancement is an agent-compatibility strategy.
Keep JSON-LD aligned with visible content. Stable @id values, explicit types, typed relationships, provenance. Structured data must never describe a different entity or state than the page.
Publish enhanced entity representations for strategic entities: canonical identifier, essential attributes, relationships to other entities, source and provenance links, machine-readable alternates, and enough visible context for an agent to synthesise.
Separate read checks from state-changing actions. Short idempotent GET routes under 256 characters. POST only for real state changes — clear, bounded, reversible, deterministic in confirmation, with an HTML form fallback and WebMCP as progressive enhancement.
On that last point: WebMCP needs a maturity ladder, because “detected” tells you almost nothing. None → declared → parseable → callable → agent-proven. A static audit reliably measures the first three. Only a compatibility fixture reaches the last two — and no M02/M03 observation earned independent browser-host proof this round. That does not invalidate WebMCP. It shows why declaring a tool and proving deployed support must stay separate claims.
Instrument before optimising. Without origin telemetry you cannot tell whether an agent failed to discover the page, failed to receive its content, misunderstood it, or lacked permission to act.
Test the runtimes your users actually use. Do not extrapolate from an API model evaluation to a consumer product. Audit the exact products, plans, regions, and connector states that matter to the business.
How this connects to the WordLift AI Audit
The benchmark gives the AI Audit a clearer architecture — four layers, four kinds of evidence.
| Layer | Question | Evidence | Output |
|---|---|---|---|
| Static readiness | Has the publisher created an AI-readable surface? | HTML, structured data, rendering strategy, directives, entity signals, tool declarations | Readiness findings and prioritised fixes |
| Runtime compatibility | Can a named deployed runtime use that surface? | Fresh three-run capability fixture | Runtime-capability matrix |
| Origin verification | What can the publisher prove happened? | Requests, render events, attestations, state changes, target-origin logs | Auditable evidence trail |
| Governed execution | Can the enterprise’s own assistant use approved knowledge and actions? | MCP resources, typed tools, authentication, permissions, execution logs | A bounded Agent WordLift capability layer |
The public AI Audit stays the entry point: fast, repeatable analysis of a page’s readiness. The Agent Web Compatibility Test is the research and validation layer that tells us whether those readiness signals translate into behaviour.
Together they support a more honest progression:
present → parseable → retrievable → understandable → actionable → agent-proven
Which changes the quality of the advice. Instead of “add structured data,” the Audit can say which entity facts are missing, whether they are also in HTML, and which tested runtime paths are likely to consume them. Instead of “WebMCP detected,” it can distinguish declared from callable from agent-proven. Instead of “AI agents can access this page,” it can name the evidence boundary: static potential, observed retrieval, verified comprehension, or confirmed action.
Limitations
This is a dated snapshot, not a verdict.
- Tests ran from Europe on July 25–27, 2026.
- Consumer runtimes are opaque; we cannot always separate product policy, fetcher design, model behaviour, and caching.
- No matched free-versus-paid configurations, so tier cannot be established as a cause.
- Configurations with only one or two observations are excluded from the completed-series table.
- Perplexity Pro was not tested. One pre-execution safety refusal is recorded as
not_run. - WebMCP remains a proposed technology and deployed browser-agent support is still evolving.
- The Agent WordLift 23/24 applies to the tested Slack/NanoClaw runtime only.
- The fixture measures deterministic capability tasks, not answer quality across the open web.
Which is exactly why it should be rerun. Our proposed cadence: a short monthly journey, a full quarterly suite, plus extra runs after major product or runtime changes.
The broader lesson
For years, web visibility meant being crawled and ranked. The agentic web adds a longer chain:
discover → retrieve → parse → identify → understand → act → verify
Most audits stop near the beginning of that chain. Most product demos show one successful run near the end. We need both views; the publisher’s and the runtime’s; with deterministic tasks, repeated attempts, and evidence from the system where the action supposedly happened.
The web does not become agent-ready because a page contains an AI-related file, or because an assistant summarises it once.
It becomes agent-ready when knowledge is clearly represented, actions are safely exposed, multiple runtimes can use them, and the origin can prove what happened.
That is the standard we want the AI Audit to help developers reach.