Google Published AI Guidance. The Observability Gap Still Remains.

Yesterday Google published their official guide to optimising for generative AI features in Google Search. It is worth reading — not because it changes how AI systems work, but because of what it reveals about what the industry still does not understand.

The guide confirms three things that should have been obvious: AI features depend on crawlability, semantic structure, and content type. It dismisses several things that were probably never real: llms.txt files, content chunking, and rewriting content specifically for AI audiences. And it quietly acknowledges something that almost nobody is building infrastructure for — agentic systems that interact with your content directly, inspecting DOM structure and accessibility trees.

On llms.txt specifically, Google is unambiguous: “You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them” — and creating them “won't harm (nor help).” “Structured data isn't required for generative AI search.” An Ahrefs study of 137,210 domains (May 2026) confirms the picture empirically: 97% of llms.txt files received zero requests from any AI system. Of the 3% that did receive traffic, AI search and retrieval bots represented just 1.1% of those requests. The file is not being read.

So far, sensible. But here is what the document does not address.

The guide covers one surface.

Google's AI Overviews and AI Mode use retrieval-augmented generation from Google's own search index. That is a well-defined system with documented crawlers, known quality signals, and established ranking infrastructure. Following Google's guidance should improve your position on Google's AI surface. That statement is probably true.

But there are other surfaces.

GPTBot indexes content for OpenAI's systems. ClaudeBot accesses content for Anthropic. PerplexityBot runs its own retrieval layer. Grok retrieves real-time data for X. Amazonbot, Google-Extended operating independently of standard search, CCBot — each represents a separate system with its own retrieval patterns, access behaviour, and interpretation logic.

Google's guide does not cover these. It cannot. That is not a criticism — it would be like asking Google Maps to describe Apple Maps' routing algorithm. Different systems. Different infrastructure. Different classifications. The guide is accurate about what it covers. The problem is what it leaves out.

Optimisation is not the same as verification.

The guide tells you what structural conditions AI systems prefer. It does not tell you whether those conditions are present on your site. It does not tell you whether automated systems are actually accessing your content, extracting it, or classifying it by content type.

Those are different questions requiring a different instrument.

A site that follows every guideline in Google's document and a site that ignores all of it look identical from the outside. Both lack participation data. Both are operating without observability. The guidance creates no feedback loop. You follow it and you wait, which is not a verification strategy — it's an assumption.

This is the gap. Not an optimisation gap. An observability gap.

Search systems expose indexing. They do not expose participation conditions.

What Google confirmed without knowing it.

The three structural dimensions Google's guide describes — crawlability, semantic legibility, and content classification — map closely to the three structural gates aater.ai evaluates.

Gate 1 (Reachability): Can automated systems access the content? Gate 2 (Legibility): Can they extract meaningful structure from it? Gate 3 (Content Type): is it informational or commercial? These three gates are checkable structural facts — the free baseline. We no longer collapse them into a single score: structure tells you what AI can access and read, not whether it will cite you.

Google's guide did not validate these gates by design. Both were independently derived from the same underlying reality: the structural conditions that determine whether automated systems can access, understand, and cite content. When two independently developed frameworks converge on the same three dimensions, that convergence is evidence about the domain, not about the frameworks.

aater.ai currently has over 100 public domains checked, with structural gates resolving continuously. Early distributions suggest search performance and structural reachability frequently diverge. Domains with strong organic rankings are not systematically positioned for automated extraction. These are independent systems resolving against independent structural conditions.

As automated systems fragment across search engines, model providers, browser agents, enterprise retrieval pipelines, and autonomous workflows, publisher-side observability becomes increasingly difficult to infer from any single surface. Google's guide is authoritative for Google's surface. It describes nothing about the others. That fragmentation is not a temporary state — it is the structural condition of the post-search web.

What this changes.

Google just normalized AI structural readiness as an operational concern. They commoditised optimisation advice in the same move. They exposed the measurement vacuum. That sequence creates the right conditions for an observability layer to be necessary, not merely useful.

Sites will follow the guidance. Agencies will tell clients they have done the work. And nobody will have any way to verify whether participation conditions actually changed — across Google's systems or anyone else's. That is the state of the market as of today.

aater.ai was built to measure what Google's guide cannot verify. Not whether you optimised. Whether the structural conditions that determine AI participation are present — and whether the automated systems that process your content are actually operating on it.

The observability gap is not closing because someone published a guide. It closes when you can see what is actually happening.