UI Parity Validation Protocol

Sampling

• Quarterly, roughly 50 brand probes per cycle, drawn from the live probe pool and stratified across query intent the same way production polling is (branded, category, competitor, local, buyer-intent).
• Every sampled probe runs on both arms in the same 48-hour window, so model drift between arms is bounded.
• API arm — the standard production probe route: the engine's official API, run cold, model id recorded.
• UI arm — the same question asked manually, by a human, in the engine's consumer product.

UI-arm session rules

The UI arm is manual human use only — no automation, no scripted browsers, no scraping — and must not violate any product's terms of service.

• Fresh, logged-out sessions (clean browser profile, no cookies, no account) wherever the product allows anonymous use.
• Where a product requires login: a fresh account created for validation, with memory, personalization, and chat-history training turned off wherever the product exposes those controls. The session mode used is recorded per engine.
• No custom instructions, default model selection, one question per conversation — no follow-ups, since context accumulation is exactly the product-layer behavior being isolated.
• Responses are captured verbatim before the next question is asked.

Scoring and metrics

Both arms are scored with the same six judge dimensions used in production — accuracy, citation, sentiment, quality, recommendation, and competitive position — by the same agreement-validated judge pipeline. Per engine and dimension we report percent agreement (share of probes where both arms receive the same verdict) and Cohen's κ (chance-corrected agreement), each with its sample size. Cells with n < 20 are reported as directional only.

Publication

The full table and raw transcripts are retained internally as an auditable evidence trail. A summary — per-engine agreement and κ, sample sizes, and the validation date — is published on the methodology page each cycle.

When parity drops

1. Investigate a product-layer change first — consumer products ship retrieval, memory, and routing changes without notice; the affected probes are re-run to separate a one-off from a shift.
2. If the shift is real, the engine's fidelity tier on the methodology page is annotated or moved in the same release as the published results — the inventory never claims a parity the latest validation contradicts.
3. If reported statistics are affected, the change is recorded in the versioned methodology changelog and affected figures are restated.

Scope notes

• Engines are validated against their own consumer product (e.g. the OpenAI API arm vs the ChatGPT app). Meta AI is a named proxy — AITWIRE probes Llama, the model Meta AI is built on — so its parity result measures that proxy distance itself and is expected to be the widest.
• Google AI Overviews, Microsoft Copilot, and Apple Intelligence are not probed at the engine layer and are therefore not part of this validation.