Measurement Methodology

The AITWIRE Assurance Loop

The loop is how a claim earns the right to be reported. Each stage feeds the next, and prove feeds the next probe — measurement never stops:

1. Probe. We ask the major AI answer engines controlled questions ("polls") about you — run "cold," and repeated across phrasings, engines, and cycles.
2. Judge. Each response is scored against facts you've confirmed, and the scorer itself is checked against human-confirmed ground truth (agreement-validated).
3. Fix. Measured gaps become prioritized recommendations and data-backed campaign briefs. AITWIRE directs and proves; it does not write your prose.
4. Re-measure. The same metric, before and after, on an escalating schedule.
5. Prove. We report a movement as real only when it clears the statistics (below) — and, where the content is exclusive to a surface we operate, a controlled retrieval test can distinguish genuine retrieval from chance, with false-positive controls.

How we measure

AITWIREsends controlled questions ("polls," also called probes) to the major AI answer engines and scores each response across quality dimensions (accuracy, citation, sentiment, quality, recommendation) and signal categories.

• Polls are run "cold" — no personalization, account history, or AITWIRE context is injected — at low randomness, and repeated across phrasings, engines, and cycles.
• This measures the default answer an un-personalized user is likely to get, as a sample over many queries — not a single check.
• Each response is graded, and results are aggregated per dimension with a sample size and an interval (below).

What we measure — and what we don't

AITWIRE measures the engine layer: each AI system is probed through its official, publicly available API, so a score reflects the model's knowledge — plus, where the engine retrieves live, what its retrieval stack returns. The engine layer is the substrate every consumer surface is built on, and it is the layer that can be measured honestly: probes are reproducible (the same question can be re-asked and re-scored), versioned (the resolved model id is recorded with every poll), and run within the engines' published API terms. Consumer applications built on those engines additionally layer personalization, conversation memory, and product behavior that varies per user and per session — a layer that cannot be sampled reproducibly, so we do not claim to measure it.

For the search-grounded engines, the overlap between API and product is direct: ChatGPT Search and Grok's live web search are probed through the same retrieval stacks that power the consumer products, and Perplexity's API is the product.

How faithful each probe is to the consumer product built on it — for the 9 probed AI systems covering 90%+ of AI-answer traffic:

Grok appears twice because it is probed on both routes — its base model and its live web-search variant. It is one engine, not two.

• Google AI Overviews is not probed. It is tracked through Google Search Console impressions, which Google reports blended with regular search — so it carries presence data, not per-answer scores.
• Microsoft Copilot and Apple Intelligence are not probed, and we do not claim coverage of them.

Where the consumer product itself needs checking, we run UI parity spot-checks: a sample of live probes is re-asked manually, by a human, in fresh logged-out consumer sessions and scored on the same dimensions, so the agreement between the engine layer and the product layer is itself measured. Spot-checks are available per Assurance engagement, and quarterly validation results will be published on this page as they land. The protocol is public: UI Parity Validation Protocol.

Precision — what "high confidence" means

Every dimension score is reported with its sample size (n) and a 95% confidence interval. We label a dimension "high confidence" only when n ≥ 30 and the 95% interval is within ±7 points.

Because the margin narrows with the square root of n, a noisy (near 50/50) metric typically needs roughly 150–200 pollsto reach ±7 points — so "high" usually reflects far more than 30 polls. We always show the actual n and interval, not just the label. Confidence here is the statistical precision of the sample — never certainty.

Validity — the limits the interval does not capture

A tight interval tells you the sample is precise. It does not, by itself, tell you the measured value is true. The honest caveats:

• Scoring can err. Response grading uses automated and machine-learning classifiers, which can mis-grade. We mitigate with independent judge cross-checks and inter-rater agreement — a standard measure (Cohen's κ) of how well the scorer matches human-confirmed ground truth — but do not eliminate error. A tight interval around a mis-scored value is "confidently wrong."
• Samples are not fully independent. Repeated, similar prompts to the same model are correlated, so the effective sample is smaller than the raw count and true intervals can be modestly wider than computed. We discount for this correlation (a standard design-effect adjustment) so confidence is earned, not over-counted.
• Scope. We measure un-personalized, point-in-time responses across a selected set of engines — not every personalized answer, every phrasing, every surface, or future model states. AI systems also drift as their models change.

How we keep the numbers honest

Beyond a precise interval, we apply controls so a number is only reported as meaningful when it earns it:

• Statistical power. When a sample is too small to detect a meaningful change, a non-result is reported as "inconclusive," not "flat" — absence of evidence is not evidence of absence.
• Multiple-comparison control. We test several dimensions at once, so we apply a standard false-discovery-rate correction (Benjamini–Hochberg): a movement is "confirmed" only if it survives correction, otherwise it is flagged "exploratory."
• Representativeness. We stratify polls by query intent (branded, category, competitor, local, buyer-intent) and equal-weight the covered strata, so the headline is not dominated by whichever intent was polled most — and we show the gap versus a naive average.
• Citation integrity. A brand mention is not proof. Only a cited source that actually substantiates the answer counts toward the evidence rate; stale, third-party, competitor, or unresolved citations are labelled, not counted.
• Change vs. model drift. AI engines move on their own. We measure and control for model-wide volatility and report your lift net of it — downgrading attribution to inconclusive when the model itself is too volatile.
• Auditability. Every poll stores a reproducible evidence record — the scorer and rubric versions, the resolved model, and the market — so a historical score can be re-checked under today's methodology.

Evidence you can defend

The point of the loop is not a score — it is a record. Behind every number sits an evidence trail you can inspect and export, not a claim you have to take on faith:

• A reproducible record per measurement — the scorer and rubric versions, the resolved model, and the market — so any historical score can be re-checked under today's methodology.
• An append-only, tamper-evident action log. Privileged actions — measurement runs, verdict decisions, publishes, and account changes — are recorded in a hash-chained log whose integrity can be re-verified on demand. (Append-only and tamper-evident, not "immutable": breaks are detectable, and the chain evidences the record.)
• Exportable on Custom plans. Custom-tier workspaces can export their full evidence trail as CSV or JSON — every measurement, verdict decision, signal, publish, and account change — for a compliance or audit review.

What we disclose — and what we protect

We publish our framework so you can judge it. We do not publish the implementation that makes it work — both to keep the measurement hard to game and because parts of it are the subject of pending intellectual property.

How to read the numbers

• Trust the trend and relative comparison more than any single absolute number.
• Use the confidence label and interval to decide which dimensions are reliable enough to act on; treat "insufficient" or "low" dimensions as directional only.
• The strongest evidence is measured lift — the same metric before and after a change — together with citation delivery, i.e. when an engine actually quotes your published source.

What we do not claim

• We do not claim to read every user's personalized answer.
• We do not guarantee that any AI system will adopt, cite, or correctly interpret your information.
• We do not hold any third-party security certification we have not earned; we state a compliance status only when we can substantiate it.
• Analytics are informational and should not be the sole basis for legal, financial, medical, or compliance decisions.

Versioning

This methodology is versioned. When we change how scores are computed, we restate affected figures and note the change, so period-over-period comparisons stay honest. See the AITWIRE Terms of Service for the legal terms that govern measurement and estimates.