Measurement Methodology

The AI Answer Assurance Loop

The loop is how a claim earns the right to be reported. Each stage feeds the next, and prove feeds the next probe — measurement never stops:

1. Probe. We draft controlled prompts; you approve them, and we send them to eight leading public AI systems and answer surfaces on the schedule you choose — run "cold," and repeated across phrasings, engines, and cycles.
2. Judge. Each response is scored against facts you've confirmed, and the scorer itself is checked against human-confirmed ground truth (agreement-validated).
3. Calibrate. Measured gaps become prioritized recommendations and data-backed campaign briefs, which you approve before anything is published. AITWIRE directs and proves; it does not write your prose, and it does not change what an AI system says.
4. Re-measure. The same metric, before and after, re-asking the prompts you approved on an escalating schedule.
5. Prove. We report a movement as real only when it clears the statistics (below) — and, where the content is exclusive to a surface we operate, a controlled retrieval test can distinguish genuine retrieval from chance, with false-positive controls.

These five are the operational view of the three stages named everywhere else on this site: Measure (probe and judge), Calibrate, Prove (re-measure and prove). Same loop, same order — this page splits it where the statistics change.

How we measure

AITWIRE sends the controlled prompts ("probes") you approve to eight leading public AI systems and answer surfaces, on the schedule you choose, and scores each response across quality dimensions (accuracy, citation, sentiment, quality, recommendation) and signal categories.

• Probes are run "cold" — no personalization, account history, or AITWIRE context is injected — at low randomness, and repeated across phrasings, engines, and cycles.
• This measures a controlled, unpersonalized engine-layer baseline. Consumer products may add retrieval, system behaviour, personalization and interface-specific processing, so AITWIRE does not claim that an API probe reproduces every consumer-session answer.
• Each response is graded, and results are aggregated per dimension with a sample size and an interval (below).

Judged by a panel — and calibrated by humans

Grading is not one model's opinion. Each response is scored by a panel of independent quality-control judges; where they disagree, the result is held back rather than averaged into a false-confident number. This is the calibrate step between measure and prove, and it has two human checkpoints:

• You calibrate the ground truth. Accuracy is graded against the facts you confirm about your own business — your account's verified canon — so a score measures agreement with what you've verified, not against a guess. AITWIRE suggests; you approve.
• Humans calibrate the judges. On a rolling sample, trained reviewers score responses by hand, and an automated judge is trusted only where it agrees with those humans at a standard inter-rater bar (Cohen's κ). When agreement slips, the affected scores fall back to directional until the judge is re-tuned.

We publish that this calibration happens and the standard measure we hold it to. We do not publish the rubric text, the dimension weightings, or the agreement thresholds themselves — those are what keep the measurement hard to game.

What we measure — and what we don't

AITWIRE measures the engine layer: each AI system is probed through its official, publicly available API — except the one row the fidelity table below declares a named proxy, which names the model we actually run, so a score reflects the model's knowledge — plus, where the engine retrieves live, what its retrieval stack returns. The engine layer is the substrate every consumer surface is built on, and it is the layer that can be measured honestly: probes are reproducible (the same prompt can be re-run and re-scored), versioned (the resolved model id is recorded with every probe), and run within the engines' published API terms. Consumer applications built on those engines additionally layer personalization, conversation memory, and product behavior that varies per user and per session — so we do not claim that an API probe reproduces every consumer-session answer. Where a consumer product's displayed answer is observed, it is a captured surface (below): a record of what the product showed at a particular moment, reported separately — never as the engine baseline. Today consumer-surface capture runs for the AITWIRE account only, never for a customer account: Google AI Overviews is captured since 15 October 2026, and Copilot Search is not captured.

For the search-grounded engines, the overlap between API and product is direct: ChatGPT Search and Grok's live web search are probed through the same retrieval stacks that power the consumer products, and Perplexity's API is the product.

How faithful each measurement is to the consumer product built on it, across the public AI systems and answer surfaces we cover:

Grok appears twice because it is probed on both routes — its base model and its live web-search variant. It is one engine, not two.

• Google AI Overviews is not measured for customer accounts. Since 15 October 2026 AITWIRE captures the overview itself off the search results page and scores it like any other answer, for the AITWIRE account only (a first period ran from 4 August to 20 September 2026); see how we capture Google AI Overviews below. Search Console impressions are a separate, optional attribution signal about traffic to your pages — they are not an answer measurement. Since 31 August 2026, Search Console's generative-AI report shows AI Overviews and AI Mode impressions separately, by page, country, date and device (impressions only: no clicks or search terms). It is inside Search Console; AITWIRE can't read it automatically today, and it still can't tell you what an overview said.
• Microsoft Copilot — two different things, and we measure neither today. The Copilot chat app (copilot.microsoft.com) is not probed: Microsoft exposes no public API that returns its grounded answer, so there is no faithful way to measure it and we claim no coverage of it. Copilot Search — the AI answer Bing shows on its own results page — was captured off the page until 25 September 2026, when that capture was retired; it is not measured today, and the chat app never was. Where Copilot cites or crawls your content we also track that passively (click-throughs and Bing/bingbot crawler visits) — passive tracking never enters an accuracy score.
• Apple Intelligence is not probed, and we do not claim coverage of it.

Where the consumer product itself needs checking, we run UI parity spot-checks: a sample of live probes is re-run manually, by a human, in fresh logged-out consumer sessions and scored on the same dimensions, so the agreement between the engine layer and the product layer is itself measured. Spot-checks are available in a scoped engagement, and quarterly validation results will be published on this page as they land. The protocol is public: UI Parity Validation Protocol.

Two kinds of engine — and why it changes what's fixable

The engines we probe answer in one of two modes, and we report accuracy by engine class because the fix is different for each. (Captured search surfaces are a third class, reported separately — they are not probes. Today they run for the AITWIRE account only.)

• Grounded engines retrieve live from the web before they answer (for example ChatGPT Search, Perplexity, Grok's web search). What they say responds to the authoritative sources you publish — so a grounded gap is usually movable.
• Training-prior engines answer from what the model already learned in training, with no live lookup. A gap there reflects the model's frozen knowledge — it typically shifts only when the model is retrained, so we label it rather than promise a quick fix.

Splitting the score this way keeps the promise honest: it separates what published facts can move now from what only a future model version will. Each class is reported with its own sample size and interval.

Precision — what "high confidence" means

Every dimension score is reported with its sample size (n) and a 95% confidence interval. We label a dimension "high confidence" only when n ≥ 30 and the 95% interval is within ±7 points.

Because the margin narrows with the square root of n, a noisy (near 50/50) metric typically needs roughly 150–200 probes to reach ±7 points — so "high" usually reflects far more than 30 probes. We always show the actual n and interval, not just the label. Confidence here is the statistical precision of the sample — never certainty.

Validity — the limits the interval does not capture

A tight interval tells you the sample is precise. It does not, by itself, tell you the measured value is true. The honest caveats:

• Scoring can err. Response grading uses automated and machine-learning classifiers, which can mis-grade. We mitigate with independent judge cross-checks and inter-rater agreement — a standard measure (Cohen's κ) of how well the scorer matches human-confirmed ground truth — but do not eliminate error. A tight interval around a mis-scored value is "confidently wrong."
• Samples are not fully independent. Repeated, similar prompts to the same model are correlated, so the effective sample is smaller than the raw count and true intervals can be modestly wider than computed. We discount for this correlation (a standard design-effect adjustment) so confidence is earned, not over-counted.
• Scope. We measure un-personalized, point-in-time responses across a selected set of engines — not every personalized answer, every phrasing, every surface, or future model states. AI systems also drift as their models change.

How we keep the numbers honest

Beyond a precise interval, we apply controls so a number is only reported as meaningful when it earns it:

• Statistical power. When a sample is too small to detect a meaningful change, a non-result is reported as "inconclusive," not "flat" — absence of evidence is not evidence of absence.
• Multiple-comparison control. We test several dimensions at once, so we apply a standard false-discovery-rate correction (Benjamini–Hochberg): a movement is "confirmed" only if it survives correction, otherwise it is flagged "exploratory."
• Representativeness. We stratify probes by query intent (branded, category, competitor, local, buyer-intent) and equal-weight the covered strata, so the headline is not dominated by whichever intent was probed most — and we show the gap versus a naive average.
• Citation integrity. A brand mention is not proof. Only a cited source that actually substantiates the answer counts toward the evidence rate; stale, third-party, competitor, or unresolved citations are labelled, not counted.
• Change vs. model drift. AI engines move on their own. We measure and control for model-wide volatility and report your lift net of it — downgrading attribution to inconclusive when the model itself is too volatile.
• Auditability. Every probe stores a reproducible evidence record — the scorer and rubric versions, the resolved model, and the market — so a historical score can be re-checked under today's methodology.

Evidence you can defend

The point of the loop is not a score — it is a record. Behind every number sits an evidence trail you can inspect and export, not a claim you have to take on faith:

• A reproducible record per measurement — the scorer and rubric versions, the resolved model, and the market — so any historical score can be re-checked under today's methodology.
• An append-only, tamper-evident action log. Privileged actions — measurement runs, verdict decisions, publishes, and account changes — are recorded in a hash-chained log whose integrity can be re-verified on demand. (Append-only and tamper-evident, not "immutable": breaks are detectable, and the chain evidences the record.)
• Exportable in a scoped engagement. Workspaces in a scoped engagement can export their full evidence trail as CSV or JSON — every measurement, verdict decision, signal, publish, and account change — for a compliance or audit review.

What we disclose — and what we protect

We publish our framework so you can judge it. We do not publish the implementation that makes it work — both to keep the measurement hard to game and because parts of it are the subject of pending intellectual property.

How to read the numbers

• Trust the trend and relative comparison more than any single absolute number.
• Use the confidence label and interval to decide which dimensions are reliable enough to act on; treat "insufficient" or "low" dimensions as directional only.
• The strongest evidence is re-measurement — the same prompts, before and after a change, the delta reported with its interval and called real only when it clears the statistics — together with citation delivery, i.e. when an engine actually quotes your published source.

What we do not claim

• We do not claim to read every user's personalized answer.
• We do not guarantee that any AI system will adopt, cite, or correctly interpret your information.
• We do not hold any third-party security certification we have not earned; we state a compliance status only when we can substantiate it.
• Analytics are informational and should not be the sole basis for legal, financial, medical, or compliance decisions.

Versioning

This methodology is versioned. When we change how scores are computed, we restate affected figures and note the change, so period-over-period comparisons stay honest. See the AITWIRE Terms of Service for the legal terms that govern measurement and estimates.