# How we measure accuracy

**The headline result:** on a published research study of seven commercial AI agents and consenting people, the live classifier (heuristic-v4.33) caught **98.3% of AI agent sessions** (1,386 of 1,410: 1,178 at high confidence, 208 at medium) and **wrongly flagged 0 of 159 people** as AI agents. That study also informed the rules, so it shows how well they fit it, not an independent estimate. The full figures, per agent and with intervals, are in [the accuracy report](/accuracy) (opens the public report page).

**How sure are these verdicts?** Verdicts are categories from observed behaviour, not probabilities. An agent that drives the browser like a person can be read as Human, and a session that goes quiet becomes Unknown (no recent activity); Unknown is never counted as human. The rest of this page explains how we check how often each verdict is right, so you can judge it for your own traffic.

**What a verdict may do.** The behaviour rules have been checked only on research recordings, which contain no assistive-technology users, and not yet on test runs of real agents. So an AI agent verdict on one behaviour pattern alone is **medium confidence**: held for review at a protected action by default (the site owner may switch it to block, after a warning about real people such as assistive-technology users) and never charged. **High confidence** (verified identity, an automation trace, injected input, sustained behaviour, a behaviour change, or two or more independent behaviour families agreeing on a session with no physical-device input) follows the AI agent rule in full and is chargeable. The full table is in [Verdicts](/docs/verdicts#what-backs-an-ai-agent-verdict-confidence-bands).

**What the accuracy report is built from.** The public report is measured on the FP-Agent research study (seven commercial agents doing the same shopping, flight-booking and forum tasks, and consenting people), plus the adversarial bench below. It says plainly what isn't covered yet: real named-agent runs in a normal browser and assistive-technology recordings are still pending.

## Test-set evaluation

Every labelled corpus is split into a tuning set and a test set by a salted hash of its group (a person, an agent trial or a run), never by hand. The test-set membership was frozen on 2026-11-03 with a SHA-256 of the sorted test group IDs, and our test suite fails if it changes. Test rows are never used to choose or tune a rule or model.

Some corpora were already looked at, test part included, while the current rules were chosen. The FP-Agent research recordings are the main one: rules heuristic-v4.18 to v4.25 were picked after viewing all 7,728 sessions. Figures from those corpora are labelled **tuned on this data** wherever they appear. For example, "7,025 of 7,182 agents, 0 of 546 people" is a whole-dataset, tuned figure, not an accuracy estimate. Only corpora collected after the freeze give independent numbers:

| Corpus | Set aside for testing | Status |
| --- | --- | --- |
| FP-Agent research recordings (outside the repository) | the frozen test split, 20% of participants and trials | tuned on this data |
| Recorded browser agents, user-confirmed human recordings, start-up smoke corpus, SapiMouse | 30% of recordings or subjects by hash | tuned on this data |
| Adversarial humaniser agents (headless bench) | seeds 101-105 of each kind | independent (scripted agents) |
| Real named-agent runs, OS-level runs, assistive-technology recordings | every row (append-only) | independent, collection pending |

The report gives, for every test corpus and agent family: how many agents read AI agent, Unknown and Human (with ever-Human where the replay can tell), the human false-positive rate (people read as AI agent), the Unknown share, and what protected actions would see, each with a Wilson 95% interval. Corpora are never pooled into one figure. Assistive-technology sessions follow the [assistive-technology recording protocol](/docs/accuracy-assistive-protocol).

## Labelled sessions

Detection is measured on labelled sessions: sessions where someone knows who drove them. This page describes where those labels come from, how they are kept, and how the report is computed. The report never gives a single "accuracy %": that one number would mix classes and label sources whose proportions say nothing about the traffic your site sees.

## Labels: who drove the session

A label says who drove one whole session: **human** (a person, no AI or automation), **agent** (an AI agent or automation; the dashboard category "AI agent"), or **mixed** (a person and an AI agent took turns in the same session). Every label also records its **source**, and sources are reported separately, never pooled:

| Source | Where it comes from | How much to trust it |
| --- | --- | --- |
| Controlled | The operator or a harness chose the driver (for example Playwright, or Claude in Chrome started by the operator). Records the driver kind. | Ground truth for which driver ran. |
| Reviewer | Someone watched the session, or the participant told them. | As good as the observation. |
| Self-declared | The person who drove the session said so (labels recorded before 2026-11-03 on the live demo, or by a team's own tooling). | Weakest: an agent can give the answer too. |

A label is stored with the session's existing evidence (interaction timing and automation markers already collected, never typed content, page paths or identity) for 30 days, then deleted with the other evaluation samples. Erasing the session erases its labels; withdrawing consent (Stop observing on the demo) discards a self-declared label.

### Self-declared labels

The live demo no longer asks visitors to label their session. Self-declared labels collected earlier stay with their sessions under the same 30-day rule; they are reported apart from controlled and reviewer labels and never pooled with them.


### Labelling an agent run yourself

Agents that a person has to start (Claude in Chrome, ChatGPT agent, Comet, Cowork) follow a manual protocol:

1. Open a demo task, for example `https://app.trytelltale.dev/try/booking`, in the agent's browser. Use **Compare browsers** to make a comparison link if you want several agents on one task.
2. Give the agent the prompt from **Let an AI agent do it** and let it finish without touching the page. If you step in, the session is **mixed**.
3. In the dashboard, open the session (Sessions, the newest one on the Public demo site), then Technical › **Help improve detection** › **Who drove this session**: choose *An AI agent or automation*, source *Controlled run*, driver kind such as `claude-in-chrome`, `chatgpt-agent`, `comet` or `cowork`, and one run code for all sessions of that sitting. Tick the confirmation and save.

## Your own evaluation report (dashboard)

Separate from the public accuracy report, the dashboard computes the same figures on sessions your team labelled: Developer › Detection evaluation › **Report by label source**, or `GET /api/admin/evaluation-report?site=…` (dashboard only).

For every model (the live classifier and every shadow model in the build; shadow models are scored only and never change the verdict your site receives) and every source, the report gives a confusion matrix with counts and:

- **Human false-positive rate**: sessions labelled human that the model called AI agent, over all sessions labelled human.
- **Agent recall**: sessions labelled agent that the model called AI agent.
- **Unknown rates** for human and agent labels. Unknown is not human.
- **Agent labels called human** under the model’s stated prediction scope, and for mixed labels, how many were called agent.
- **Agent labels ever called human**: agent sessions the model called human at any evaluated moment, even if it said agent later. A site acting on the verdict treated those agents as people for a while.
- **Agent time read as human**: of the evaluated moments on agent labels, the share the model called human. Moments in one session are correlated, so no interval is given.
- **Strong-flag false positives**: human labels whose own evidence carries a strong flag (the conflicts below) that the model called agent, over all human labels including them. If those labels are right, these are the false positives the strong-flag rules cause.

Each rate shows count / denominator and a Wilson 95% interval. Sessions of one run are correlated, so the interval is optimistic when there are few groups; the group counts are shown beside it.

The live model and v5 evaluate the **final saved snapshot**, not the whole session. Historical ever-human, time-share and latency metrics are unavailable when the required input history was not saved. Other event-only models summarize the evaluated retained receipt times: agent if any evaluated moment was agent, otherwise human if any was human, otherwise unknown. A signature can force AI agent only with retained verification and expiry times valid at the evaluated moment; an old signed Boolean alone is insufficient.

**Offline snapshot latency coverage.** The JSON evaluation report counts all samples, eligible snapshots, resolved snapshots and snapshots that stayed unresolved. Its latency percentiles describe only resolved snapshots; they are not a success rate. Snapshots with missing feature history, no timestamped interaction, or more than 500 distinct receipt times are excluded from latency measurement and counted separately. A bounded replay of the end of a long history cannot establish its first detection.

**Test.** About 30% of groups (a controlled run, or one session when there is no run) are set aside for testing, chosen by a hash of the group, never by hand. The report shows test groups and all groups separately; model changes should be judged on test groups. Imported evidence or connected groups that cross partitions stay visible in all-groups results but are excluded from test metrics and listed as partition conflicts.

**Conflicts.** A human label whose own evidence carries strong automation (`navigator.webdriver`, a headless user agent, an automation framework artifact, or a verified signature) is listed as a conflict and left out of the human row, and counted in the strong-flag false-positive figure. It is never dropped silently.

**Same classifier as live.** Labels keep page visibility switches with the other evidence, because the live classifier starts a new timing segment at each one; new labels also retain final behaviour, network, signature and configuration inputs. Final-snapshot replay uses the live classification path. Older labels can lack these inputs; coverage is reported explicitly, and final snapshots cannot reconstruct historical changes. A session labelled differently by two sources is listed as a disagreement; each label still counts in its own source.

These are measurements on labelled sessions, not production accuracy, and verdicts remain categories, not probabilities.

## Detection bench and fixture corpus

Our detection bench runs scripted browsers against a temporary collector, headless and muted. Every run is automation the harness drives, so it never measures false positives: the report's three classes are human (always 0 runs: the bench has no people), generic automation and AI-agent framework mechanics, and it lists which live rule fired for each run, so the ladder from protocol flags (WebDriver, headless) to behaviour (clicks, typing) is visible per kind. A run fails the bench when any automation reads Human, live or in the behaviour shadow. Kinds:

- `headless`, `stealth` (normal user agent, `navigator.webdriver` hidden), `marker-only` (plus an overlay marker) and `centre-clicks` (presses with no pointer movement).
- Framework mechanics over raw CDP, reproducing each framework's published input (no framework code runs): `playwright-mechanics`, `puppeteer-type-delay`, `browser-use-mechanics`, `agent-browser-mechanics`, `magnitude-mechanics` and `cua-mechanics`.
- Adversarial humanisers (added 2026-11-03): agents that try to look like a person. `humaniser-minjerk` (smoothed minimum-jerk paths timed by Fitts's law, overshoot, a settle before each press, log-normal typing with real Shift presses), `humaniser-trace-replay` (human cursor speed profiles from SapiMouse bent onto each target), `humaniser-jitter` (timing jitter only) and `humaniser-fill` (humanised pointer, text inserted at once). Each kind runs once per seed, and seeds 101-105 are set aside for testing. Results are in the accuracy report.
- Third-party kits (ghost-cursor, CloakBrowser's humanize wrapper, Cursory, browser-use, Stagehand, Patchright, rebrowser, nodriver) run only when installed and are otherwise reported skipped, never passed. Kinds that need a visible window (a debugging bar, an automation banner, operating-system input) are listed as deferred.

A fixture corpus needs no browser: it replays scripted input through the real browser probes, an SDK-shaped event stream and every model, for agents (stealth typing, markers, Playwright, Puppeteer with delay, browser-use, a screenshot-driven agent, an extension agent behind the debugging bar, a fill-and-scroll script, an agent calling the site's API directly) and for people (at a desktop, on a phone with autofill, with a password manager, dictating, with autocorrect, keyboard-only, with a screen reader, by voice control, with CapsLock or Sticky Keys, reading with the pointer resting on Next, as a power user with shortcuts and with DevTools open). These are synthetic fixtures written from how each input path behaves, not recordings of real people, so the results are rule checks, not rates.

People who never reach Human (keyboard-only, screen reader, dictation, password manager, voice control) stay Unknown, which is never counted as human. The marker rule holds the live verdict at Unknown while an overlay is present, even when the input looks like a person's. A separate table of people cases (voice control, screen readers, switch access, phones, pens, autofill, CapsLock, Sticky Keys, non-US layouts, input methods, head pointers, on-screen keyboards, text expanders, readers holding still, dictation, and fast, slow, one-finger and tremor typists) is checked against every live rule in the test suite; none may be AI agent.

## Shorter raw diagnostic retention

Settings → Data & privacy → Raw diagnostic retention offers an explicit per-site window for **future** behavior batches: one hour, one day, seven days, or the operator's server limit. The server API for the loopback dashboard is `GET /api/admin/diagnostic-retention?site=…` and owner/admin `PUT /api/admin/diagnostic-retention` with `{siteId, minutes}` (`null` inherits the server limit; integers 1–10,080 select minutes). The effective window is always capped by the server limit the operator sets. The UI shows the effective window before saving.

Changing this policy never rewrites existing expiry timestamps or runs deletion. Raising it cannot recover expired records. The current 60-second detector window is unchanged. Independently labeled evaluation snapshots contain their own bounded evidence/features, follow their separate 30-day retention, and do not extend raw-batch storage. Subject erasure removes associated raw and evaluation records; collection-consent withdrawal immediately discards raw batches and visitor self-declared labels under the existing consent policy. Controlled operator labels remain separate.

The default remains seven days unless the operator configured a shorter global limit. One application trial does not justify reducing every site's investigation/replay window. The one-day and one-hour choices are explicit, prospective data-minimization options, with regression tests proving live-window behavior and retained labeled evaluation do not depend on expired raw batches.


## Changelog

Earlier rule changes and measurements. They describe what changed at the time; the current rules are in [Verdicts](/docs/verdicts).

### Injected input reads AI agent (heuristic-v4.28)

read AI agent at high confidence instead of Unknown. Other engines keep Unknown until real people's sessions there are checked. The report's "What changed" section has the before and after.

### Human needs a physical device (heuristic-v4.27)

read Human. Since heuristic-v4.27, input that only looks human is not enough: the page's behaviour stream must also show an input-pipeline delay or steady device polling. On the re-run test seeds all 20 read Unknown. The cost is more Unknown: in the FP-Agent replay, which has no device timing, no person can read Human any more (that replay is tuned data, not a measurement of real visitors). The report's "What changed" section shows the full FP-Agent set, the frozen test split and the adversarial bench before and after, and how the rule was chosen (never on the frozen test split).

### Bench and fixture snapshot, 2026-10-28 (superseded)

A historical snapshot, kept for the record: these numbers were measured on 2026-10-28 with heuristic-v4.3, which was live then. They do not describe the current classifier. In one headless run with the wire v2 SDK, all 10 browser kinds that ran read AI agent except `cua-mechanics` (integer clicks, whole-text insertion), which read Unknown; none read Human, live or in the shadow model. Before heuristic-v4.2 (same kinds, older SDK), only `headless` and `centre-clicks` read AI agent and `magnitude-mechanics` read Human; that run led to the rule that eight key presses spaced within 2 ms of each other never count toward Human. One run per kind is a smoke check, not a rate. The fixture corpus, over 3 seeds per scenario:

| Model | Agent scenarios ever called human | Agent scenarios called agent | People ever called agent | People reaching human |
| --- | --- | --- | --- | --- |
| heuristic-v4.3 (live then, superseded) | 0 of 36 | 30 of 36 | 0 of 51 | 17 of 51 |
| heuristic-v4.2 (comparison) | 0 of 36 | 18 of 36 | 0 of 51 | 17 of 51 |
| heuristic-v5 (shadow) | 0 of 36 | 21 of 36 | 0 of 51 | 17 of 51 |
| behaviour-v2 (shadow) | 2 of 36 (a person-like input beside an overlay marker) | 33 of 36 | 9 of 51 (voice control, and readers who click without moving the pointer) | 7 of 51 |

Shadow models (heuristic-v5, behaviour-v2) are scored only: they never change the verdict your site receives. The 51 people here are the synthetic fixture corpus; the accuracy report counts the 159 consenting people of the research study, a different set.

The 12 runs heuristic-v4.3 adds are the four new agents, each through two families (screenshot cursor flicker and still pauses; the debugging bar and still pauses; script fills and scroll-to-centre clicks; requests the page never sent and still pauses). The two marker agents stay Unknown (the marker holds them there; one of them meets only the timing family). Readers resting the pointer on Next meet the timing family alone and stay Unknown. Before heuristic-v4.3 (at 9cb1e07, without the new scenarios) the same table read 0 of 24, 18 of 24, 0 of 33 and 14 of 33 for heuristic-v4.2.