# Measured workflow comparisons

Tools → Performance compares the same task through the UI and through a tool using explicit controlled trial reports. Existing tool execution times and inferred completed-work channels are **not** a UI baseline. No savings appear until comparable measurements exist.

## Run a matched trial

1. Choose a task with a verifiable outcome, such as exporting a report and checking its expected row count. Define a start boundary and an end boundary for the entire task, including waits, confirmations and downloads. A successful HTTP response alone is not proof that the task completed.
2. Write a protocol that fixes the driver, browser, permissions, application version, input complexity, dataset and network conditions. Use opaque identifiers for the protocol, scenario and revision. Never send input values, output contents, credentials or personal names as identifiers.
3. Run one UI attempt and one tool attempt with the same task, server-associated user and account, and conditions, resetting application state between attempts. Alternate or randomize the order across pairs to reduce cache and learning effects. Both attempts use the same `pairId`; every new pair gets a fresh ID.
4. Measure elapsed duration with the same monotonic clock and boundaries. Count retries with one documented rule for both paths (for example, each repeated failed logical step; ordinary asynchronous polling is not a retry). Report failed and cancelled attempts as well as successes. Report missing retries as missing instrumentation, not zero; the API requires an explicit measured count.
5. Check the outcome in your authenticated application backend and send the terminal measurement. Use a live Telltale session token associated with the user or account by your server. Keep the Site server key on your backend. The Telltale collector records your assertion; it does not independently verify the outcome or test conditions.

## Report an attempt

```js
const response = await fetch(`${collector}/api/v1/workflow-trials`, {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.TELLTALE_SERVER_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    sessionToken,
    eventId: 'export-pair-001-ui',
    pairId: 'export-pair-001',
    taskId: 'export_report',
    protocolId: 'same-agent-same-browser-v1',
    scenarioId: 'report-small-fixture',
    appRevision: 'release-2026-10-28',
    controlled: true,
    mode: 'ui',
    outcome: 'succeeded',
    durationMs: measuredElapsedMs,
    retries: measuredRetries,
  }),
});
if (!response.ok) throw new Error(`Trial report rejected: ${response.status}`);
```

For the matching tool attempt, use a new `eventId`, the same `pairId` and comparison metadata, `mode: 'tool'`, and that attempt's measured outcome, duration and retries. Outcomes are `succeeded`, `failed` or `cancelled`. Report one terminal attempt per mode per pair. Do not overwrite a failed trial with a successful retry; include task retries within that attempt's total duration and retry count, or start a new matched pair under your protocol.

All identifiers are 1–80 ASCII letters, digits, dots, underscores or hyphens, starting with a letter or digit. Duration is an integer 1–86,400,000 milliseconds; retries is an integer 0–10,000. Unknown fields are rejected. `controlled: true` attests that you followed your matching protocol; it does not turn observational traffic into a controlled experiment.

An identical `eventId` and unchanged facts returns the original record with `replayed: true`; conflicts and duplicate modes return 409. Retrying requires a still-valid session token. After expiry or consent withdrawal, do not restart collection merely to resubmit a measurement. Recording trials never starts an application action, approves a tool, updates a verdict or creates chargeable usage.

## Read the comparison

The selected Performance period uses report arrival time. Both counterparts must be retained inside that period. Unmatched attempts are counted as excluded, never as failed or zero-duration attempts. Trial metadata is compared exactly; a pair with different task, protocol, scenario, app revision or server-associated identity is rejected.

For matched pairs, the table shows attempts, completions, failures, cancellations, median total duration and median retries. Duration and retries include all outcomes. The observed time difference is the median of each pair's UI duration minus tool duration, using only pairs where both succeeded. It appears only after at least three such pairs. A negative difference means the tool was slower. This is a descriptive sample, not a significance test, a causal claim or a production-wide savings estimate. Completion differences and failures remain visible alongside it to prevent a faster unsuccessful path looking better.

Measurements are backend-reported, including any browser timing your backend accepts. Independently validate that instrumentation before drawing conclusions. Token counts and monetary savings are not collected or inferred.

Trial records are kept for 90 days, with a 10,000-attempt cap per Site. New reports are rejected at the cap until retention cleanup frees space. Deleting a subject removes their trials; deleting a site removes its records. Retention and erasure can leave unmatched counterparts. Only workspace members allowed to read that Site can read its dashboard comparison. No measurement history is synthesized for existing sessions.

The reporting API is available on the loopback application listener. The optional public demonstration listener does not expose this new path. Do not expose the dashboard merely to submit trials; run the reporting backend alongside the local collector.