Docs · Platform
How Evals and replay works
Ship a change with evidence.
1Pick real work
Choose past sessions from the part of the business the change affects.
2Replay it both ways
Run the current version and the candidate on the same work with the same budgets.
3Compare with confidence
A regression is flagged only when the 95% confidence intervals for baseline and candidate do not overlap.
4Roll out, then keep watching
Post-rollout checks and your own telemetry show how the change holds up.
Evaluation built into the platform
- Fair comparisons
- Baseline and candidate runs share a workspace and slice of work, never overlap, and use matching budgets.
- The failures that matter
- Standard cases cover prompt injection, expired approvals, and revoked connections.