If you’re deciding how much a coding agent may do without you, this page is for you. How much should you trust it? As much as it has earned, and no more. This page explains how an agent goes from “ask me first” to “go ahead”, what sends it back, and the numbers Evolve keeps on itself so you can check its work too.
Trust, built on purpose
Evolve applies Organizational Trust Engineering to itself. The idea: trust is confidence that an outcome can meet an expectation. What you expect of Evolve is simple. Only proven changes ship (go live) without you, and you can always see why. Evolve measures how well it’s living up to that, and adjusts how much it lets agents do.
Supervised, then autonomous
It works a lot like a new driver. First they drive with someone beside them. After enough clean trips they go solo. One crash, and it’s back to lessons.
- A new agent on a repository (a project’s home in Git) starts supervised. Its chosen change waits for you to approve or reject it, with all the evidence beside it.
- After a streak of clean promotions (three, unless you change it) it becomes autonomous. A promotion is a change going fully live after its canary, the first small share of real traffic. An autonomous agent’s chosen changes ship by themselves, still through every check and canary.
- It goes back to supervised the moment:
- one of its changes passes every check but is caught in production, the live version real people use;
- you regret a change it shipped, by saying so with the
evolve regretcommand; - it misses one of your confidence objectives, the targets you set for the numbers in the table below. Only the ones about whether bad changes get through count here: escapes, regrets, unknowns, time to a safe state, wrong picks and escaped defects. The ones about how often people step in (unattended rate, interaction load, escalation load) are only there for you to review. Supervision itself moves those numbers, so they never hold an agent back. The same goes for the ones about how smoothly the work goes (first attempt rate, wasted spend, stuck rate, plan calibration). They’re about cost and effort, so they stay in the review too.
Every change of level is recorded with its reason. “Why is this agent allowed to ship by itself?” always has an exact answer.
What Evolve measures about itself
Some of these count runs: everything Evolve does for one goal, a thing you’ve asked for. A run can have several attempts, each one agent’s try at the goal.
| Indicator | The question it answers |
|---|---|
| check escape rate | how often do changes get past the checks and fail later? |
| regret rate | how often do promoted changes turn out to be harmful? |
| unknown rate | how often is there too little evidence to judge? |
| interaction load | how often do people have to step in? |
| unattended rate | how much ships without anyone approving? |
| time to safe state | how long does a rollback take? |
| wrong pick rate | how often does a person later say a run picked the wrong change (evolve run overrule)? |
| escaped defect rate | how often is a bug found later in a change that was delivered (evolve run defect)? |
| first attempt rate | how often is a goal reached by the first attempt, with nobody stepping in? |
| wasted spend rate | how much of the money spent went on attempts, plans and reviews that weren’t used? |
| stuck rate | how often does a run end with nothing left to try, or waiting on a person? |
| plan calibration | when the planner (the agent that plans a run’s approaches) was 75% sure or more, how often was it right? |
| escalation load | how many questions do the planner and the reviewer (an agent that judges changes) ask a person, per run? |
You set an objective and a threshold for each. Cross a threshold on one of the six that count, and agents go straight back under supervision.
The confidence review
The evolve confidence command, and the Confidence page in Evolve’s web interface, show the lot: indicators against objectives, each agent’s autonomy and why, runs waiting for your approval, and the most common reasons confidence was lost. That last one tells you which checks to improve.