If you’re the one who gets the call when production (the live version real people use) breaks, this page is for you. Tests only catch what they test for, so what happens when a bad change gets past them? This page follows a real run where exactly that happened, and shows you what Evolve did about it.
A change that passes everything
Same goal as before: make checkout faster. The chosen change cached the tax table and also sped up looking up coupons. It passed the build, the unit tests (small tests of one piece of code each), the checkout contract (a list of requests with the answers they must get) and the speed test.
It also had a bug. Using a coupon called SPRING made it crash. None of the tests used that coupon, so none of them noticed.
Real traffic finds it
Real customers do use SPRING. Evolve’s first step with a new version is a canary: it gets a small share of the real requests, while the current version handles the rest. Some of those requests failed.
| Step | New version | Current version | What Evolve saw |
|---|---|---|---|
| 5% of traffic | 31 requests, 3 failed | 573 requests, 0 failed | error rate 9.7%, over the 0.1% limit |
That limit is called a guardrail, and it comes from the policy, the rules file Evolve keeps in your project. Go over it and the change is out, no debate.
What Evolve did by itself
- Sent every request back to the last version known to be good.
- Checked that the good version really was the one answering requests again.
- Marked the run as rolled back (the new version taken away, the old one put back) and raised an alert.
About 95% of customers never touched the new version. The ones who did were on it for less than a minute.
What you learn from it
The record shows exactly which check was missing: nothing tested the SPRING coupon. So the fix is a to-do item: add the missing case to the checks, and every change after this one is held to it.
This is also where Evolve’s confidence comes in: its measure of how far it can trust each agent. An agent whose change gets caught like this loses the right to ship (put changes live) without your say-so, until it earns it back. See Earned autonomy.
Next
The pieces, one at a time: start with Goals.