If you care whether a change actually helped, this page is for you. A canary, the new version tried on a small share of real traffic first, checks it’s safe; an experiment checks whether people do better with it. This page shows how Evolve can run a fair test between the new version and the old one, and why it refuses to call a winner early.
Is it better?
A canary, the new version’s first small share of real traffic, asks: is the new version broken? An experiment asks a second question: do people do better with it? For example, do more of the people who reach checkout go on to place an order?
How it works
- Measure where you’re starting from. Evolve measures the current version’s rate, the share of people who do the thing you want. Say 9.1% of checkouts turn into orders.
- Decide up front how much data you need. From that rate, and the smallest improvement you’d care about, Evolve works out how many visits each version needs. It writes that plan down before the experiment starts.
- Split by person. Each person is given one version and stays on it, so nobody flips back and forth.
- Keep the guardrails on. Guardrails are the limits on errors and speed set in your policy, your project’s rules file. If the new version breaks something, the experiment stops right there.
- Decide once, at the end. When both versions reach the planned size, a standard statistical test (maths that says how likely a difference is to be luck) decides. Evolve never names a winner early. Peeking early is how luck gets mistaken for success.
Think of flipping a coin. Three heads in a row doesn’t mean the coin is rigged. You have to flip it enough times before you can say anything. Same idea.
A real result
In Evolve’s own test, pretend shoppers bought less when checkout was slow:
| Version | Checkouts | Orders | Rate |
|---|---|---|---|
| current | 5,529 | 503 | 9.1% |
| new (faster) | 5,504 | 694 | 12.6% |
That’s a lift of 38.6%: the order rate rose by 38.6% of what it was. And it’s very unlikely to be luck. So the new version was promoted: it now serves everyone.
Honest limits
- Experiments need traffic. A quiet product may run out of time and end inconclusive. When that happens, the current version stays, unless you’ve said otherwise.
- If the split itself looks broken (far more people on one version than planned), the experiment is thrown out.