The policy file

If you want to change how your project is judged, this page is for you. It’s the reference for evolve.yaml: every section, every setting. You don’t need to read it top to bottom. evolve init writes a good one for you, so come here when you want to change something and need to know what it does.

Words this page uses

• Agent: an AI tool that writes code. Run: everything Evolve does for one goal. Attempt: one agent’s try at it. Candidate: an attempt with a change worth checking.
• Check: a test Evolve runs on every candidate, which agents can’t change.
• Commit: a saved snapshot of your code in Git. Branch: a named line of commits, like main. Diff: the line-by-line list of what a change did.
• Sandbox: a sealed-off space a program runs in, cut off from the rest of the machine.
• Canary: a new version given a small share of real traffic first. Promotion: it going fully live. Rollback: putting the old version back.
• p95 latency: how long the slowest 5% of requests take to answer.
• Operator config: Evolve’s own settings file, kept by whoever runs Evolve (the operator). Reviewer and planner: agents the operator can set up to judge changes and to plan runs, both described below.
• YAML: the plain-text format these settings are written in.

How it’s used

Evolve reads evolve.yaml from the commit a run starts at, and locks it for that run. Nothing an agent changes can alter the rules it’s judged by. To check a file, run evolve policy check --file evolve.yaml.

toolchain

The tools your project is built and tested with, its compiler and test runner: go (what you get if you leave the line out), node, python, rust, or none for a project that only needs what every system has, like a Makefile or a shell script. evolve init writes the line for you when the project isn’t Go.

Checks that run your project’s own commands (build, command, gui) are given that toolchain. A few things to know:

• Checks have no network. So whatever your project depends on has to be in the repository, or already in the verifier’s cache (the store of downloads Evolve’s checker keeps for itself).
• The toolchain’s caches are kept apart from every agent’s.
• If the machine doing the checking doesn’t have the toolchain, those checks come back unknown, with the reason.
• A coding agent can see every toolchain the machine has, and may run your project’s test and build commands by itself.

repository

What agents may touch.

attempts

How many agents try, and within what limits.

Second chances

When an attempt says it’s done, the checks run next, and if there’s a reviewer, it reviews the change too.

If the checks fail, or the reviewer finds the change works but has faults, the agent is told exactly what was found and asked to fix it. That’s an intervention. It happens in the same folder of code and, where the agent can manage it, the same conversation with its AI, so the agent keeps what it already knows.

A change only fails for good when it can’t be improved: the reviewer calls its approach a dead end, or it runs out of interventions. Then the run goes back to where the approach was chosen and tries the next one, told why the first was dropped. If every planned approach is dropped, it plans once more with what it learned.

Two more things. An attempt that ran out of time is carried on the same way, but if two run out of time with much the same change, the run stops as blocked. That usually means the goal is too big, or the description of what’s wanted has a problem. And a rejection for touching a protected file, going out of scope or going over budget never gets a second chance.

The plan

If the operator config names a reviewer, and nobody has already said which approaches to try (with strategies, in the objective or here), a run is planned before any attempt starts.

Here’s how. The planner reads the goal and the code, and lists the approaches it can see (up to five). For each, it says how confident it is that an agent taking that approach would pass the goal’s checks within one attempt’s limits.

Attempts only go to approaches at or above min_confidence, best first, and no more than count of them. Usually that’s one at a time, with the next one tried only if the ones before weren’t enough. The exception: when the goal leaves a real design choice and the two best approaches are close, those two run side by side and the reviewer compares them.

Each attempt is told its approach and what the planner found, so it doesn’t have to explore the code all over again.

And if no approach clears the bar? The run stops as blocked before spending a single attempt, with the planner’s reasons and any way it suggests to split the goal. The planner can also ask you a question first (answer on the run’s page, or with evolve run answer). The plan is recorded with the run, and evolve run show shows it.

service

How to build and start your program, for the checks that actually run it.

checks

The trusted judges. Each one runs in its own sandbox against the candidate, using your own protected copies of the files.

Every check has an id. It can also set required (a candidate that fails it is out), repeat (run it up to 10 more times, to catch results that come and go) and timeout_seconds.

A contract lists cases like this:

schema_version: 1
id: checkout-v1
cases:
  - name: applies welcome coupon
    method: POST
    path: /checkout
    body: '{"region":"us-ny","coupon":"WELCOME5","lines":[{"sku":"sku-cheese","quantity":2}]}'
    expect:
      status: 200
      json: {discount_cents: 89, total_cents: 1777}

A load profile says how hard to push:

schema_version: 1
id: checkout-load-v1
concurrency: 25
requests: 440
warmup: 20
max_error_ratio: 0.01
mix:
  - {weight: 3, method: POST, path: /checkout, body: '…', expect_status: 200}

selection

How to pick between candidates that passed every required check. rank is a list of keys, tried in order. A later key breaks the ties an earlier one leaves.

tolerance treats values that close together as equal. So with {key: lowest_p95_ms, tolerance: 5}, a 3 ms difference counts as a tie, and the next key decides.

deployment

How the chosen candidate gets to production. Leave this section out and Evolve stops at a verified candidate, which you merge yourself.

experiment

See experiments for how the decision is made.

spec

Tests first. With this on, each goal’s tests are written, and agreed, before anyone builds the thing. It’s off unless you set it.

spec:
  tests: true
  check: tests        # the command check that runs the tests (default: tests)
  adopt: reviewer     # who adopts the tests the reviewer chose: reviewer (default) or operator

With it on, evolve run next, evolve run "…" and evolve run start begin with a spec run for the goal. Its agent writes only new test files (the ones repository.protected_existing covers) that say what the goal asks for, as behaviour you could see from outside.

Evolve accepts those tests when the build still passes and the tests fail on the current code. That’s on purpose: a test of something new should fail until the new thing exists.

Then it’s your turn. You see the tests and adopt them. Or you say what they’re missing with evolve context add, and have them written again with --new-tests. Adopted tests are merged into the branch, which makes them existing tests: every implementation has to pass them, no agent may change them, and they stay on as regression tests for every later goal.

With a reviewer. If a reviewer is set up, the tests it chooses are adopted straight away (it asks when it isn’t sure). Want to read them first? Set spec.adopt: operator, or pass --read-tests for one goal. The chosen tests are shown, and wait for your yes.

When a test is wrong. The reviewer is given what the tests printed when Evolve ran them. It checks that each one fails because of the behaviour the goal asks for (a test that trips over its own setup doesn’t count), and that any expected values worked out by hand are right. If you still spot a bad test or a wrong number, --amend-tests adopts the tests and stops before building. Fix them on the branch (you may; agents may not), commit, and start the goal again. Evolve builds against the tests as you left them.

When an old test gets in the way. The reviewer also looks for existing tests that contradict the goal: a test that pins down the very behaviour the goal changes. No agent may change an existing test, so nothing could pass while it stands. When it finds one, the goal’s own tests are still adopted, but nothing is built. Evolve names the test and stops. Either let the goal change it (unprotect: [path] in the goal), or change it yourself and commit. Then start the goal again.

One more thing: a command check can be asked to fail on purpose. expect_fail: true passes only when the command fails.

record

What evolve sync carries to other machines.

A complete example

The demo’s examples/checkout/evolve.yaml uses most of what’s on this page. evolve init writes a smaller one for your project (Your first project).