Checks and evidence

If you’re the one deciding what counts as proof for your project, this page is for you. A check is a test Evolve runs on every change, and checks are how a change proves it works. Here you’ll find the seven kinds of check Evolve can run, how to write the ones that act like a person using your app, and what makes a result count as evidence.

Words this page uses

• YAML: the plain-text format the examples below are written in.
• Verifier: the part of Evolve that runs the checks. Its verdict on a check is pass, fail or unknown.
• Arguments: the words typed after a program’s name to run it. Environment variables: named settings a program reads when it starts.
• stdin, stdout, stderr: what a program reads as input, what it prints, and what it prints about errors.
• Exit code: the number a program finishes with. 0 means it went fine; anything else means something went wrong.
• Regular expression: a pattern for matching text, like ^hi for “starts with hi”.
• JavaScript: the programming language web pages run.

Seven kinds of check

Browser scenarios

A browser check runs scenarios: what a person does, step by step, and what they should then see. They’re a good fit for web apps, including ones written in Go and compiled to WebAssembly (a way to run such programs inside a browser), where the things that matter happen through clicks and drags. Each step names the element it acts on with a selector, the same short pattern a web page’s styles use, like .node for everything marked node or #app for the element named app.

- name: wire two nodes
  steps:
    - visit: /
    - wait: "#app[data-ready=true]"
    - click: "[data-test=add-node]"
    - click: "[data-test=add-node]"
    - drag: {from: ".node[data-id='0'] .pin-out", to: ".node[data-id='1'] .pin-in"}
    - expect: {selector: .wire, count: 1}
    - eval: {js: "JSON.stringify(app.graph().edges)", equals: '[[0,1]]'}

When several elements match, you can pick one by position: {selector: .node, nth: 2} is the second.

Every step waits up to 10 seconds for the elements it names. If the page throws an error nobody caught, the scenario fails, unless it sets allow_errors: true. The browser runs in a sandbox (a sealed-off space that can’t reach the rest of your machine) of its own, apart from your app’s, and the only thing it can reach is the app. With no browser on the machine, the result is unknown.

Native apps

A gui check runs scenarios against a native app built with Gio, which runs on Linux, Windows, macOS, Android, iOS and in the browser. It doesn’t need a window, a screen or a graphics card. Each project has a driver, a small test program that works the app like a person would: ui/evolve_gui_test.go, written for you by evolve new --kind desktop and protected like every test, so agents can’t change it. The driver starts the app fresh for each scenario, then presses, drags and types through Gio’s own input handling.

Elements are found by their accessibility labels, the names screen readers use: the text a label or button shows, or a description the app gives to things with no text, like node 0 output. * matches anything. In a browser you could ask the page with JavaScript; a native app has none, so a state step compares what the app reports about itself:

- name: wire two nodes
  steps:
    - click: "Add node"
    - click: "Add node"
    - drag: {from: "node 0 output", to: "node 1 input"}
    - expect: {selector: "wire *", count: 1}
    - state: {path: edges, equals: [[0, 1]]}

The steps are the same as a browser scenario’s, minus visit and eval. * matches any run of characters, slashes included (cannot open *). A state step checks a value (equals), how many items a list or object has (count), or that a path isn’t there (absent: true).

Display scale. Apps measure sizes in dp (density-independent pixels), which a screen turns into more or fewer real pixels depending on how sharp it is. Each scenario runs at 1 real pixel per dp unless it sets scale, and a check with scales: [1, 1.6] runs every scenario at each one. expect: {selector: "node 0 output", within: "node 0"} says one element has to sit inside another. That catches a common mistake: a layout that mixes raw pixels with dp looks fine at 1x and breaks at 1.6x. It compares the areas elements take up, so give anything you want held inside its container an area of its own.

A failing step says what it looked for and which labels were actually there, so the next attempt knows what to fix. The driver runs the app in its own process (a separate running program), so it’s trusted exactly as much as the project’s tests are.

Command-line programs

Command-line programs are checked by a transcript: a list of runs of the program, and what each one should produce. Each case runs in a fresh, empty directory in a sandbox of its own, with no network, only the two basic settings PATH and HOME, and a time limit (10 seconds unless it says otherwise).

cases:
  - name: counts the words of a file
    args: [words.txt]
    files: {words.txt: "a b\nc d e\n"}
    expect: {stdout: {equals: "5\n"}}
  - name: a missing file is an error
    args: [nope.txt]
    expect: {exit: 1, stderr: {contains: [nope.txt]}}

A goal for a program doesn’t need a speed target. evolve run "support counting lines with -l" is reached when the transcript passes and every other check keeps passing. For a Go program that reads its options with Go’s standard flag package, evolve init (the command that sets a project up for Evolve) suggests a transcript to start you off, beginning with -h, the option that prints help.

Clean and offline

Each check runs on a fresh copy of the attempt, in a sandbox with no network. The attempt can’t phone out during its test, and nothing left over from an earlier check can help it pass.

The verdict is never made inside the attempt’s own sandbox, either. For HTTP, benchmark and command-line checks, your program runs alone in there, and Evolve asks the questions and marks the answers from outside. A sneaky program can answer however it likes, but the marking happens somewhere it can’t reach.

Checks run with your project’s own tools: its toolchain, the compiler and test runner for its language. The toolchain: line in the policy (the rules file evolve.yaml) says which: Go, Node, Python or Rust. Or none, for a project that needs nothing past the basic system programs in /usr/bin and /bin. That’s what evolve init picks for one whose tests run with make test. They’re kept offline, so whatever your project depends on needs to be in the repository, or already in the verifier’s cache, the store of downloads Evolve’s checker keeps for itself.

To catch flaky tests (ones that pass or fail by luck), a check can be set to run several times, and one failure counts as a failure.

Evidence is signed and specific

Each result becomes a piece of evidence, signed with a key the agents can’t reach and tied to:

• the exact commit (a saved snapshot of the code in Git) and the exact set of files in it;
• the exact built program, by its fingerprint (a SHA-256 digest, a short code that changes if even one byte does);
• the exact policy and version of each check;
• an expiry time.

If any of those is different, even by one byte, the evidence doesn’t count. And when evidence is missing or out of date, the answer is unknown. Unknown never lets a change through.

Checks only prove what they measure

Here’s the honest limit. A check that can be passed by deleting the thing it measures proves nothing. When it goes wrong shows a change that passed every check and still had a bug. That’s why the next steps exist.

Next

How the winner is chosen.