Test measurement proposal (template-driven)

Summary

We want a built-in measurer that can validate correctness by materialising a test file from a template into the worktree, then running a test command, then capturing the result as a measure.

This is distinct from “tests pass” (which is just commandExitCode against a repo’s normal test suite). This measurer is for targeted correctness checks that can be applied across tasks, even if the repo doesn’t already contain the relevant test.

Terminology alignment

Lifecycle (intended execution order)

  1. Worktree is created
  2. Agent/harness runs (modifies repo)
  3. Measurements are applied sequentially (one by one)
  4. When the runner reaches a “test template” measurement:
    • write the templated test file into the worktree
    • run a test command targeting that file (or a configured runner)
    • capture stdout/stderr/exitCode and store as the measure

Name ideas (best-first)

Notes:

Proposed measurement config shape

Minimum viable config:

Example (conceptual):

{
  measurementId: "add_function_correctness",
  measurerName: "templateTest",
  config: {
    templatePath: "evaluation-sandbox/tests/add.test.ts.template",
    outputPath: "test/add.test.ts",
    run: "bun test test/add.test.ts",
    expectedExitCode: 0,
  }
}

Measure/result shape (what gets recorded)

Recommended stored value for this measurer:

{
  exitCode: number,
  stdout: string,
  stderr: string
}

Pass/fail is derived from exitCode === expectedExitCode (default expectedExitCode=0).

Open questions / decisions