The Experiment Loop
Point an agent at code, give it a score to chase and git as the undo button, and let it keep or discard experiments until the budget runs out.
Contents
Use it when You have a number you want to improve and a command that measures it, but no time to run the experiments yourself.
One loop, found twice. Two unrelated projects, in different languages, for different problems, landed on the same pattern:
- Propose a change and commit it.
- Run the tests. Fail? Reset and try something else.
- Run the benchmark and extract a score.
- Score improved? Keep the commit. Otherwise reset.
- Log the experiment and repeat.
The human writes the instructions. The agent writes the code. Git is the undo mechanism. The score is the compass.
Five pieces make it work anywhere: a file to change, a command to run, a number to improve, a time budget, and a keep or discard rule backed by git.
The eval harness is the real work. The loop is mechanical; an agent can run it all day. Creating a command that produces a meaningful number is a design problem.
The goal tells you what to test. Your goal predicts what the agent will try, and what it will break. Lock those tests so it can’t improve the score by weakening the guardrails.
Not AutoML. A grid search tops out at the best combination you parameterized. This loop tops out at whatever the agent can think of.