The Experiment Loop

Point an agent at code, give it a score to chase and git as the undo button, and let it keep or discard experiments until the budget runs out.

Contents
    Hand-drawn loop: propose and commit, run tests, score, then ask whether the score improved. Yes keeps the commit as the new baseline (in red); a failed test or no improvement goes to git reset, and both paths loop back until the budget runs out.

    Use it when You have a number you want to improve and a command that measures it, but no time to run the experiments yourself.

    One loop, found twice. Two unrelated projects, in different languages, for different problems, landed on the same pattern:

    • Propose a change and commit it.
    • Run the tests. Fail? Reset and try something else.
    • Run the benchmark and extract a score.
    • Score improved? Keep the commit. Otherwise reset.
    • Log the experiment and repeat.

    The human writes the instructions. The agent writes the code. Git is the undo mechanism. The score is the compass.

    Five pieces make it work anywhere: a file to change, a command to run, a number to improve, a time budget, and a keep or discard rule backed by git.

    The eval harness is the real work. The loop is mechanical; an agent can run it all day. Creating a command that produces a meaningful number is a design problem.

    The goal tells you what to test. Your goal predicts what the agent will try, and what it will break. Lock those tests so it can’t improve the score by weakening the guardrails.

    Not AutoML. A grid search tops out at the best combination you parameterized. This loop tops out at whatever the agent can think of.

    Topics

    Agents in Production 10Building with AI 9AI Product Strategy 24

    Start here

    How Agents Run in ProductionBeyond LLMs: The Whole Product Framework for AIThe Verification Bottleneck: Why AI's Real Cost Is Human Attention

    ↑ ↓ to move, Enter for all results