Outcome Loops

Put a rubric-graded quality gate between the agent's output and the user: a separate judge scores, the agent revises until the rubric passes or the budget runs out.

Contents
    Circular loop diagram: the agent generates output, a rubric and judge score it, the score and feedback go back to the agent, and the agent revises, with iteration at the center.

    Use it when Agent output is usually fine but rarely first-draft ready, and you keep fixing it by hand.

    An outcome loop is a judge call between the agent’s output and the user’s inbox. The agent finishes its work. A separate judge, which does not see the agent’s reasoning, scores the final output against a rubric. Fail, and the score and feedback go back for revision. Pass, and the output ships.

    It wraps around the agentic loop as a quality gate; it does not replace it.

    Three lessons from running it:

    • Build the judge first. A lenient judge passed everything on the first try and the loop never fired. A strict one made the loop valuable. The judge is the investment, not the loop.
    • Stop at the first pass. More iterations can make things worse: one model went 4, 8, 10, 9, 8, fixing the original problems and then adding new ones.
    • Start the rubric loose. The rubric is a product. The one you start with will not be the one you end with: tighten where the judge is too lenient, loosen where it is too strict.

    Humans still validate the judge.

    Topics

    Agents in Production 10Building with AI 9AI Product Strategy 24

    Start here

    How Agents Run in ProductionBeyond LLMs: The Whole Product Framework for AIThe Verification Bottleneck: Why AI's Real Cost Is Human Attention

    ↑ ↓ to move, Enter for all results