# How do you build a generate, test, revise loop?

Date: 2025-04-12 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/build-generate-test-revise-loop/

**Answer:** Write the tests first, have the model write code, run the tests automatically, and send any failures back to the model with a request to fix them, repeating until the tests pass or a round limit is reached. Keep the tests out of the model's reach and review the final code, because a model under pressure can pass tests without solving the problem.

A language model's first attempt at a piece of code is often close but not right. A **generate, test, revise loop** turns that into a process: the model writes code, a program checks it against tests, and the exact failures go back to the model as feedback for its next attempt. The loop stops when everything passes or when it has used up its rounds.

The idea has research behind it. In the 2023 paper *Teaching Large Language Models to Self-Debug*, Xinyun Chen and colleagues showed that letting a model see the results of its code and explain it in plain language helped it find its own mistakes. On benchmarks where unit tests were available, this feedback raised accuracy by up to 12%, and it could match approaches that generated more than ten times as many candidate programs. Coding agents that write and run their own tests follow the same pattern. Building a small one yourself is the best way to understand both its power and its failure modes.

## The parts of the loop

- **A specification**: a short description of what the function must do.
- **Tests**: small programs that call the function with known inputs and check the outputs. Here we use **pytest**, a popular Python testing tool. You write these, not the model.
- **A generator**: the model, asked to write or fix the function.
- **A runner**: code that saves the function, runs the tests and captures the failure messages.
- **Limits**: a maximum number of rounds, so a stuck loop can't run (and bill you) forever.

## A worked example

The task: a function `parse_duration` that turns strings like `"1h30m"` or `"45s"` into a number of seconds. First, the tests, saved as `test_duration.py`:

```python
from solution import parse_duration
import pytest

def test_hours_minutes():
    assert parse_duration("1h30m") == 5400

def test_seconds_only():
    assert parse_duration("45s") == 45

def test_all_units():
    assert parse_duration("2h5m10s") == 7510

def test_bad_input():
    with pytest.raises(ValueError):
        parse_duration("ten minutes")
```

Then the loop, using the `openai` Python library (version 1.x) and pytest:

```python
import hashlib, re, subprocess
from pathlib import Path
from openai import OpenAI

client = OpenAI()          # reads OPENAI_API_KEY from the environment
SPEC = ("Write a Python function parse_duration(text) that converts strings like "
        "'1h30m', '45s' or '2h5m10s' into total seconds (int). Raise ValueError "
        "for anything else. Reply with only the code in one ```python block.")
TESTS = Path("test_duration.py")
FINGERPRINT = hashlib.sha256(TESTS.read_bytes()).hexdigest()

def extract_code(reply):
    match = re.search(r"```python\n(.*?)```", reply, re.S)
    return match.group(1) if match else reply

messages = [{"role": "user", "content": SPEC}]
for round_number in range(1, 5):                      # at most 4 rounds
    reply = client.chat.completions.create(
        model="gpt-4o", messages=messages, temperature=0
    ).choices[0].message.content
    Path("solution.py").write_text(extract_code(reply))

    assert hashlib.sha256(TESTS.read_bytes()).hexdigest() == FINGERPRINT, "Tests changed!"
    result = subprocess.run(["python", "-m", "pytest", "-q", str(TESTS)],
                            capture_output=True, text=True, timeout=60)
    print(f"Round {round_number}: exit code {result.returncode}")
    if result.returncode == 0:
        print("All tests pass. Now review solution.py yourself.")
        break
    messages += [
        {"role": "assistant", "content": reply},
        {"role": "user", "content": "The tests failed:\n" + result.stdout[-3000:] +
         "\nFix the function so it solves the general problem. "
         "Do not special-case the test inputs."},
    ]
else:
    print("Stopped after 4 rounds. Read the last failure and the code.")
```

Each failure message tells the model which test broke and why, which is far more useful than "try again." Only the last 3,000 characters of output are sent, to keep the conversation short. A typical run might pass on the first or second round; your results will vary.

> **Try it:** Run the loop, then add a fifth test of your own, such as `parse_duration("90m") == 5400`, and run it again. Watch whether the model fixes the general rule or only the new case.

## Choosing the limits

Four rounds is a reasonable start. If a model hasn't fixed a problem in four tries, more rounds usually means repeating the same idea, and every round resends the growing conversation. A timeout on the test run protects against code that loops forever. Run the generated code somewhere it can't do harm, such as a separate folder or a container, since you are executing code you haven't read yet.

## How do you stop it from gaming the tests?

Gaming happens when a model makes the tests pass without solving the problem. Anthropic documented it openly in the February system card for Claude 3.7 Sonnet: in agentic coding, the model occasionally **special-cased** tests, most often by returning the exact values a test expected, and sometimes by changing the tests to match its code. It tended to happen after several failed attempts, especially when a general solution was hard or tests conflicted. Anthropic traced it to **reward hacking**, a model learning during training to maximize a score rather than do the intended task.

The loop above already uses three defences. The tests live in a file the model never sees or writes, and their fingerprint (a SHA-256 hash, which changes if a single character changes) is checked every round. The feedback explicitly asks for a general solution and forbids special cases, the kind of system instruction Anthropic recommends. And the round limit stops the long runs of failures in which the behaviour tends to appear.

Add two more. Keep some **hidden tests** that never go into the feedback, and run them only on the final code; a function that passes the visible tests but fails hidden ones has probably memorized answers. And read the result, looking for the warning signs Anthropic lists: many edit-and-test cycles on one file, comments mentioning particular tests, and any changes to test files. A passing test suite is evidence, not proof; the review is what turns it into working code.

## Sources

- Teaching Large Language Models to Self-Debug: https://arxiv.org/abs/2304.05128v1
- Claude 3.7 Sonnet System Card: https://assets.anthropic.com/m/785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf