How do you build an AI loop with a real gate?

A loop that only asks the model "is this good?" will usually hear "yes". Anthropic reported this plainly in March 2026: when agents grade their own output, they tend to praise it, even when a person can see it is mediocre. The fix is not a stricter question. It is a gate: a check written in code that returns pass or fail on its own, so a bad result is stopped whether or not anyone is looking.

This article builds one. The script below is about 170 lines of plain Python, needs nothing beyond the standard library, and runs as is. It uses a stub in place of a model, so you can run it, watch each gate fail and pass, and only then connect a real model. The output shown further down comes from running it with Python 3.14 on a Mac.

The parts, in plain words

A unit is one piece of work small enough to check on its own: here, one bug fix with its change note. A worker produces a draft of the unit. In a real system the worker is a model call; here it is a stub that replays fixed drafts so the run is repeatable. A gate inspects the draft and returns a list of failures; an empty list means pass.

The script has three gates, each of which can fail the work without a person:

Around the gates sit three rules. A failed unit goes back to the worker with the failure messages as evidence, and only that unit goes back; the others keep their result. Each unit gets at most three attempts, after which it is marked for a person instead of being retried forever. And nothing is copied to the approved/ folder until a person reads the diffs and types y.

These choices follow published practice. Anthropic's guide to agents says agents need "ground truth" from their environment after every step (the result of running code, for instance) and that a maximum number of iterations is a common stopping rule. In its write-up on long-running coding agents, Anthropic told the agent that removing or editing tests was unacceptable, since it could lead to missing or buggy features. A loop needs the same protection, because an agent pushed to pass can make the tests easier instead of the code better. The scope gate turns that instruction into a check: a draft that touches the test file fails, however good the rest of it is.

The script

Save this as gate_loop.py in an empty folder and run python3 gate_loop.py. It writes loop_log.jsonl (one line per attempt) and, if you approve, an approved/ folder.

"""A small loop with real gates. Standard library only (Python 3.10+).

Each unit of work is produced by a worker, then checked by three gates that can fail it:
  1. tests:   the unit's tests must exit with code 0
  2. sources: every claim in the change note must cite an existing source line
  3. scope:   the change may touch only the files the plan allows
A unit that fails goes back to the worker with the evidence. After MAX_ATTEMPTS it goes
to a person instead. Nothing is written to approved/ until a person types "y".
"""
import difflib, json, pathlib, re, subprocess, sys, tempfile

MAX_ATTEMPTS = 3
LOG = pathlib.Path("loop_log.jsonl")

# ---------- The work, already split into units (the splitter's output) ----------
UNITS = [
    {"id": "slugify-accents",
     "task": "Make slugify() turn accented letters into plain ones, as issue.txt asks.",
     "allowed": {"slugify.py", "CHANGES.md"},
     "files": {
        "slugify.py": 'import re\n\ndef slugify(text):\n    text = text.lower()\n'
                      '    text = re.sub(r"[^a-z0-9]+", "-", text)\n    return text.strip("-")\n',
        "test_slugify.py": 'import unittest\nfrom slugify import slugify\n\nclass T(unittest.TestCase):\n'
                           '    def test_plain(self):\n        self.assertEqual(slugify("Hello World"), "hello-world")\n'
                           '    def test_accents(self):\n'
                           '        self.assertEqual(slugify("Crème brûlée à Montréal"), "creme-brulee-a-montreal")\n',
        "issue.txt": 'Issue 12: slugs break on accented letters.\n'
                     '"Crème brûlée" becomes "cr-me-br-l-e".\n'
                     'Expected: strip accents first, then build the slug.\n'}},
    {"id": "reading-time-minimum",
     "task": "Make reading_time() never show 0 minutes, as issue.txt asks.",
     "allowed": {"reading_time.py", "CHANGES.md"},
     "files": {
        "reading_time.py": 'def reading_time(text):\n    return len(text.split()) // 200\n',
        "test_reading_time.py": 'import unittest\nfrom reading_time import reading_time\n\nclass T(unittest.TestCase):\n'
                                '    def test_short(self):\n        self.assertEqual(reading_time("word " * 50), 1)\n'
                                '    def test_rounds_up(self):\n        self.assertEqual(reading_time("word " * 401), 3)\n',
        "issue.txt": 'Issue 15: a 50-word post shows "0 min read".\n'
                     'Expected: at least 1 minute, rounded up.\n'}},
    {"id": "french-dates",
     "task": "Make format_date() return French dates such as '4 novembre 2026'.",
     "allowed": {"dates.py", "CHANGES.md"},
     "files": {
        "dates.py": 'def format_date(d):\n    return d.strftime("%B %-d, %Y")\n',
        "test_dates.py": 'import unittest, datetime\nfrom dates import format_date\n\nclass T(unittest.TestCase):\n'
                         '    def test_fr(self):\n'
                         '        self.assertEqual(format_date(datetime.date(2026, 11, 4)), "4 novembre 2026")\n',
        "issue.txt": 'Issue 18: French pages show English dates.\n'}},
]

# ---------- The worker ----------
# A stub that replays fixed drafts, so the run is repeatable and costs nothing.
# A real worker would send build_prompt(unit, feedback) to a model and parse the JSON it returns.
DRAFTS = {
    "slugify-accents": [
        {"slugify.py": 'import re\n\ndef slugify(text):\n    text = text.lower().replace("é", "e")\n'
                       '    text = re.sub(r"[^a-z0-9]+", "-", text)\n    return text.strip("-")\n',
         "CHANGES.md": "- Slugs now handle accented letters.\n"},
        {"slugify.py": 'import re, unicodedata\n\ndef slugify(text):\n'
                       '    text = unicodedata.normalize("NFKD", text)\n'
                       '    text = "".join(c for c in text if not unicodedata.combining(c)).lower()\n'
                       '    text = re.sub(r"[^a-z0-9]+", "-", text)\n    return text.strip("-")\n',
         "test_slugify.py": 'import unittest\nfrom slugify import slugify\n\nclass T(unittest.TestCase):\n'
                            '    def test_plain(self):\n        self.assertEqual(slugify("Hello World"), "hello-world")\n',
         "CHANGES.md": "- Slugs strip accents before building the slug (source: issue.txt:3)\n"},
        {"slugify.py": 'import re, unicodedata\n\ndef slugify(text):\n'
                       '    text = unicodedata.normalize("NFKD", text)\n'
                       '    text = "".join(c for c in text if not unicodedata.combining(c)).lower()\n'
                       '    text = re.sub(r"[^a-z0-9]+", "-", text)\n    return text.strip("-")\n',
         "CHANGES.md": "- Slugs strip accents before building the slug (source: issue.txt:3)\n"}],
    "reading-time-minimum": [
        {"reading_time.py": 'import math\n\ndef reading_time(text):\n'
                            '    return max(1, math.ceil(len(text.split()) / 200))\n',
         "CHANGES.md": "- Reading time is at least 1 minute and rounds up (source: issue.txt:2)\n"}],
    "french-dates": [
        {"dates.py": 'def format_date(d):\n    return d.strftime("%-d %B %Y")\n',
         "CHANGES.md": "- Dates use French month names (source: issue.txt:1)\n"}] * 3,
}

def build_prompt(unit, feedback):
    files = "\n\n".join(f"--- {p}\n{c}" for p, c in unit["files"].items())
    prompt = (f"Task: {unit['task']}\nYou may change only: {sorted(unit['allowed'])}.\n"
              "In CHANGES.md, end every line with (source: file:line).\n"
              f"Return JSON: {{path: new file content}}.\n\nFiles:\n{files}")
    if feedback:
        prompt += "\n\nYour last attempt failed these checks. Fix only these:\n" + "\n".join(feedback)
    return prompt

def worker(unit, attempt, feedback):
    prompt = build_prompt(unit, feedback)   # what a model would receive
    # Real model: return json.loads(call_your_model(prompt))
    return DRAFTS[unit["id"]][attempt - 1]

# ---------- The gates: each returns a list of failures (empty = pass) ----------
def gate_tests(workdir):
    try:
        r = subprocess.run([sys.executable, "-m", "unittest", "-q"], cwd=workdir,
                           capture_output=True, text=True, timeout=60)
    except subprocess.TimeoutExpired:
        return ["tests: still running after 60 seconds"]
    if r.returncode == 0:
        return []
    lines = [l for l in r.stderr.splitlines() if l.startswith(("FAIL:", "ERROR:", "AssertionError"))]
    return [f"tests: exit code {r.returncode}: " + " | ".join(lines[:4])]

def gate_sources(unit, change):
    problems = []
    for claim in change.get("CHANGES.md", "").splitlines():
        if not claim.strip():
            continue
        m = re.search(r"\(source: ([\w.]+):(\d+)\)\s*$", claim)
        if not m:
            problems.append(f"sources: no source line for claim {claim!r}")
            continue
        lines = unit["files"].get(m.group(1), "").splitlines()
        if not 1 <= int(m.group(2)) <= len(lines):
            problems.append(f"sources: {m.group(1)}:{m.group(2)} does not exist")
    if "CHANGES.md" not in change:
        problems.append("sources: no change note")
    return problems

def gate_scope(unit, change):
    outside = sorted(set(change) - unit["allowed"])
    return [f"scope: changed files outside the plan: {outside}"] if outside else []

# ---------- The loop for one unit ----------
def run_unit(unit):
    feedback = []
    for attempt in range(1, MAX_ATTEMPTS + 1):
        change = worker(unit, attempt, feedback)
        with tempfile.TemporaryDirectory() as tmp:
            for path, text in {**unit["files"], **change}.items():
                pathlib.Path(tmp, path).write_text(text, encoding="utf-8")
            failures = gate_tests(tmp) + gate_sources(unit, change) + gate_scope(unit, change)
        with LOG.open("a", encoding="utf-8") as f:
            f.write(json.dumps({"unit": unit["id"], "attempt": attempt, "failures": failures}) + "\n")
        print(f"[{unit['id']}] attempt {attempt}: " + ("PASS" if not failures else "FAIL"))
        for line in failures:
            print("    " + line)
        if not failures:
            return {"status": "passed", "change": change, "attempts": attempt}
        feedback = failures          # only this unit goes back, with the evidence
    return {"status": "needs a person", "last_failures": feedback, "attempts": MAX_ATTEMPTS}

# ---------- The run: units are independent, a failure stays with its unit ----------
results = {u["id"]: run_unit(u) for u in UNITS}
units = {u["id"]: u for u in UNITS}
print("\n=== Summary ===")
for uid, r in results.items():
    print(f"{uid}: {r['status']} after {r['attempts']} attempt(s)")

passed = {uid: r for uid, r in results.items() if r["status"] == "passed"}
print("\n=== For approval ===")
for uid, r in passed.items():
    for path, new in r["change"].items():
        old = units[uid]["files"].get(path, "")
        sys.stdout.writelines(difflib.unified_diff(old.splitlines(True), new.splitlines(True),
                                                   f"{uid}/{path}", f"{uid}/{path} (new)"))
answer = input(f"\nApply {len(passed)} passed unit(s) to approved/? [y/N] ").strip().lower()
if answer == "y":
    for uid, r in passed.items():
        for path, text in r["change"].items():
            out = pathlib.Path("approved", uid, path)
            out.parent.mkdir(parents=True, exist_ok=True)
            out.write_text(text, encoding="utf-8")
    print("Written to approved/. Units that need a person are listed above.")
else:
    print("Nothing written.")

The test gate uses subprocess.run with capture_output=True and a timeout. Python's documentation describes returncode as the exit status of the child process, where 0 usually means success; a timeout kills the child and raises TimeoutExpired, which the gate turns into a failure instead of a crash. The test runner, not the worker, decides whether the tests passed.

What happens when you run it

Run it and answer y at the prompt (or pipe the answer in with printf 'y\n' | python3 gate_loop.py). This is the full output:

[slugify-accents] attempt 1: FAIL
    tests: exit code 1: FAIL: test_accents (test_slugify.T.test_accents) | AssertionError: 'cr-me-br-lee-montreal' != 'creme-brulee-a-montreal'
    sources: no source line for claim '- Slugs now handle accented letters.'
[slugify-accents] attempt 2: FAIL
    scope: changed files outside the plan: ['test_slugify.py']
[slugify-accents] attempt 3: PASS
[reading-time-minimum] attempt 1: PASS
[french-dates] attempt 1: FAIL
    tests: exit code 1: FAIL: test_fr (test_dates.T.test_fr) | AssertionError: '4 November 2026' != '4 novembre 2026'
[french-dates] attempt 2: FAIL
    tests: exit code 1: FAIL: test_fr (test_dates.T.test_fr) | AssertionError: '4 November 2026' != '4 novembre 2026'
[french-dates] attempt 3: FAIL
    tests: exit code 1: FAIL: test_fr (test_dates.T.test_fr) | AssertionError: '4 November 2026' != '4 novembre 2026'

=== Summary ===
slugify-accents: passed after 3 attempt(s)
reading-time-minimum: passed after 1 attempt(s)
french-dates: needs a person after 3 attempt(s)

=== For approval ===
--- slugify-accents/slugify.py
+++ slugify-accents/slugify.py (new)
@@ -1,6 +1,7 @@
-import re
+import re, unicodedata
 
 def slugify(text):
-    text = text.lower()
+    text = unicodedata.normalize("NFKD", text)
+    text = "".join(c for c in text if not unicodedata.combining(c)).lower()
     text = re.sub(r"[^a-z0-9]+", "-", text)
     return text.strip("-")
--- slugify-accents/CHANGES.md
+++ slugify-accents/CHANGES.md (new)
@@ -0,0 +1 @@
+- Slugs strip accents before building the slug (source: issue.txt:3)
--- reading-time-minimum/reading_time.py
+++ reading-time-minimum/reading_time.py (new)
@@ -1,2 +1,4 @@
+import math
+
 def reading_time(text):
-    return len(text.split()) // 200
+    return max(1, math.ceil(len(text.split()) / 200))
--- reading-time-minimum/CHANGES.md
+++ reading-time-minimum/CHANGES.md (new)
@@ -0,0 +1 @@
+- Reading time is at least 1 minute and rounds up (source: issue.txt:2)

Apply 2 passed unit(s) to approved/? [y/N] y
Written to approved/. Units that need a person are listed above.

Read it unit by unit.

slugify-accents needs three attempts. The first draft only replaces "é", so the accent test fails with the exact strings it got and expected, and the change note has no source. Both messages go back to the worker. The second draft fixes the code properly but also deletes the accent test from test_slugify.py. The tests now pass, which is why a test gate alone is not enough: the scope gate fails the draft because the test file is outside the plan. The third draft fixes the code, leaves the tests alone and cites line 3 of the issue. It passes.

reading-time-minimum passes on the first try. It never goes back, and the retries of the other units do not touch it. This is the point of returning single failed units rather than rerunning the whole batch.

french-dates never passes. The stub keeps returning English month names, the test keeps failing, and after three attempts the unit is marked "needs a person". A real model can get stuck the same way, for example when the fix needs information it does not have. The attempt cap turns an endless loop into a short note for a human.

The approval step shows only what passed, as diffs against the original files. Nothing reaches approved/ before the person answers. Answer anything other than y and the script prints "Nothing written."

What these gates do not check

Each gate checks something narrow, and it is worth saying what slips through.

• The tests check only what they test. If the tests are thin, a wrong fix can pass. That is why the scope gate protects the test file and why a person reads the diff at the end. Anthropic notes that automated tests help verify code but that human review is still needed to check that a solution fits the wider requirements.
• The sources gate checks that a citation exists, not that it supports the claim. The note - Slugs strip accents (source: issue.txt:1) would pass even though line 1 says nothing about stripping. To go further you can require that key words of the claim appear on the cited line, or add a second model call as a reviewer, which is a weaker check because a model can be wrong in the same way as the worker.
• The scope gate checks file names, not intent. A worker can still make a harmful change inside an allowed file.

The loop also differs from Self-Refine (2023), where one model writes, critiques its own output and revises it. Self-Refine improved results on the tasks its authors tested, but its feedback is the model's own opinion. Here the feedback is the test runner's output, a missing citation or a list of files: evidence the worker cannot argue with.

How do you swap the stub for a real model?

Replace the body of worker(). The function build_prompt() already assembles what a model needs: the task, the files it may change, the rule for change notes, the current files and, after a failure, the exact failure messages. For a failed first attempt on slugify-accents, the end of that prompt reads:

Task: Make slugify() turn accented letters into plain ones, as issue.txt asks.
You may change only: ['CHANGES.md', 'slugify.py'].
In CHANGES.md, end every line with (source: file:line).
Return JSON: {path: new file content}.

[current files here]

Your last attempt failed these checks. Fix only these:
tests: exit code 1: FAIL: test_accents (test_slugify.T.test_accents) | AssertionError: 'cr-me-br-lee-montreal' != 'creme-brulee-a-montreal'
sources: no source line for claim '- Slugs now handle accented letters.'

Send that prompt to any model you use, local or hosted, and return json.loads() of its reply. Add a guard for replies that are not valid JSON: treat them as a failed attempt with the message "reply was not valid JSON", so they count toward the cap like any other failure. Keep the gates, the cap and the approval step exactly as they are; they are the part that makes the loop trustworthy, and none of them depends on which model writes the drafts.

Two settings deserve a second look once a real model is in place. Three attempts is a starting point: if most units that pass do so on the first or second try, a higher cap mostly buys more cost on units that will not pass. And the 60-second timeout should fit your slowest real test suite, with some margin, so a hanging test fails the attempt instead of stalling the run.