# What makes a check in an AI loop real?

Date: 2026-07-14 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/what-makes-ai-check-real/

**Answer:** A check is real when it can fail the work on its own, with nobody watching, using evidence the worker cannot edit or argue with, such as a test that exits with an error, a claim with no matching source line, or a change that touches files outside the plan. "Looks good", "the model is confident" and "no errors were raised" are not checks, because they pass almost everything.

An AI loop repeats four moves: produce some work, check it, correct what failed, and try again. Everything depends on the second move. If the check cannot fail, the loop is just the model producing output several times, and the extra rounds add cost without adding correctness.

A **real check** has three properties. It can fail the work by itself, without a person reading every result. It gets its verdict from something the worker does not control: a test run, a source document, a list of allowed files. And when it fails, it says what failed, so the correction can be aimed at the right spot. The rest of this article shows what that looks like in practice, and which popular "checks" do not qualify.

## Three questions to ask of any check

**1. Have you seen it fail?** In test-driven development, a programming practice in which you write a failing test before the code that makes it pass, the first run of a new test is expected to fail. That first failure proves the test is actually able to catch something; a test that passes before any code exists is testing nothing. The same goes for any check in an AI loop. Feed it a deliberately bad output (a wrong answer, a made-up citation, an edit to a forbidden file) and confirm it says no.

**2. Can the worker get around it?** If the model that does the work can edit the check, skip it, or persuade it, the check will eventually stop working. OpenAI researchers documented this in a March 2025 paper on reasoning models trained to make unit tests pass. The models found shortcuts: calling `exit(0)` so the test run ended early and looked successful, raising a "skip this test" exception, writing empty stand-ins for functions the tests barely covered, and editing test files so they passed trivially. Their definition of cheating is itself a good check: a solution counts as cheating if it passes the tests but fails once the agent's edits to the tests are undone.

Anthropic's work on long-running coding agents points the same way. Its agents worked from a list of features, each marked failing at the start, and the instructions allowed them to change only the pass or fail status, not to remove or rewrite the tests. Without end-to-end testing, the agents tended to mark features done that did not work.

**3. Does a failure carry evidence?** "Failed" alone gives the next attempt nothing to work with. "Test `test_refund_rounding` expected 10.05, got 10.04" does. The Reflexion paper (Shinn and colleagues, 2023) built its method on this: test results and error messages are turned into a short written note about what went wrong, which the agent reads before its next try. In their experiment on hard Rust problems, removing the tests made the agent worse than no loop at all (52% against a 60% baseline), because it could no longer tell when its code was right and kept making harmful edits.

## Non-checks and what to use instead

These are the most common things that look like checks but are not.

| Non-check | Why it is not a check | Real check instead |
|---|---|---|
| The model rereads its work and says it looks good | A model judging its own answer tends to approve it; without outside feedback, self-correction can even make answers worse | A separate check with outside evidence: tests, sources, a schema |
| "I am 95% confident" | The number comes from the same model that did the work, not from the work | A check whose result does not depend on what the model says about itself |
| No errors were raised | Silence is not success; a script can exit cleanly after doing nothing | Assert on the result: expected rows, expected files, expected values |
| The tests pass (tests the worker could edit) | The worker can change the test instead of the code | Run the original tests, from a copy the worker cannot touch |
| The tests pass (tests the worker wrote itself) | Weak tests pass wrong answers | Add tests written by someone else, including some the worker never sees |
| A model judge gave it 8 out of 10 | Scores drift and judges have known biases | A pass or fail answer per written criterion, checked against human labels |

The first row rests on two findings. An October 2023 paper by Jie Huang and colleagues at Google DeepMind, titled to say that models cannot yet self-correct their reasoning, found that when models revised their own reasoning answers with no outside feedback, accuracy did not improve and sometimes dropped. And in Anthropic's 2026 experiments with long coding sessions, agents asked to grade their own work tended to praise it; a separate, deliberately skeptical evaluator did better.

The fifth row comes from Reflexion again. Its coding agent wrote its own unit tests, at most six per problem. When those tests were weak, they passed wrong solutions, and the agent stopped and submitted them. On one benchmark, 16.3% of solutions that passed the agent's own tests were wrong anyway, against 1.4% on another, and the authors connect that gap to the method doing worse than the plain model on the first benchmark.

## Real checks by kind of work

Every kind of work has checks that can fail without a person. The trick is to write them down before the loop starts.

| Work | Check that can fail | Evidence it returns |
|---|---|---|
| Code | The project's tests pass, run from an unmodified copy of the test files | Names of failing tests and their error output |
| Code | The change touches only files listed in the plan | Files changed outside the plan |
| Code | Linter and type checker exit with no errors | File, line and rule |
| Writing | Length, required terms and banned phrases | Which limit or term failed |
| Writing | Every number and name in the text also appears in the brief or notes | The unmatched numbers and names |
| Writing | Every link returns a working page | The broken links |
| Research | Every claim has a source and a quote, and the quote is found in that source's saved text | Claims with no source, or quotes not found |
| Research | Every source is dated on or before the cutoff date | Sources that are too new |
| Data | The output matches its schema (columns, types, no empty keys) | Rows and fields that fail |
| Data | Totals match the source system, row counts match the input | The difference and where it appears |

Notice what these have in common. Each one is written in code or as a fixed rule, each one runs the same way every time, and none of them asks the worker's opinion.

## A worked example: every claim needs a source line

Research and writing loops often fail in the same way: a confident sentence with nothing behind it. The check below catches that. The worker must return its claims as a list, each with a source id and a short quote copied from that source. A small script then looks for each quote in the saved text of the source and fails the run if any claim is missing a source or a quote cannot be found.

```python
import json, sys
from pathlib import Path

def norm(text):
    return " ".join(text.split()).lower()

claims = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
failures = []
for i, c in enumerate(claims, 1):
    src = Path("sources") / f"{c.get('source', '')}.txt"
    if not c.get("source") or not c.get("quote"):
        failures.append(f"claim {i}: no source or quote: {c['claim'][:60]}")
    elif not src.exists():
        failures.append(f"claim {i}: unknown source '{c['source']}'")
    elif norm(c["quote"]) not in norm(src.read_text(encoding="utf-8")):
        failures.append(f"claim {i}: quote not found in {c['source']}")

for f in failures:
    print("FAIL", f)
print(f"{len(claims) - len(failures)}/{len(claims)} claims supported")
sys.exit(1 if failures else 0)
```

Run against three claims about a factory, with one saved source that says it "opened in 1998 and employs 240 people", it prints:

```
FAIL claim 2: quote not found in annual-report
FAIL claim 3: no source or quote: It is the region's largest employer.
1/3 claims supported
```

The exit code is 1, so the loop treats the work as failed and sends back claims 2 and 3, with those two lines as the evidence. The worker never gets to say whether the check passed.

To get claims in that shape, ask for them directly:

```prompt
List every factual claim in your draft as JSON: an array of objects with
"claim" (the sentence), "source" (the id of one file in the sources folder)
and "quote" (words copied exactly from that source that support the claim,
at most 25 words). If a claim has no supporting source, keep it in the list
with "source": "" and "quote": "". Do not paraphrase inside "quote".
```

Be clear about what this check does not do. It proves the quote exists in the source; it does not prove the quote supports the claim. That second question needs either a stricter rule (the number in the claim must appear in the quote, for instance) or a judge, which brings us to the follow-up question.

> **Try it:** Take one loop or automation you already run and write down its checks. For each one, feed it a deliberately bad output and see whether it fails. Any check that passes the bad output goes in the "non-check" column until you fix it.

## Can an AI judge ever count as a real check?

Yes, for qualities that code cannot measure, such as whether a summary misrepresents its source or whether a reply answers the question asked, as long as the judge is set up to be able to fail the work. Research on model judges, such as the 2023 MT-Bench study by Zheng and colleagues, found strong models agreeing with human preferences about as often as humans agree with each other, and also documented their biases: favoring whichever answer comes first, favoring longer answers, and possibly favoring their own writing.

That suggests a few rules. Ask the judge for a pass or fail on each written criterion rather than an overall score. Use a different model, or at least a different prompt and role, from the one that did the work, and tell it its job is to find problems. When comparing two versions, run the comparison twice with the order swapped and count a win only if both runs agree. Before trusting the judge, score a sample of outputs yourself and compare its verdicts with yours; if you disagree often, fix the rubric first.

Even then, put a judge beside the hard checks, not in place of them. Tests, schemas and source matching cannot be talked out of a failure. A judge, however well prompted, is still a model reading text, and on any step where a mistake would be costly, its pass should lead to a person's review rather than straight to the action.

## Sources

- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation: https://arxiv.org/abs/2503.11926v1
- Reflexion: Language Agents with Verbal Reinforcement Learning: https://arxiv.org/abs/2303.11366v4
- Large Language Models Cannot Self-Correct Reasoning Yet: https://arxiv.org/abs/2310.01798v1
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: https://arxiv.org/abs/2306.05685v4
- Harness design for long-running application development: https://www.anthropic.com/research/harness-design-long-running-apps
- Effective harnesses for long-running agents: https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents
- Test-driven development (Wikipedia, revision of 23 March 2026): https://en.wikipedia.org/w/index.php?title=Test-driven_development&oldid=1344926507

---

From the Guide to Artificial Intelligence, by Robert Scales, published by Holistic Hybrid Inc. Share and adapt it under CC BY-NC-SA 4.0 (https://creativecommons.org/licenses/by-nc-sa/4.0/): credit the guide, link to https://guidetoartificialintelligence.com/en/learn/what-makes-ai-check-real/, no commercial use without permission, and keep the same licence.