# How do you set up a self-improving agent loop safely?

Date: 2026-02-08 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/set-up-self-improving-agent-loop-safely/

**Answer:** A self-improving loop lets a model propose a better version of an agent's prompt, scores each version on a fixed set of test cases, and keeps the best one. Make it safe by limiting what it can change to the prompt, capping rounds and spending in code, logging every attempt, and requiring a person to approve any new version before it goes live.

"Self-improving" sounds grander than what most working systems do. In the published examples, an agent does not rewrite its own code or grant itself new powers. A separate step reads feedback on the agent's recent outputs, drafts a revised **prompt** (the instructions the agent runs on), and tests that draft. If the draft scores better on a fixed set of checks, it becomes a candidate to replace the old prompt. Then the cycle repeats.

OpenAI's cookbook calls this a self-evolving agent and walks through it with a summarizing agent for regulatory documents. Anthropic describes the same shape as an **evaluator-optimizer** workflow: one model call produces work, another judges it and gives feedback, round after round. Both stress that it works only when you have clear criteria for "better", and both add limits and human checks around it.

## The four parts of the loop

1. **A baseline agent** with a versioned prompt, so every change can be undone.
2. **An eval set**: a few dozen realistic inputs, each with checks the output must pass. An **eval** is simply an automated test for model output.
3. **Graders** that score each output. Some are plain code (is it under 120 words? does it mention the order number?); one can be an **LLM-as-judge**, a model that scores against a written rubric. OpenAI's example combines four graders, two in Python, one measuring similarity to the source text, and one rubric-based judge, each with its own pass threshold.
4. **An optimizer** that reads the failures and proposes a revised prompt.

The loop stops when the score passes a target or when it runs out of attempts. In the cookbook, the example target is 0.8 and the example limit is ten retries; reaching the limit alerts engineers that a person needs to step in.

## A worked example

The sketch below improves the prompt of a summarizing agent. It uses the `openai` Python library's Responses API. Prices, thresholds and the model choice are placeholders to set from your own account and tests.

```python
import json, time
from openai import OpenAI

client = OpenAI()                       # reads OPENAI_API_KEY
MODEL = "gpt-5-mini"                    # any model your account offers
MAX_ROUNDS = 5                          # hard cap on improvement rounds
MAX_COST_USD = 2.00                     # hard cap on spend for the whole run
TARGET = 0.85                           # average score needed to propose a change
PRICE_IN, PRICE_OUT = 0.25e-6, 2.00e-6  # USD per token: illustrative, use your price list
spent = 0.0

def call(instructions, text):
    global spent
    r = client.responses.create(model=MODEL, instructions=instructions, input=text)
    spent += r.usage.input_tokens * PRICE_IN + r.usage.output_tokens * PRICE_OUT
    if spent > MAX_COST_USD:
        raise RuntimeError(f"Budget reached: ${spent:.2f}")
    return r.output_text

def grade(case, summary):
    checks = [len(summary.split()) <= 120]
    checks += [term.lower() in summary.lower() for term in case["must_mention"]]
    return sum(checks) / len(checks)

def evaluate(prompt, cases):
    results = []
    for case in cases:
        summary = call(prompt, case["text"])
        results.append({"id": case["id"], "score": grade(case, summary), "summary": summary})
    return sum(r["score"] for r in results) / len(results), results

def log(entry):
    with open("loop_log.jsonl", "a", encoding="utf-8") as f:
        f.write(json.dumps(entry) + "\n")

cases = json.load(open("eval_cases.json"))   # [{"id", "text", "must_mention": [...]}]
prompt = open("prompt_v1.txt").read()
best_score, results = evaluate(prompt, cases)
log({"round": 0, "score": best_score, "prompt": prompt})

for rnd in range(1, MAX_ROUNDS + 1):
    if best_score >= TARGET:
        break
    failures = [r for r in results if r["score"] < 1][:5]
    candidate = call(
        "You improve instructions for a summarizing assistant. Return only the revised "
        "instructions. Do not add tools, permissions or new tasks.",
        f"Current instructions:\n{prompt}\n\nFailed cases:\n{json.dumps(failures)[:6000]}")
    score, cand_results = evaluate(candidate, cases)
    log({"round": rnd, "score": score, "spent": round(spent, 4), "prompt": candidate, "time": time.time()})
    if score > best_score:
        prompt, best_score, results = candidate, score, cand_results

with open("prompt_candidate.txt", "w", encoding="utf-8") as f:
    f.write(prompt)
print(f"Best score {best_score:.2f}, spent ${spent:.2f}. Review prompt_candidate.txt before deploying.")
```

Three design choices carry the safety. The optimizer can change only the wording of one prompt, and its own instructions forbid adding tools or tasks. Every round, with its score and cost, lands in `loop_log.jsonl`. And the script never deploys anything: it writes `prompt_candidate.txt` and stops, so a person compares it with the current version and reads a sample of outputs before switching. The cookbook makes the same point, calling its example simplified and saying a real system should have guardrails on optimization attempts and a human approving new prompts.

> **Try it:** Build an eval set of 15 inputs from your own work, run only round 0, and read the five lowest-scoring outputs yourself. If you disagree with the grader's scores, fix the graders before you let any optimizer near the prompt.

## What limits stop it from running away?

**A round limit and a spending cap, enforced in code.** `MAX_ROUNDS` and `MAX_COST_USD` above stop the loop no matter what the model writes. Anthropic notes that maximum-iteration limits are a common way to keep agents under control, and that autonomy brings higher costs and errors that compound. Add a monthly usage limit in your provider's account as a second layer.

**An eval gate.** A new version is considered only if it beats the old one on the same tests. Keep a few cases the optimizer never sees, and check the winner on those too, so the prompt does not simply learn to please the graders.

**A narrow scope.** Let the loop change prompts, not code, tools, permissions or its own limits. Those stay with people.

**Version history and rollback.** The cookbook keeps every prompt version so a regression can be reversed. The log above serves the same purpose.

**A person at the end, and an alert in the middle.** When the loop hits its limit without reaching the target, or a run is stopped by the budget, notify someone rather than retrying. Anthropic's guidance on agents makes the same case: test in a sandbox, add checkpoints where a person can review, and keep human judgment on anything that ships.

## Sources

- Self-Evolving Agents: A Cookbook for Autonomous Agent Retraining: https://github.com/openai/openai-cookbook/blob/b3ee6e549d35e4c3de372375ed5374ae75c79992/examples/partners/self_evolving_agents/autonomous_agent_retraining.ipynb
- Building effective agents: https://www.anthropic.com/engineering/building-effective-agents