How do automatic prompt optimizers work?

When you write a prompt by hand, you guess, test a few cases, and adjust. An automatic prompt optimizer does the same loop, only systematically and many more times. You give it three things: a task with inputs and outputs, a set of examples with the right answers, and a metric, a small function that says whether an answer is good. The optimizer then generates candidate prompts, runs them, scores them and keeps the best.

The best-known open-source tool for this is DSPy, from Stanford researchers. Their 2023 paper argued that prompt templates found by trial and error are fragile, and proposed writing the task as code and letting a "compiler" optimize the prompts toward a metric. A 2024 follow-up paper introduced MIPRO, an optimizer that improves both the written instructions and the worked examples inside a prompt.

What the optimizer actually changes

A prompt for a task like classification has two parts an optimizer can tune:

• The instruction, the sentence or paragraph telling the model what to do.
• The demonstrations, a few worked examples of input and correct output (also called few-shot examples).

MIPRO handles both. For demonstrations, it runs the task on your training examples and keeps runs that pass your metric as candidate examples. For instructions, it asks a language model to propose new wordings, informed by your data and your task. It then tests combinations on small random batches of examples and uses the scores to steer toward promising choices instead of trying everything. The DSPy documentation lists this as dspy.MIPROv2, next to simpler optimizers that only pick examples (dspy.BootstrapRS) or that fine-tune model weights (dspy.BootstrapFinetune).

A worked example: routing customer messages

Suppose a small online shop wants each incoming message labeled as order, return, product question or other. You have 200 past messages, each labeled by a person. You split them: 100 to train the optimizer, 100 kept aside to test it.

from typing import Literal
import dspy

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))

class Route(dspy.Signature):
    """Classify a customer message for the support team."""
    message: str = dspy.InputField()
    category: Literal["order", "return", "product question", "other"] = dspy.OutputField()

classify = dspy.Predict(Route)

# labeled = [("Where is my parcel?", "order"), ...]  # your 200 labeled messages
examples = [dspy.Example(message=m, category=c).with_inputs("message") for m, c in labeled]
trainset, testset = examples[:100], examples[100:]

def correct(example, pred, trace=None):
    return example.category == pred.category

def accuracy(program, data):
    return sum(correct(ex, program(message=ex.message)) for ex in data) / len(data)

print("before:", accuracy(classify, testset))

optimizer = dspy.MIPROv2(metric=correct, auto="light", num_threads=8)
tuned = optimizer.compile(classify, trainset=trainset)

print("after:", accuracy(tuned, testset))

Notice what you did not write: the prompt itself. The Route signature declares the input, the output and the allowed labels; DSPy turns that into a prompt, and the optimizer improves it. The auto="light" setting asks for a small, cheap search. DSPy's documentation estimates a typical simple run at around two US dollars and twenty minutes, and warns that large models or datasets cost more.

Suppose (illustrative numbers) accuracy on the test set goes from 78% before to 88% after. The tuned program now contains a reworded instruction and a handful of example messages with their labels, picked because they helped. You can print and read them.

Why it works, and its limits

The optimizer has one advantage over you: patience. It will test dozens of wordings and example sets on the same data and compare them fairly. The DSPy paper reported large gains over standard few-shot prompts on tasks such as math word problems and multi-step question answering, and MIPRO beat earlier optimizers on five of seven multi-stage programs in its tests.

But it only optimizes what you measure. If your metric checks the label and nothing else, the optimizer will not care about tone, length or safety. If your labeled examples are unrepresentative, say all from one season, the tuned prompt will fit that season. And the result is tied to the model you tuned it on; switch models and you should run the optimization again.

When should you trust one?

Trust the result when it holds up on data the optimizer never saw. That is why the example keeps a separate test set. A prompt that scores well on its own training examples has only shown it can fit them. The score on held-out examples is the one to report and compare.

Trust it more when the metric matches what you actually care about. For a classifier, exact label match is a fair metric. For open-ended writing, a metric is harder to define, and an optimizer can learn to please a weak metric rather than to write well. In that case, check samples by hand, or use a careful grading rubric and spot-check the grader too.

Read what it produced. The tuned instruction and examples are plain text. If an example contains a customer's personal details, or the instruction includes a strange rule that happens to raise the score, fix it before deployment.

Finally, keep the baseline. If the tuned prompt beats your hand-written one by a point or two on a small test set, the difference may be noise. Larger, clearer gains on a decent number of examples are worth adopting; small ones are a reason to collect more test data first.