What are evals, and how do you test a prompt before you trust it?

Most prompts are judged on one try. You write a prompt, paste in an example, like the answer, and start using it. The trouble shows up on the fortieth input, the one with an unusual format or a tricky detail, when the model does something you never saw in testing. An eval, short for evaluation, is the habit that catches those failures early: a set of test cases you run every time, with a clear rule for what counts as a pass.

In March, when OpenAI released GPT-4, it also open-sourced OpenAI Evals, its framework for automatically testing model performance. OpenAI invited anyone to contribute evals that expose weak spots in its models. You do not need the framework to use the idea, though. A spreadsheet works.

How OpenAI's framework describes an eval

In the Evals documentation, an eval is two things: a dataset and a way of grading it. The dataset is a JSONL file, a plain text file with one JSON record per line. Each record has an input, which is the prompt, and for simple evals an ideal, the answer you expect:

{"input": [{"role": "system", "content": "Answer with the capital city only."}, {"role": "user", "content": "France"}], "ideal": "Paris"}

The framework then compares the model's answer with the ideal. Simple templates check for an exact match, check whether the answer includes the ideal text, or allow a fuzzy match. For answers with no single right wording, such as a summary, there are model-graded evals: a second model reads the answer and grades it against a rubric. OpenAI recommends testing the grader too, by checking its verdicts against labels people assigned by hand.

The documentation also lists what makes an eval worth having. It should stay on one theme, be hard enough that a strong model does not pass everything, have clear reference answers or a rubric, and be spot-checked by a person before anyone trusts the score.

A worked example: a 20-case eval for a summary prompt

Say a small nonprofit uses this prompt to turn long donor emails into notes for its database:

Summarize the email below in exactly 3 bullet points for our donor database. Include the donor's request and any date or amount mentioned. Do not add information that is not in the email.

Email: """{email}"""

Instead of trying it on one email, collect 20 real ones (with names removed) and build a sheet. Choose them to cover the range: short and long, polite and angry, emails with two dates, emails with no request at all, one written in French. For each, write down what a correct summary must contain. Then run the prompt on all 20 and grade each answer against three checks.

The rows and results above are illustrative.

A case passes only if all three checks say yes. If 15 of 20 pass, the prompt scores 75 percent. Now the useful part begins. Look at the failures together: if they cluster around emails with two dates, add a line to the prompt about choosing the date the donor asks about. Then rerun all 20 cases, not just the ones that failed, because a fix for one problem can break something that used to work.

What evals change in practice

Evals turn opinions into numbers you can compare. They let you answer questions like "Is the new prompt better?" or "Does the cheaper model do this task well enough?" with evidence. They also give you a safety net when something changes outside your control, such as a model update: rerun the sheet and compare scores. Keep the test set fixed between runs; if you change the cases, start a new version so you are not comparing different tests.

How many test cases do you need?

Enough to cover the kinds of input you really get, and enough that one case does not swing the score wildly. OpenAI's contribution rules for its Evals repository set a floor: since April, a contributed eval must include at least 15 high-quality examples. That is a reasonable minimum for a personal eval too.

Twenty is a practical place to start, because each case is worth 5 percent and the arithmetic stays easy. Choose the cases deliberately rather than at random: include the common inputs, the strange ones, and every failure you have already seen in real use. A test set that only holds easy cases will give you a flattering score that means little.

Grow the set when you need finer distinctions. If two prompt versions score 80 and 85 percent on 20 cases, that is a single case of difference, which could easily be chance. To trust a small improvement, you need more cases, often a hundred or more. And every time the prompt fails on a real input, add that input to the set. Over a few months, the eval becomes a record of everything your prompt has to get right.