# How do you build an eval set from your own work?

Date: 2026-01-12 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/build-eval-set-own-work/

**Answer:** Collect about 30 real tasks you have already done, write down for each one the input and what a good answer must contain, and choose how each will be graded, by exact match, by a short rubric, or by a model acting as judge. Run every prompt or model change against the whole set and compare scores, keeping in mind that 30 examples detect large differences, not small ones.

An **eval** (short for evaluation) is a fixed set of test cases you run an AI system against to measure how well it does a job. Public benchmarks are evals built by researchers for general skills. An eval built from your own work answers a narrower and more useful question: does this prompt, model or agent do *my* tasks well, and did the change I just made help or hurt?

You do not need special software to start. A spreadsheet works, and a simple file format lets you move to code later. What matters is that the cases are real, that "good" is written down before you look at any output, and that you run the same set every time.

## Step 1: Collect 30 real tasks

Go through recent work and copy out 30 tasks you would want an AI to help with. Suppose you manage communications for a small nonprofit and want help turning board-meeting notes into a short public update. Pick:

- about 20 **typical** cases (ordinary meetings),
- about 7 **hard** cases (long notes, conflicting decisions, sensitive items that must stay private),
- about 3 **edge** cases (notes with no decisions, notes in shorthand, a meeting that was cancelled).

Spread them across different months and topics. If ten cases come from one long meeting, they are not ten independent tests, a point the next sections return to.

## Step 2: Write what a good answer must contain

For each case, write the **expected output**. Sometimes that is one exact answer, such as a category or a number. For writing tasks, it is usually a short list of must-haves and must-nots rather than a perfect text:

| Case | Must include | Must not include |
|---|---|---|
| 07 – March meeting | New program start date (April 14); budget approved | Names of the two staff members discussed in private session |
| 12 – June meeting | Board vote on the new office | Draft figures marked "not final" |

OpenAI's guidance for its open-source Evals framework names four qualities of a good eval: its cases share a theme, they are challenging, it is clear what a good answer looks like through reference answers or a rubric, and someone has checked the cases carefully. That last point matters more than it sounds: a wrong expected answer quietly penalizes a correct model.

## Step 3: Choose how each case is graded

There are three common grading methods, and one eval can mix them:

1. **Exact or contains match.** The output must equal, or include, a specific string. Fast, free and objective; good for dates, labels and numbers.
2. **A rubric scored by a person.** A short checklist, for example: all must-haves present (2 points), no must-nots (2 points), under 150 words (1 point). Slower, but trustworthy.
3. **A model as judge.** Another model scores the output against the rubric. The 2023 MT-Bench study found that strong models used as judges agreed with human preferences over 80% of the time, but also showed known biases: they favor longer answers, may favor the answer shown first, and can prefer their own writing. If you compare two versions with a judge, run it twice with the order swapped and count a win only when both runs agree.

## Step 4: Store it in a simple format

OpenAI Evals stores test cases in **JSONL**, a text file with one JSON object per line, using an `input` key for the prompt and an `ideal` key for the expected answer. Even if you never use that framework, the format is easy to read and to load from any language:

```json
{"id": "07", "input": [{"role": "user", "content": "Write a 150-word public update from these notes: ..."}], "ideal": ["April 14", "budget approved"], "must_not": ["private session"], "type": "typical"}
{"id": "12", "input": [{"role": "user", "content": "Write a 150-word public update from these notes: ..."}], "ideal": ["new office"], "must_not": ["not final"], "type": "hard"}
```

A ten-line script can then send each input to a model, check the `ideal` and `must_not` lists automatically, and save the outputs for the rubric review. Keep the file under version control, and when you change a case, note it, so old and new scores are not compared as if they were the same test.

> **Try it:** Open a blank spreadsheet with four columns: input, must include, must not include, case type. Fill in ten rows from last month's work before you run any model. You now have the start of an eval.

## Step 5: Use it every time something changes

Run the full set whenever you change the prompt, switch models or adjust an agent's tools. Record the score, and read every case that changed from pass to fail. Those few outputs usually teach more than the overall number.

## How many examples are enough to trust the result?

It depends on how small a difference you need to see. Anthropic's research on eval statistics recommends reporting a **standard error** with each score, a measure of how much it would move if you had drawn a different set of questions, and a 95% confidence interval of about 1.96 standard errors either side. For a pass rate, the standard error is the square root of p × (1 − p) ÷ n, where p is the pass rate and n the number of cases.

For a system that passes 80% of cases, that gives roughly ±14 percentage points with 30 cases, ±8 with 100 and ±4 with 400. So 30 cases can tell a 60% prompt from a 90% one, but not 80% from 84%. If two versions run on the same cases, compare them case by case, a **paired** comparison, which Anthropic notes gives a more precise answer than comparing two totals. And if many cases come from the same source, such as one long meeting, they count for less than their number suggests; Anthropic found that ignoring such clusters could understate the error by a factor of three or more.

A practical path: start with 30 to catch big problems, grow toward 100 as real tasks come in, and treat any difference smaller than the margin above as a tie.

## Sources

- Building an eval (openai/evals docs): https://github.com/openai/evals/blob/bd3b4d0afa7785f0374c46c32a32dd4c55105c28/docs/build-eval.md
- A statistical approach to model evaluations: https://www.anthropic.com/research/statistical-approach-to-model-evals
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: https://arxiv.org/abs/2306.05685v4