# How do you evaluate a RAG system?

Date: 2024-12-24 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/evaluate-rag-system/

**Answer:** Test the two halves separately. First check whether retrieval finds the right passages for a fixed set of questions, then check whether the answers stick to those passages, and have a person review a sample to confirm the automatic scores.

A RAG system, short for retrieval-augmented generation, answers questions in two stages. A search step pulls the most relevant passages from your documents, and a language model writes an answer using those passages. The idea comes from a 2020 research paper that paired a model's built-in knowledge with a searchable store of text, so answers could be more specific and easier to trace back to a source.

Because there are two stages, there are two ways to fail. The search can miss the passage that holds the answer, or the model can ignore a good passage and write something it made up. A single "is the answer good?" score hides which one happened. A useful evaluation measures each stage on its own, with the same questions every time, so you can tell whether a change helped.

## Step 1: Build a question set

Start with 30 to 50 real questions, ideally taken from what users actually ask. For each one, write down which document and passage holds the answer, and a short note on what a correct answer must contain. Include a few questions your documents can't answer; a good system should say so instead of guessing.

Here is a small sample for a town library's help chatbot. The documents and questions are illustrative.

| # | Question | Where the answer lives | Must contain |
|---|---|---|---|
| 1 | How many books can I borrow at once? | Lending policy, section 2 | The limit, and that it applies per card |
| 2 | Can I renew a book online? | Account help page | Yes, and how many times |
| 3 | Do you lend telescopes? | Not in the documents | "I don't know" or a referral |

Save the set in a spreadsheet or a file and never edit it casually. It is your ruler; if it changes, old and new scores can't be compared.

## Step 2: Measure retrieval hit rate

Run every question through the search step alone and record which passages come back. The hit rate is the share of questions where the correct passage appears among the top results, for example the top 5. In the illustrative run below, the system found the right passage for 41 of 47 answerable questions, a hit rate of 87%.

Look closely at the misses. Often a passage loses its meaning once it is cut out of its document: a line like "the limit is ten items" doesn't say ten items of what. Anthropic reported in September that adding a short description of where each chunk comes from, and combining meaning-based search with plain keyword search, cut its retrieval failures roughly in half on its test sets. Changes like chunk size, chunk overlap and the number of passages returned are all worth testing against your hit rate.

## Step 3: Check faithfulness

Faithfulness asks a narrow question: is every claim in the answer supported by the passages the system retrieved? It is not the same as being true in general. An answer can be correct from the model's memory and still be unfaithful, which matters because you can't trace it.

A 2023 framework called RAGAS made this measurable without writing reference answers. A language model breaks the answer into short statements, then checks each statement against the retrieved text. The score is the supported statements divided by all statements. You can run the same idea with any capable model and a grading prompt:

```prompt
You are checking an answer against source passages.
1. Split the ANSWER into short, single-fact statements.
2. For each statement, say SUPPORTED if the PASSAGES state or clearly imply it, otherwise NOT SUPPORTED. Give a one-line reason.
3. End with: score = supported / total.

PASSAGES:
[paste the retrieved passages]

ANSWER:
[paste the system's answer]
```

When the RAGAS authors compared this kind of automatic check with human judges, the two agreed on faithfulness 95% of the time, which makes it one of the more trustworthy automatic scores.

> **Try it:** Pick ten questions your own documents can answer, paste each answer and its retrieved passages into the grading prompt above, and record the scores in a spreadsheet. Then check two of them by hand and see whether you agree with the grader.

## Step 4: Add human review

Automatic graders are fast but not final. Each time you evaluate, have a person read a sample, say ten answers, with the passages beside them. Ask three things: Is it correct? Is it supported by the passages? Would a real user find it helpful? Note every disagreement with the automatic score. If the grader and the human often disagree, fix the grading prompt before trusting its numbers.

Then make it a habit. Run the whole set after every change to documents, chunking, search settings or the model, and keep a simple log of the scores. A drop tells you which stage broke.

## Which measures matter most?

Start with retrieval hit rate, because nothing downstream can fix a missing passage. If the right text isn't retrieved, the model either refuses or invents, and no prompt will save it. Hit rate is also cheap to measure: it needs no language model at all, only your question set and the expected passages.

Next comes faithfulness, the measure that protects users from confident made-up answers. In the RAGAS study, it was also the easiest dimension for an automatic grader to judge reliably. Answer relevance (does the reply address the question asked?) and context relevance (are the retrieved passages focused, not padded with noise?) are useful second-tier measures; the same study found them harder to score automatically, so lean more on human review there.

Finally, keep one measure that only people can give: whether the answer actually helped. A system can score well on every number and still miss what users need, such as a renewal limit buried under three paragraphs. A small, regular human sample keeps the numbers honest.

## Sources

- RAGAS: Automated Evaluation of Retrieval Augmented Generation: https://arxiv.org/abs/2309.15217v1
- Introducing Contextual Retrieval: https://www.anthropic.com/news/contextual-retrieval
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401v1