Should you improve the prompt, give the AI your documents, or fine-tune a model?

When an AI tool gives you disappointing answers, there are three main ways to improve it. You can improve the prompt: clearer instructions, a few examples, a better structure. You can give it your documents, either by pasting them in or through retrieval-augmented generation (RAG), where a search step finds the relevant passages and adds them to the prompt. Or you can fine-tune a model: train it further on hundreds of your own examples so the behavior is built in.

These are not three rungs of one ladder. OpenAI's guide to improving accuracy describes them as levers that fix different problems, and the useful question is not "how advanced should I go?" but "what exactly is wrong?"

Two kinds of problem

OpenAI's guide sorts failures into two families, with an exam analogy that is worth keeping in mind.

• The model doesn't know something. Your prices, your policies, last week's decisions, a client's file. That is a knowledge problem. The fix is context: give it the textbook during the exam. In practice that means pasting documents, attaching files, switching on a connector, or building RAG.
• The model knows enough but behaves wrongly. The format drifts, the tone is off, it skips a step in your method, it answers in three paragraphs when you need one line. That is a behavior problem. The first fix is still the prompt (instructions and examples), and fine-tuning becomes worth considering only when the prompt can't make it consistent. Fine-tuning is like having studied before the exam.

Many real problems are a mix, and the two fixes stack: a fine-tuned model can still use retrieved documents.

What changed by 2026

Two trends shift the balance for ordinary users.

First, context windows are large and connectors are common. Anthropic's advice since 2024 has been that if your material is under about 200,000 tokens, roughly 500 pages, you can often just include all of it in the prompt with no retrieval system at all. Today's assistants also let you attach a folder or switch on a connector to your drive, which is RAG done for you. For a small team, "give it the documents" usually means a setting, not a project.

Second, instructions travel better than they used to. Projects, custom instructions and reusable skills let you save a strong prompt with its examples and use it every time. Many problems that once seemed to call for fine-tuning, such as "always answer in our house style", are now solved by a saved prompt with three good examples.

Fine-tuning still has a place: high-volume tasks where you want a smaller, cheaper model to match a bigger one, or very specific output patterns that hundreds of examples teach better than any instruction. OpenAI's guide suggests starting with at least 50 high-quality examples that look exactly like real inputs, and keeping some aside to test the result.

A worked diagnosis

A small property-management office uses an assistant to answer tenant questions. Here are five failed answers and the fix each one points to.

Four of five are fixed without training anything. That proportion is typical.

Here are five answers from my AI assistant that I was unhappy with, each followed by what the correct answer should have been.
For each one, tell me whether the problem is missing knowledge (it lacked a fact) or behavior (it had the facts but handled them badly).
Then suggest the smallest fix: a prompt change, a document to provide, or, only if clearly needed, fine-tuning. Explain each choice in one sentence.

How do you decide in five minutes?

Take five recent answers you weren't happy with and ask one question of each: if a smart new employee had the right documents in front of them, would they have got this right? If yes, the model lacked knowledge, so give it the documents. If no, because the employee would also need to be told how you like things done, it is a behavior problem, so improve the prompt first.

Then apply three quick rules. If the documents fit in the assistant's file upload or a connector, use that before building any retrieval system. If a better prompt with two or three examples fixes most of the behavior failures, stop there. Consider fine-tuning only when you have dozens to hundreds of clean examples, the same task runs very often, and a prompt has already been tried and measured.

That last word matters. Before you change anything, keep the five failed cases and the answers you wanted. Run them again after each fix. Twenty such cases make a small test set, which OpenAI's guide treats as the right baseline before trying anything more advanced, and it turns "it seems better" into a count you can trust.