# Should a small business fine-tune a model or build a RAG system?

Date: 2026-04-25 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/small-business-fine-tune-model-build-rag/

**Answer:** For most small businesses, start with RAG, because the usual need is for answers based on your own facts, and those facts change. Fine-tune only when you have a solid set of clean, consistent examples and a problem of style or format that clear instructions and retrieval have not solved.

Both approaches make a general AI model more useful for your business, but they change different things. **Fine-tuning** means training an existing model further on your own examples, so it picks up a style, a format or a way of handling certain requests. **RAG**, short for retrieval-augmented generation, leaves the model unchanged: before each answer, your system searches your own documents and passes the most relevant passages to the model along with the question.

A simple way to remember the difference: fine-tuning changes *how* the model answers, and RAG changes *what it knows* at the moment it answers. The researchers who named RAG in 2020 pointed out the core weakness it fixes: models store facts inside their parameters, but they cannot easily update those facts or show where an answer came from.

## What each one is good at

| Need | Better fit | Why |
|---|---|---|
| Answers based on your prices, policies, hours, products | RAG | Edit a document and the next answer uses it |
| Showing which page an answer came from | RAG | The retrieved passage can be cited |
| A consistent tone or fixed reply structure | Fine-tuning (or instructions first) | Learned from many examples |
| Sorting tickets into your own categories | Fine-tuning | A narrow, repeated task with clear right answers |
| Facts that change every month | RAG | A fine-tuned model would need retraining each time |

OpenAI describes fine-tuning as a way to shape the structure and tone of responses, or to teach a model to follow complex, domain-specific instructions. It does not recommend fine-tuning as the way to load a product catalog into a model. Its help pages describe RAG for exactly that purpose, with the example of a support assistant that answers better when it can look up past tickets about similar problems.

## A worked decision: a small outdoor-gear shop

The shop has two staff members answering email. Most questions are about order status, return rules, sizing, and repairs. It has a 40-page help center, a returns policy that changes each season, and about two years of answered emails. The goal is an assistant that drafts replies for staff to approve.

**Step 1. Sort a sample of real questions.** Take 50 recent emails and label each one. Illustrative result:

| Type of question | Share | What the answer depends on |
|---|---|---|
| Returns and exchanges | 40% | Current policy |
| Sizing and product details | 30% | Product pages |
| Order status | 20% | The order system (a tool, not AI knowledge) |
| Complaints | 10% | Judgment and tone |

**Step 2. Ask what fails today.** Paste a few questions into a general assistant with your policy text included, and look at the drafts. If the facts are wrong or missing, you have a knowledge problem: that points to RAG. If the facts are right but the tone is off, you have a style problem: try better instructions first.

**Step 3. Build the smallest RAG version.** OpenAI's suggested pipeline is short: split each help article into chunks, turn each chunk into an **embedding** (a list of numbers that captures its meaning), store them, and at question time send the closest chunks to the model with the question. Many no-code tools, including custom GPTs with uploaded files, do this for you.

**Step 4. Write the instructions.**

```prompt
You draft email replies for an outdoor-gear shop. Use only the policy
and product passages provided. If they don't answer the question, say
that a staff member will follow up, and don't guess. Keep replies under
120 words, friendly and plain. End with the name of the page you used.
```

**Step 5. Decide on fine-tuning later, with evidence.** After a month, if staff still rewrite most drafts for tone, the two years of answered emails could become training examples. OpenAI says useful fine-tuning results can come from a few dozen well-chosen examples, but they must be clean and consistent.

For this shop, the answer is RAG now, a tool connection for order status, and fine-tuning only if a measured tone problem remains.

> **Try it:** Pick 20 questions your customers asked this month. Next to each, write whether a correct answer needs a *fact from your documents* or a *style you could describe in a sentence*. If most need facts, you have your answer.

## When fine-tuning is the right call

Fine-tuning earns its place when the task is narrow and repetitive, the right output is clear, and you have many examples: tagging tickets, filling a fixed form from messy text, or matching a house style that resists instructions. Keep in mind that your training data needs the same privacy care as any customer data.

## What does each cost to maintain?

Prices change often, so check the current pricing page of whichever provider you use. What differs most between the two approaches is the *work* involved after launch.

**RAG** costs you document upkeep and small usage fees. Each time a policy or product changes, you update the source text and the index refreshes, often automatically. Each question costs a little more than a plain prompt, because the retrieved passages are sent along with it. The ongoing task is reviewing wrong answers and fixing the document behind them.

**Fine-tuning** costs you a training run each time the examples change, plus the work of preparing and cleaning those examples. Using the fine-tuned model is then billed per use at its own rate, listed separately on the provider's pricing page. And when a provider releases a better base model, you may have to train again to benefit from it.

For a small team, the hidden cost is attention. A RAG system needs someone to own the documents; a fine-tuned model needs someone to own the training set and to retest after every change. Pick the one you can realistically keep up to date.

## Sources

- Retrieval Augmented Generation (RAG) and Semantic Search for GPTs: https://help.openai.com/en/articles/8868588-retrieval-augmented-generation-rag-and-semantic-search-for-gpts
- Fine-tuning now available for GPT-4o: https://openai.com/index/gpt-4o-fine-tuning/
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401v1