How do you build a chatbot that answers from your own documents?

A language model knows only what was in its training data. The training text behind gpt-3.5-turbo and GPT-4 largely ends in September 2021. It has never seen your employee handbook, your product manuals or last month's meeting notes. You could try to teach it with fine-tuning, but OpenAI's own cookbook advises against using fine-tuning to add knowledge: it is better at teaching a style or a task than at reliable recall of facts.

The approach that works is simpler. Find the relevant pieces of your documents at question time and put them in front of the model, like letting a student sit an exam with open notes. A 2020 paper by Patrick Lewis and colleagues named the idea retrieval-augmented generation, or RAG. "Retrieval" means looking things up; "generation" means writing the answer.

The four steps

1. Chunk. Split each document into short sections that make sense on their own, such as a heading and its paragraphs, a few hundred words at most. The model can only read so much at once: about 4,096 tokens (roughly five pages) for gpt-3.5-turbo and 8,192 for GPT-4. A token is a word or a piece of a word.

2. Embed. Send each chunk to OpenAI's embeddings endpoint, for example the text-embedding-ada-002 model. An embedding is a long list of numbers that captures meaning, so two passages about the same idea get similar numbers even when they use different words. Store each chunk's text, its embedding and where it came from (file name and page). For a few thousand chunks a spreadsheet-like table is enough; for millions, use a vector database, a store built for this kind of search.

3. Retrieve. When a question arrives, embed it the same way and compare it with every stored chunk. The comparison is a score called cosine similarity: the closer to 1, the closer in meaning. Keep the top three to five chunks.

4. Answer. Build a prompt that contains the retrieved chunks, their sources and the question, and send it to the chat model with strict instructions.

Steps 1 and 2 happen once per document. Steps 3 and 4 happen for every question.

A worked example in Python

Here is a minimal version using the openai Python library and a small company handbook already split into chunks. It is a sketch to show the flow, not production code.

import openai, numpy as np

chunks = [
    {"text": "Employees receive 15 vacation days per year...", "source": "handbook.pdf, p. 4"},
    {"text": "Expense reports are due within 30 days...", "source": "handbook.pdf, p. 9"},
    # ... the rest of your chunks
]

def embed(text):
    r = openai.Embedding.create(model="text-embedding-ada-002", input=text)
    return np.array(r["data"][0]["embedding"])

for c in chunks:                      # step 2, run once
    c["vec"] = embed(c["text"])

def answer(question, k=3):
    q = embed(question)                # step 3
    best = sorted(chunks, key=lambda c: -np.dot(q, c["vec"]))[:k]
    context = "\n\n".join(f"[{c['source']}]\n{c['text']}" for c in best)
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
    ]
    r = openai.ChatCompletion.create(model="gpt-3.5-turbo", messages=messages, temperature=0)
    return r["choices"][0]["message"]["content"]

The handbook text and figures here are made up for illustration. The dot product of two embeddings is the similarity score from step 3, and a temperature of 0 keeps the wording as predictable as possible.

The system prompt is where you set the rules:

You answer employee questions using ONLY the context passages provided in the user message.
Rules:
1. If the answer is in the context, answer in at most four sentences and cite the source in square brackets, for example [handbook.pdf, p. 4].
2. If the context does not contain the answer, reply exactly: "I don't know based on the documents I have." Do not use outside knowledge, and do not guess.
3. If passages disagree, say so and cite both.

When answers go wrong

The cookbook gives a useful way to debug. When an answer is wrong, first print the passages the model received. If the right passage is missing, the retrieval step failed: try smaller or better-labeled chunks, or retrieve more of them. If the right passage was there and the model still got it wrong, the answering step failed: tighten the prompt or use a stronger model such as GPT-4.

Expect this setup to shine on direct lookups ("How many vacation days do I get?") and to struggle with questions that need several scattered passages combined ("Compare our travel and remote-work policies").

How do you make it say 'I don't know'?

Give it explicit permission, and an exact phrase. Models are trained to be helpful, so without instructions they tend to fill gaps with plausible-sounding text from their general knowledge. Rule 2 in the prompt above tells the model that "I don't know" is a correct answer, forbids outside knowledge, and fixes the wording. OpenAI's cookbook example does the same, with the fallback "I could not find an answer."

A fixed phrase also helps your code. You can detect it and respond differently, for instance by offering a link to a human contact or logging the question so you know which documents are missing.

Then test it on purpose. Keep a short list of questions your documents do not answer and run them after every change to your prompt or chunking. Also check retrieval itself: if the top passages are only loosely related, the model may stretch them into an answer. Showing the cited source with every reply lets users check it themselves, which is the strongest safeguard you have.