How do you set up a knowledge base an AI can search?

A language model knows a great deal in general and nothing about your business in particular. It has never read your return policy, your price list or last month's procedure change. A searchable knowledge base fixes that. When someone asks a question, the system first looks up the most relevant passages in your documents, then hands them to the model along with the question, so the answer rests on your material rather than on a guess.

This approach is called retrieval-augmented generation, or RAG: retrieval (finding the right passages), augmented (adding them to the prompt), generation (the model writing the answer). The model itself doesn't change. What changes is what it reads just before it answers, which is why the quality of your documents matters more than any setting.

How AI search finds the right passage

Ordinary search matches words. If a customer asks "can I bring it back?" and your policy says "returns are accepted within 30 days," a keyword search may find nothing. AI knowledge bases use semantic search instead, which matches meaning.

To make that possible, each piece of text is turned into an embedding, a long list of numbers that captures what the text is about. Texts with similar meanings get similar numbers. These embeddings are stored in a vector database (a database built to compare those number lists quickly). When a question arrives, it gets an embedding too, and the system pulls out the pieces whose numbers are closest. Those pieces, not the whole library, go to the model.

A worked plan in five steps

Picture a small bicycle shop that wants a chatbot for staff and customers. Here is the plan, step by step.

1. Collect. List the questions people actually ask, then gather the documents that answer them: the returns and warranty policy, the service menu and prices, store hours, a page on bike sizing, and the 30 most common email replies. Leave out drafts, outdated versions and anything private, such as supplier contracts or staff records. Everything you include can surface in an answer.

2. Clean. Convert each document to simple, single-column text. Text extraction tools often stumble on multi-column PDFs, tables laid out as images and slides where meaning depends on position. Remove repeated headers and footers, fix headings so each section says what it covers, and put a "last updated" date at the top of every file.

3. Chunk. Split the text into pieces small enough to be precise but large enough to make sense alone. A paragraph or a short section is a good starting point. Splitting by heading usually works better than cutting every fixed number of words, because each piece then covers one topic. Different chunking strategies give noticeably different results, so test a few.

4. Embed and store. Run each chunk through an embeddings model, such as OpenAI's Embeddings API, and store the chunk, its embedding and a little metadata (source file, section title, last-updated date) in a vector database. The metadata lets the bot say where an answer came from and lets you replace a document cleanly later.

5. Test. Write 20 real questions with the answers you expect, including tricky ones ("Do you service e-bikes?" when the answer is "only some brands"). Run them, and for each wrong answer check whether the right chunk was retrieved. Most failures trace back to steps 2 and 3, not to the model.

Here is an illustrative test sheet:

The second row shows an outdated file; the third shows a missing one. Both are content problems.

The no-code route: GPTs and Projects

If your collection is modest, you may not need to build anything. A custom GPT in ChatGPT accepts up to 20 files, each up to 512 MB, and does the chunking and embedding for you when you upload them. It then picks between searching for relevant passages and reading a whole short document, depending on the question. OpenAI advises simple formatting for best results, and notes that GPTs hide file names by default, so add an instruction such as "Name the document you used" if you want sources shown.

Claude Projects, available to Claude Pro and Team users since June 2024, take a different approach. Each project holds up to 200,000 tokens of material, which Anthropic compares to a 500-page book, plus custom instructions. For a small shop's policies and price lists, that is often enough to load everything at once.

How do you keep it current?

Give every document an owner and a review date. A knowledge base goes stale quietly: nobody notices until the bot confidently quotes last year's price. A simple schedule helps, such as prices and hours checked monthly, policies each quarter, and everything else twice a year, plus an immediate update whenever something changes.

Replace, don't pile up. When a policy changes, remove the old file and its chunks before adding the new one. If both versions stay in the database, the search may retrieve either, and the answer will be a coin toss. This is where the source and date metadata from step 4 pays off: you can find and delete every chunk from the old file.

Know the limits of the no-code tools. In a custom GPT, files are managed by hand in the builder, so OpenAI suggests it for material that changes rarely, such as handbooks and policies. Fast-changing information, like stock levels or daily specials, belongs in a system that pulls from the live source rather than in uploaded files.

Finally, rerun your test questions after every significant update, and add each new question that the bot gets wrong. Over time that list becomes the best measure of whether your knowledge base is still telling the truth.