What is a vector database, and do you need one?

If you have tried to build a "chat with your documents" app this spring, you have met the phrase vector database within minutes. Tutorials pair ChatGPT with names like Pinecone, Chroma and pgvector as if they were required. They are useful, but they are not magic, and many small projects can skip them entirely.

The idea starts with embeddings. An embedding model turns a piece of text into a long list of numbers, a vector, so that texts with similar meanings end up with similar numbers. A vector database is software built to store those vectors and answer one question fast: which stored vectors are nearest to this new one?

How similarity search works

Ordinary databases are excellent at exact matches: find the order with number 4471, or every customer in Quebec. A keyword search finds documents containing the words you typed. Searching by meaning is different. A customer who asks "I can't get into my account" should find the help article titled "Resetting your password," even though they share no words.

Here is the process, step by step.

1. Split your documents into chunks, such as paragraphs or help articles.
2. Embed each chunk with an embedding model and store the vector, the original text and any labels (called metadata) together.
3. Embed the question with the same model.
4. Compare the question's vector with the stored ones and return the closest few.
5. Pass those chunks to a language model as context, so it answers from your material. This pattern is often called retrieval-augmented generation, or RAG.

Closeness is measured with a formula. The common choices are Euclidean distance (straight-line distance), cosine similarity (whether two vectors point the same way) and the dot product.

A worked example with tiny vectors

Real embeddings have hundreds or thousands of numbers. To see the idea, imagine an embedding with just three. The vectors below are invented for illustration.

The question "I can't get into my account" becomes [0.9, 0.1, 0.2]. Cosine similarity runs from -1 to 1, and higher means closer in meaning. Articles A and C come out on top, which is what a human would pick. B and D trail far behind.

With four articles, comparing the question to every one is instant. With ten million, checking each vector takes too long, so vector databases build an index for approximate nearest neighbor search (ANN). An ANN index finds very close matches without comparing every item, accepting a small chance of missing the exact best one in return for much more speed.

Do you actually need one?

Ask three questions.

• How many chunks? For a few hundred or a few thousand, you can keep the vectors in a file, load them into memory and compare against all of them with a short loop or a library such as NumPy. That is exact, free and easy to inspect.
• Does the collection change often? If users add documents all day, a database that handles adding, updating and deleting vectors saves you rebuilding a file.
• Do you need filters? Questions like "only search the French manuals from 2022" need metadata filtering alongside similarity, which vector databases provide.

If all three answers are small, start with a file. You can move to a database later without changing your embeddings, since the vectors are just numbers.

How do Pinecone, Chroma and pgvector differ?

They solve the same problem from three directions.

Pinecone is a hosted service: you send vectors and queries over an API, and Pinecone runs the indexing, splitting across machines and copies for reliability. You manage no servers, but your data lives with a third party, and beyond its free sign-up you pay for capacity.

Chroma is free under the Apache 2.0 license and installs with one pip install chromadb. It can run entirely in memory for quick experiments. By default it embeds text for you with a Sentence Transformers model, or you can plug in OpenAI or Cohere embeddings.

pgvector adds a vector column type to Postgres. You store embeddings next to your other data and find neighbors with a normal query sorted by distance. It does exact search by default; adding an ivfflat index switches to faster, approximate search. If your app already runs on Postgres, this is often the smallest step: no new system to learn or pay for.