What are embeddings, and how do they power search by meaning?

On January 25, OpenAI added an embeddings endpoint to its API. Send it a piece of text, or code, and it returns a vector: a list of numbers, 2,048 of them for the babbage-sized similarity model. On their own the numbers mean nothing to a person. Their value lies in comparison. Two texts about the same idea come back with vectors that point in nearly the same direction, and two unrelated texts do not.

That simple property powers a whole family of tasks OpenAI lists for the new endpoint: semantic search (search by meaning), clustering similar items, topic modeling and classification. The idea is not new; word2vec showed in 2013 that single words could be placed in such a space. What is new is getting high-quality vectors for whole sentences and documents with one API call.

A map of meaning

Picture a map where every sentence is a dot. Sentences about pets cluster in one neighborhood, sentences about tax returns in another, sentences about football in a third. Distance on the map stands for difference in meaning. OpenAI's own illustration: the vector for "canine companions say" lands closer to "woof" than to "meow".

A real embedding space has not two dimensions but thousands. Each dimension captures some aspect of the text, though usually not one a person could name. To compare two vectors, you compute their cosine similarity, a score from -1 to 1 that measures how closely they point the same way. Higher means more similar. OpenAI's examples compute it with a single dot product, one line of code.

OpenAI's models are descendants of GPT-3. According to the accompanying paper, they were trained with contrastive pre-training: the model sees pairs of texts that belong together and learns to place them close, while pushing apart texts that do not. The paper reports large gains over earlier unsupervised methods on search benchmarks such as MSMARCO, Natural Questions and TriviaQA.

Why this beats keyword search

Classic keyword search matches words. If a customer types "I can't get into my account" and your help page says "reset your password", a keyword engine may find nothing, because no important word is shared. Embedding search matches meaning, so the password page still ranks high. OpenAI says embedding-based search generalizes better than word-overlap methods for exactly this reason. One customer, Fabius, reports finding about twice as many relevant call transcripts as with fuzzy keyword search.

For search, OpenAI offers paired models: a "query" version for the short question and a "doc" version for the longer passages being searched. Document vectors can be computed once and stored, so each new search only needs one call for the question, followed by fast arithmetic.

A worked example: ranking five FAQ answers

To see the mechanics, shrink the map to two made-up dimensions: how much a text is about billing and how much it is about account access. These toy coordinates are invented for teaching; a real model returns thousands of numbers per text.

Question: "I can't get into my account" → (0.2, 0.9)

The two access-related answers rise to the top, although neither contains the words "get into" or "account". The billing answers sink. A real system does the same thing with far richer vectors, which is why it can tell apart shades of meaning a two-axis toy never could.

How do you search your own documents by meaning?

The recipe has five steps. First, split your documents into passages short enough to be about one thing, such as a paragraph or a help article section. Second, send each passage to a "doc" search model and save the vector alongside the text; a spreadsheet, a file or an ordinary database table is enough for a few thousand passages. Third, when someone asks a question, embed it with the matching "query" model. Fourth, compute the cosine similarity between the question and every stored passage. Fifth, show the top few results.

OpenAI's examples use a few lines of Python and the NumPy library for the arithmetic. For a small collection, comparing against every passage on each search is fast enough. Larger collections need smarter indexing, but the principle stays the same.

You can go one step further and hand the best passages to GPT-3 along with the question, so it writes an answer grounded in your own material. OpenAI's existing Answers endpoint already works this way: it searches documents you provide and adds the relevant context to the prompt before generating a completion. Embeddings let you build that pipeline yourself, with control over which passages the model sees. Whatever you build, show users the source passages, so they can check the answer against the original text.