What is word2vec, and how did it teach computers the meaning of words?

Computers handle numbers well and words badly. For decades, most language software treated each word as an isolated symbol: "cat" and "kitten" were as unrelated as "cat" and "carburetor." On 16 January 2013, four Google researchers posted a paper that offered a fast, simple way out. Their method gives every word a position in a space of a few hundred dimensions, so that words with related meanings sit close together.

The paper is titled "Efficient Estimation of Word Representations in Vector Space." Its authors were Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean, all at Google in Mountain View, California. In August 2013, Google released the code as an open-source toolkit under the name word2vec, short for "word to vector," and the name stuck to the whole approach.

The idea: you know a word by its neighbors

The method rests on a simple observation: words that appear in similar contexts tend to have similar meanings. "Coffee" and "tea" both show up near "cup," "hot," "drink" and "morning." A system that only ever sees which words keep company with which can still work out that coffee and tea are alike.

Word2vec turns that observation into a training game. It reads billions of words of text and, at each position, tries to predict missing words from their surroundings. Each word starts with a random vector, a list of, say, 300 numbers. Every time a prediction is wrong, the vectors are nudged slightly so the next guess is better. After enough text, the vectors encode a surprising amount about how each word is used.

Two models: CBOW and skip-gram

The paper proposed two simple network designs.

• Continuous bag-of-words (CBOW) looks at the words around a gap and predicts the word in the middle. "Bag of words" means the order of the surrounding words is ignored.
• Continuous skip-gram does the reverse: given one word, it predicts the words likely to appear nearby.

Here is the same sentence seen by each model, with a window of two words on each side:

Both are deliberately shallow, much simpler than the deep networks then making headlines in image recognition. That simplicity was the point. The authors reported that one computer could learn good 300-dimensional vectors for a vocabulary of a million words from 1.6 billion words of text in less than a day. For their main experiments they trained on a Google News collection of about 6 billion tokens. According to the authors' own notes, CBOW trains faster, while skip-gram does better on rare words.

Arithmetic with meaning

The striking part was what you could do with the finished vectors. Because related words line up in consistent directions, you can add and subtract them. The paper recalls an earlier finding that the vector for "King," minus "Man," plus "Woman," lands closest to "Queen." It gives its own examples: subtract "big" from "biggest," add "small," and the nearest word is "smallest." Capital cities line up with their countries the same way, so France relates to Paris as Germany does to Berlin.

To measure this, the team built a test of nearly 20,000 such analogy questions: 8,869 about meaning (like capitals and countries) and 10,675 about grammar (like past tenses and plurals). A question counted as correct only if the closest word to the computed vector was exactly the expected answer.

Why it mattered

Word2vec made good word representations cheap. Anyone could download the code, run it on one machine and get vectors that captured real relationships. Google's announcement listed possible uses such as machine translation, question answering, knowledge extraction and conversational systems. The two Mikolov papers from 2013 had been cited tens of thousands of times by 2020. The approach also inspired close relatives, such as doc2vec for whole documents.

It also showed a limit. Each word gets exactly one vector, so "bank" the riverside and "bank" the lender share the same numbers. Solving that problem is a large part of what language models have done since.

How do word vectors relate to today's language models like GPT-3?

Word vectors are the ancestors, and they are still inside. GPT-3, which OpenAI described in May 2020, begins by turning each piece of text into a vector too. Those starting vectors are learned in a similar spirit: by predicting text, with no human labels.

The big difference is context. Word2vec gives "bank" one fixed vector. A modern language model built on the transformer design updates each word's vector based on every other word in the passage, so "bank" in "river bank" and "bank loan" ends up with different internal representations. That is why it can keep track of meaning across whole paragraphs.

The other difference is scale. Word2vec's networks were shallow by design, and its training took a day on one machine. GPT-3 has 175 billion parameters and needs large clusters of specialized chips to train. Yet the core lesson is the same one the 2013 paper taught: meaning can be learned from how words are used, if you read enough text.