# What is BERT, and why did it change how computers understand language?

Date: 2021-12-19 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/what-is-bert/

**Answer:** BERT (Bidirectional Encoder Representations from Transformers) is a language model that Google researchers described on 11 October 2018; it learns by filling in hidden words using the context on both sides of each gap. Pre-trained once on books and Wikipedia, then briefly fine-tuned, it set new records on eleven language tasks, and Google released it free for anyone to use in November 2018.

On 11 October 2018, four researchers at Google AI Language, Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, posted a paper titled "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." The paper reported new best results on eleven different language tasks, some by large margins, using one model with almost no task-specific changes.

Three weeks later, on 2 November 2018, Google released the code and trained models for free. Anyone could download BERT, add a small layer for their own task, and get strong results in hours rather than months.

## The problem: reading in one direction

Computers have long struggled with the fact that a word's meaning depends on its neighbors. Older tools such as word2vec gave each word a single fixed **embedding**, a list of numbers standing for its meaning. Google's own example shows the problem: "bank" gets the same numbers in "bank account" and "bank of the river."

Newer **contextual** models fixed part of this by building a fresh representation of each word from the sentence around it. But most of them read in one direction. OpenAI's GPT, published that June, predicted each word only from the words before it. In "I accessed the bank account," such a model understands "bank" using "I accessed the" but not "account," the word that makes the meaning clear. ELMo, another 2018 model, read left-to-right and right-to-left separately and joined the results, which the BERT team called only shallowly bidirectional.

Why not just read both ways at once? Because of how these models are trained. If a model must predict a word while seeing the words on both sides, in a deep network the word can indirectly "see itself," and the task becomes trivial.

## How BERT works

BERT solves this with a fill-in-the-blank game, similar to the cloze tests used in language classrooms. During pre-training, 15% of the tokens (words or word pieces) in each passage are hidden, and the model must guess them from everything around them. Here is a simple example:

| Step | Text |
|---|---|
| Original | The man went to the store to buy a gallon of milk. |
| Masked input | The man went to the [MASK] to buy a gallon of [MASK]. |
| BERT's job | Predict "store" and "milk" using words on both sides |

A second training game teaches BERT how sentences relate. It sees two sentences and decides whether the second really followed the first in the original text, or was picked at random. This helps with tasks like question answering, which depend on links between sentences.

Under the hood, BERT is the **encoder** half of the Transformer, the attention-based design Google researchers introduced in 2017. **Attention** lets every word look at every other word in the passage and decide which ones matter. Google released two sizes:

- **BERT-Base**: 12 layers, 110 million parameters (the adjustable numbers inside the model).
- **BERT-Large**: 24 layers, 340 million parameters.

Both were pre-trained on BooksCorpus, about 800 million words, and English Wikipedia, about 2.5 billion words. Training ran on Google's Cloud TPUs, custom chips the team credited with letting them experiment quickly.

After pre-training, BERT is **fine-tuned**: one extra output layer is added for a specific job, such as labeling a sentence's sentiment or finding an answer in a paragraph, and the whole model trains briefly on that task's examples. With the released models, Google said a strong question-answering system could be trained on one Cloud TPU in roughly half an hour.

> **Try it:** Take a paragraph from a news story and blank out every seventh word. Ask a friend to fill in the gaps, first seeing only the text before each blank, then seeing the whole paragraph. Count how many more they get right the second time. That difference is the advantage of reading in both directions.

## What the results showed

On SQuAD v1.1, a widely used test where a system must find the answer to a question within a Wikipedia paragraph, BERT scored 93.2 on the F1 accuracy measure, above both the previous best and the score recorded for humans. On GLUE, a set of nine language-understanding tasks, it improved the best score by about 7.6 points according to Google's announcement (7.7 in the revised paper), a very large jump for that benchmark.

## Why it mattered

BERT made **pre-train, then fine-tune** the standard way to build language systems. It built on earlier work, including OpenAI's generative pre-training, ELMo and ULMFiT, but its bidirectional training and free release made it the default starting point. In a little over a year, a 2020 survey counted more than 150 research papers studying or improving it and described BERT as a ubiquitous baseline. The paper also won the Best Long Paper Award at NAACL 2019, a major conference in the field.

## How is BERT used today, and what can't it do?

The most visible use is search. On 25 October 2019, Google began applying BERT to English search queries in the United States. By December 2019 it was reported in more than 70 languages, and by October 2020 almost every English query on Google Search passed through BERT.

BERT has clear limits. It is built to understand text, not to write it: it fills in blanks and labels passages, but it does not produce long, fluent answers the way GPT-style models do. Its knowledge comes from its training text, with that text's gaps and biases. And researchers still do not fully understand why it works so well; a whole line of research now probes its internal representations and attention patterns to find out what it has actually learned.

## Sources

- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding: https://arxiv.org/abs/1810.04805
- Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing: https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html
- BERT (language model): https://en.wikipedia.org/w/index.php?title=BERT_(language_model)&oldid=1052964320