How is Google using BERT to understand your searches?

Yesterday, 25 October 2019, Google announced one of the largest changes to its search engine in years. Search now uses BERT, a language model Google researchers published last year, to work out what people mean when they type a query. Pandu Nayak, a Google Fellow and vice president of Search, calls it the biggest step forward in five years and one of the biggest in the history of the product.

The change is about meaning, not keywords. Older systems often matched the words in your query against words on web pages. BERT reads the query as a sentence, paying attention to how each word relates to the others. That matters most for long, conversational searches and for little words that change everything.

What Google announced

According to Nayak's post on Google's blog, the company handles billions of searches a day, and about 15 percent of them are queries it has never seen before. People also tend to type in what Nayak calls "keyword-ese": strings of words they think a computer will understand, rather than the question they would ask a person.

Here is what changes now:

• Ranking. BERT will help Google better understand about one in ten searches in English in the U.S. when deciding which pages to show. Other languages and locations will follow over time.
• Featured snippets. These are the short answer boxes at the top of some results pages. Google is using a BERT model to improve them in the roughly two dozen countries where they appear, with notable gains in Korean, Hindi and Portuguese.
• Hardware. Some BERT models are demanding enough that, for the first time, Google is serving search results with its latest Cloud TPUs, custom chips designed for machine learning.

TechCrunch notes that most search algorithm changes are far more subtle, so one that touches a tenth of searches is a very big deal. The rollout began this week, so many people are already seeing its effects.

How BERT reads a query

BERT stands for Bidirectional Encoder Representations from Transformers. It was introduced in October 2018 in a paper by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova of Google AI Language, and Google released the code and trained models for free.

A Transformer is a neural network design that looks at every word in a sentence in relation to all the others at once, instead of reading strictly left to right. "Bidirectional" means BERT uses the words on both sides of each word to decide what it means.

BERT learned this by playing a fill-in-the-blank game on huge amounts of text. During training, about 15 percent of the words were hidden, and the model had to guess them from the surrounding context. The researchers say the idea was inspired by the Cloze test, a classic reading exercise where students fill gaps in a passage. To guess "bank" in "she sat on the river ___ and fished," you need the words before and after the gap. Doing this millions of times teaches a model a great deal about how words depend on each other. The trained model can then be fine-tuned, given a small amount of extra training, for a specific job, such as matching queries to helpful pages.

A worked example: how the small words change the answer

Google shared several queries from its testing. This table shows what went wrong before and what BERT picks up:

In each case the old approach treated the query like a bag of separate words. BERT treats it like a sentence where order and small words carry meaning.

Google is also candid that BERT makes mistakes. Ask "what state is south of Nebraska," and BERT points to a small community named South Nebraska instead of the state of Kansas.

Why it matters

For everyday users, the promise is simple: you can type more like you speak and expect Google to follow. Nayak's own advice is to let go of some keyword-ese and search in a natural way.

For the AI field, this is one of the clearest signs yet that research on language models is moving from papers into products used by huge numbers of people. BERT went from a research paper to a core part of Google Search in about a year. It also shows the cost of that power: Google needed new hardware to run these models fast enough for live search.

There is a multilingual angle too. Google says models that learn from English, the language with by far the most web content, can carry some of those lessons into other languages. That helps explain why featured snippets are improving in languages such as Hindi and Korean at the same time.

Do you need to change the way you search or write for Google?

For searching, the change works in your favor. You no longer need to strip a question down to a few keywords. Full questions with words like "for," "to," "without" and "no" are now more likely to be understood, especially when they are long and specific. When results look off, rephrasing with clearer relationships between words is still a good habit, since Google admits it does not always get it right.

For people who publish on the web, Google's announcement does not list any new technical steps. What BERT rewards is understanding what a searcher actually means, so the sensible approach is to write clearly for human readers and answer real questions directly. Pages that match a query's meaning, not just its words, are the ones this change is built to find.

Google says it will bring BERT to ranking in more languages and regions over time. Many expect similar language models to appear in other products that need to understand ordinary speech.