What is DeepMind's WaveNet, and how did it make computer voices sound human?

For decades, computer voices have had a giveaway: a flat rhythm, a choppy join between syllables, a tone that never quite sounds alive. On 8 September 2016, the London lab DeepMind, owned by Google, showed a different way to make a machine speak. Its system, WaveNet, did not assemble speech from recorded pieces. It drew the sound wave itself, one point at a time.

The result surprised many audio researchers. Listeners in blind tests rated WaveNet's speech far closer to a real person than anything Google had in use. The same network could also invent short piano pieces, because it had learned to model sound in general, not just speech.

How computers spoke before WaveNet

Turning written text into spoken audio is called text-to-speech, or TTS. In 2016 most systems used one of two methods.

DeepMind's blog post, written by Aäron van den Oord of DeepMind, Heiga Zen of Google and Sander Dieleman of DeepMind, framed WaveNet as a way to get the flexibility of parametric speech with better quality than either older method.

How WaveNet works

Digital audio is a long list of numbers. Each number, called a sample, records the position of the sound wave at one instant. Speech audio typically has 16,000 or more samples per second. WaveNet is autoregressive: it predicts the next sample from all the samples before it, adds its guess to the list, then predicts the next one, and so on.

Think of drawing a long wavy line on graph paper, one dot at a time, where each new dot depends on the shape of everything you have drawn so far. To keep the predictions manageable, WaveNet treats each sample as one of 256 possible levels and picks among them using probabilities.

The trick is seeing far enough back. One second of speech is thousands of samples, and a syllable or a word spans many of them. WaveNet uses dilated convolutions: layers that look at inputs spaced further and further apart as you go deeper in the network, a bit like reading every letter, then every second letter, then every fourth. This lets its "memory" of the past grow very quickly with each added layer.

Here is what happens when WaveNet speaks a sentence:

1. The text is converted into linguistic and phonetic features: which sound, syllable and word come next.
2. Those features are fed into the network alongside the audio produced so far.
3. The network computes a probability for each of the 256 possible next values and draws one.
4. That value joins the audio, and the process repeats, 16,000 times for each second of sound.

Remove the text in step 1 and WaveNet still talks, but it babbles: real words mixed with invented ones, plus breaths and mouth sounds. Give it a speaker's identity as an extra input and the same network can switch between many voices. DeepMind noticed something unexpected here: learning from many voices improved its imitation of each individual voice.

What the tests showed

DeepMind trained WaveNet on some of Google's speech datasets and asked people to rate samples on a 1 to 5 scale, a standard measure called the Mean Opinion Score. The tests used more than 500 ratings on 100 sentences, in both US English and Mandarin Chinese. In each language, WaveNet closed over half of the gap between Google's best existing systems and recordings of real people.

The paper, posted to the research archive arXiv a few days later, on 12 September 2016, listed nine authors, including van den Oord, Dieleman, Zen, Oriol Vinyals, Alex Graves and Koray Kavukcuoglu. It also showed that WaveNet could help with recognizing speech sounds, not just producing them.

There was one big catch. Building audio one sample at a time is slow, and DeepMind said the first version needed too much computing power to be used in real products.

Why it mattered

WaveNet showed that a neural network could model raw sound directly and beat systems that experts had tuned for years. It also showed that one general method could cover speech, many voices and music. For a lab best known for game-playing programs such as AlphaGo, it was a clear sign that its learning methods reached well beyond games.

The speed problem did not last long. In October 2017, Google announced a version about 1,000 times faster with better quality, and began using it for the Google Assistant's voices in US English and Japanese. A November 2017 DeepMind paper described a method called probability density distillation that produces high-quality speech over twenty times quicker than it takes to play it.

Where is WaveNet used today, and what worries does it raise?

The most visible use is Google's own voice products. At Google's I/O developer conference in May 2018, the company presented new Assistant voices built with WaveNet, and explained that the approach needed far fewer studio recordings to create each new voice. Researchers have also used WaveNet-style models to convert one speaker's voice into another's while keeping the words, and later work cut the amount of recorded audio needed to imitate a voice to just a few minutes.

That last ability is also the main worry. A system that can copy a voice from a short recording could be used to fake a phone call or a public statement. Companies working on voice cloning have discussed hidden audio watermarks so that synthetic speech can be detected. For listeners, the practical habit is the same one we now apply to photos and text: a familiar voice alone is not proof of who is speaking, especially when a message asks for money or urgent action.