What is OpenAI's Whisper, and how does it turn speech into text?

OpenAI released something unusual yesterday: a model anyone can download and run. Whisper listens to recorded speech and writes down what was said. It handles dozens of languages, can translate what it hears into English, and copes well with the messy audio of real life, such as strong accents, a noisy street or a lecture full of jargon.

Whisper comes as code and model files that you can install on your own computer, under a license that allows commercial use.

What OpenAI released

On 21 September 2022 OpenAI published three things together: a blog post, a research paper titled "Robust Speech Recognition via Large-Scale Weak Supervision," and a GitHub repository with the code and trained models. The paper's authors are Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey and Ilya Sutskever. Both the code and the model weights (the learned numbers that make the model work) are released under the MIT License, one of the most permissive open-source licenses.

The models come in five sizes, so you can trade accuracy for speed:

All but the large model also come in English-only versions, which tend to do better on English, especially at the smallest sizes.

How Whisper works

Automatic speech recognition (ASR) means turning spoken words into written text. Whisper does it in a few steps:

1. The audio is cut into 30-second pieces.
2. Each piece is turned into a log-Mel spectrogram, a kind of picture of sound that shows which pitches are loud at each moment, adjusted to match how human hearing works.
3. An encoder, part of a Transformer network, reads that picture and summarizes it.
4. A decoder writes out the text, one token (word or word piece) at a time.

This encoder-decoder design is the same basic pattern used in machine translation, except that the input is sound instead of a sentence. The decoder also writes short special tokens that tell the model which job to do: name the language, transcribe it, translate it into English, or add timestamps. That lets a single model replace several separate tools.

The bigger idea is in the training data. Most speech systems learn from smaller collections of carefully transcribed audio. Whisper learned from 680,000 hours of audio paired with transcripts found on the internet. Those transcripts are imperfect, which is why the paper calls this weak supervision: lots of somewhat noisy examples instead of a few perfect ones. About 117,000 hours cover 96 languages other than English, and 125,000 hours pair speech in other languages with English text, which teaches translation.

How good is it?

OpenAI is candid about the trade-off. Whisper does not beat the best specialized systems on LibriSpeech, a well-known test made of audiobook recordings, because those systems are tuned for exactly that kind of clean speech. But when tested zero-shot (on datasets it never trained on) across many different collections of audio, Whisper makes about half as many errors as those specialized models. The paper says its accuracy and robustness approach human levels on English.

It also sets a new high mark for translating speech into English on a test called CoVoST2, again without training on that test. Accuracy varies a lot by language, though: widely spoken languages with plenty of web audio do much better than rare ones.

The paper lists the mistakes that remain. Whisper can get stuck repeating the same phrase, skip the first or last words of a segment, or occasionally produce a transcript that has nothing to do with the audio at all, a failure researchers call hallucination. That makes it worth checking important transcripts by ear.

Why it matters

Good speech recognition has mostly lived inside the products of big companies. Whisper puts a strong, multilingual model into anyone's hands for free, and OpenAI says it hopes developers will use it to add voice interfaces to many more applications. Practical uses are easy to imagine: subtitles for videos, searchable notes from meetings and lectures, interviews transcribed for journalists and researchers, and tools for people who are deaf or hard of hearing.

For researchers, the message is that simply training on far more, messier data works remarkably well for speech, much as it has for text.

How can you try Whisper on your own recordings?

You need a computer with Python and a free audio tool called ffmpeg. The project's instructions install Whisper with a single pip command that pulls the code from GitHub; ffmpeg is available from common package managers such as Homebrew on a Mac.

Once installed, you can use it from the command line. These are the patterns the documentation shows:

whisper interview.mp3 --model medium
whisper japanese.wav --language Japanese
whisper japanese.wav --language Japanese --task translate

The first line transcribes a file with the medium model. The second tells Whisper which language to expect. The third translates Japanese speech into English text. If you prefer Python, three lines are enough: load a model with whisper.load_model("base"), call transcribe on your audio file, and print the text. Longer recordings are handled automatically, 30 seconds at a time.

Start with the base or small model, which run on most laptops, and move up to medium or large if you need more accuracy and your machine has the memory. Keep in mind that the larger models need more memory and run more slowly.