# What is OpenAI's Jukebox, and how can an AI sing?

Date: 2020-05-01 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/what-is-openai-jukebox/

**Answer:** Jukebox is a neural network OpenAI released on 30 April 2020 that creates songs as raw audio, including rough but recognizable singing, in the style of a chosen genre and artist and with lyrics you supply. It works by compressing sound into a short code, generating new code with Transformer models, and turning that code back into audio.

Yesterday, 30 April 2020, OpenAI introduced **Jukebox**, a neural network that makes music from scratch. You give it a genre, an artist and some lyrics, and it produces a new song as actual sound, not sheet music, complete with a voice that tries to sing your words in that artist's manner.

OpenAI is releasing the model weights and code, a research paper, and a website where anyone can listen to thousands of generated songs that were not hand-picked. The results are strange, sometimes beautiful, and often clearly machine-made.

## What OpenAI released

Jukebox comes from a team of six OpenAI researchers, led on the paper by Prafulla Dhariwal and Heewoo Jun, with Alec Radford, Jong Wook Kim, Christine McLeavey Payne and Ilya Sutskever. It follows MuseNet, which OpenAI showed last year. MuseNet composed in **MIDI**, a digital format that records which notes to play and when, much like a player-piano roll. Jukebox works directly with **raw audio**, the sound wave itself.

That difference matters. Note-based systems can produce impressive piano pieces or Bach-style chorales, but they cannot capture a human voice, the grain of a guitar or the feel of a performance. Raw audio can, but it is enormous. OpenAI points out that a four-minute song at CD quality contains more than 10 million separate values. A model has to keep a melody and a song's structure together across all of them.

The release includes three things: the paper, posted to arXiv yesterday; the code and trained models; and a sample explorer. The blogger Andy Baio counted over 7,100 songs in it.

## How Jukebox turns lyrics into a song

The trick is to shrink the problem first. Jukebox uses a kind of **autoencoder**, a network that learns to compress data and then rebuild it. Its version is called a **VQ-VAE** (vector-quantized variational autoencoder). It turns every short slice of sound into one of 2,048 possible codes, a bit like turning a photo into a mosaic built from a fixed set of tiles.

Here is the path from raw sound to new song, using the numbers OpenAI gives:

| Step | What happens | Rate |
|---|---|---|
| 1. Start with raw audio | The sound wave, measured as amplitude values | 44,100 values per second |
| 2. Compress | Convolutional networks turn the audio into codes; the most compressed level is 128 times smaller | About 344 codes per second |
| 3. Generate | A Transformer, guided by artist, genre and lyrics, writes new codes one after another | About 344 codes per second |
| 4. Add detail | Two more Transformers fill in finer codes at less compressed levels | Rising back toward full detail |
| 5. Decode | The decoder turns the finest codes back into sound | 44,100 values per second |

A **Transformer** is the network design behind GPT-2: it predicts the next item in a sequence by paying attention to what came before. Here, instead of predicting the next word, it predicts the next piece of compressed music. The top level captures the big picture, melody and singing, while the lower levels add texture such as timbre, the quality that makes a piano sound different from a violin.

The model learned from 1.2 million songs collected from the web, about 600,000 of them in English, each paired with its lyrics and details such as artist, genre and year, which came from LyricWiki. The lyrics were not timed to the music, which made matching hard. A simple approach that spread the words evenly across each song worked surprisingly well, but failed on fast genres like hip hop. For those, the team separated the vocals with a tool called Spleeter and lined up each word using a lyrics-alignment tool called NUS AutoLyricsAlign.

One point is easy to miss: Jukebox does not write lyrics. As Baio notes, it only sings words it is given. Without lyrics, it produces sounds that resemble the singer but are not words. The lyrics in OpenAI's featured examples were written by GPT-2 and fed in as input.

> **Try it:** Open the Jukebox sample explorer, pick one artist you know well, and play two samples: one with lyrics supplied and one without. Write down three things that sound right (voice, rhythm, instruments) and three that sound wrong. Then listen for a chorus that comes back. You will probably not find one.

## What it cannot do yet

OpenAI is frank about the gap between Jukebox and human music:

- **No big structure.** Songs hold together from moment to moment, follow normal chord patterns and can include impressive solos, but you will not hear a chorus that repeats.
- **Noise.** Compressing and rebuilding the sound adds audible hiss and blur.
- **Slow.** Rendering one minute of audio takes about nine hours, so it cannot be used live.
- **Narrow training.** The data is mostly Western music with English lyrics.

The team also shared Jukebox with ten musicians from different genres. They found it interesting but not immediately useful for their own creative work, given these limits.

## Why it matters

Jukebox is a big step for **generative models**, systems that create new data in the style of what they learned from. Text generators like GPT-2 deal with sequences of about a thousand words. Jukebox has to manage far longer sequences and still keep a tune. Its mix of compression and stacked Transformers shows one way to handle very long data.

It also raises questions that have no settled answers. If a model can sing new songs in a famous artist's voice, who owns them? Christine McLeavey Payne, one of the researchers, wrote on Twitter that the team considers the research fair use, does not claim copyright over the samples, and has written to the U.S. Patent and Trademark Office about the issue. OpenAI says it is also researching bias and intellectual property rights. Baio points out that the law on voice imitation is largely untested.

## Can you make your own songs with Jukebox?

Yes, if you have patience and serious hardware. The code and trained models are public on OpenAI's GitHub page, linked from the announcement. According to Baio, OpenAI puts the cost at roughly three hours of computing for each 20 seconds of sound on an NVIDIA Tesla V100, a professional graphics card that costs around $10,000. That matches OpenAI's own estimate of roughly nine hours per minute.

For most people, the easier path is Google Colab, a free online service that runs code on cloud computers. It can produce short, lower-quality samples, though run times and memory limits make long songs hard. In practice, you pick an artist and a genre, add your own lyrics, and wait.

If you would rather just listen, the sample explorer is the best place to start. Its "continuations" are especially revealing: the model is given the real lyrics and the first 12 seconds of a well-known song and asked to carry on, which shows both how well it imitates a style and how quickly it can drift. OpenAI is also testing MIDI files as a guide, which could give musicians finer control.

## Sources

- Jukebox: https://openai.com/blog/jukebox/
- Jukebox: A Generative Model for Music: https://arxiv.org/abs/2005.00341
- OpenAI's Jukebox Opens the Pandora's Box of AI-Generated Music: https://waxy.org/2020/04/openais-jukebox-opens-the-pandoras-box-of-ai-generated-music/