What is OpenAI's MuseNet, and how does an AI compose music?

Yesterday, OpenAI introduced MuseNet, a program that composes music. Give it a style, a set of instruments and, if you like, the opening notes of a famous piece, and it continues with a new composition that can last up to four minutes. It can also blend styles: the opening of a Chopin nocturne carried on as a pop song with piano, drums, bass and guitar.

To show it off, OpenAI held an experimental concert streamed live on its Twitch channel, playing pieces that no one, including the team, had heard before. It also opened a web tool so that anyone can try MuseNet until 12 May.

What MuseNet is

MuseNet is a neural network, a program that learns patterns from examples rather than following hand-written rules. Nobody taught it music theory. It learned harmony, rhythm and style on its own by studying a very large collection of MIDI files. A MIDI file does not store recorded sound; it stores instructions, like a digital score: which note to play, on which instrument, how loudly and for how long.

The project lead is Christine Payne, a classically trained pianist on OpenAI's team. She told TechCrunch she was particularly pleased that MuseNet seems to handle the complex harmonies of Beethoven and Chopin, and that she is already working with a composer who plans to use it in his own pieces.

How it works

MuseNet uses the same core technology as GPT-2, the text model OpenAI announced in February. It is a Transformer, a network design built around attention, which lets the model weigh which earlier parts of a sequence matter most for predicting the next part. GPT-2 predicts the next word in a text; MuseNet predicts the next token in a piece of music.

To make music look like text, OpenAI turned each MIDI file into a sequence of tokens. After testing several encodings, the team settled on tokens that each pack three facts together: the pitch, the volume and the instrument. Other tokens mark the passage of time. A short passage, in simplified form, looks like this:

Token (simplified)	Meaning
bach	Style token: write like Bach
piano:v72:G4	Start a piano note, G in the fourth octave, at volume 72
violin:v80:G4	Start a violin note on the same pitch, a little louder
wait:12	Let time pass before the next event
piano:v0:G4	Volume 0: stop that piano note

The style and instrument tokens are the steering wheel. During training, OpenAI placed a composer or band token and a list of instruments at the start of each piece, so the model learned how they shape the notes that follow. When you ask for "Rachmaninoff on piano," you are starting the sequence with those tokens and letting MuseNet predict what comes next.

The model is large: 72 layers with 24 attention heads, able to attend to 4,096 tokens at once. That long memory is about four minutes of music, which is why MuseNet can keep a piece's larger structure, such as returning to a theme or building to a finish, rather than only stringing nice-sounding notes together. It borrows techniques from OpenAI's recent Sparse Transformer work to make such long sequences practical to train.

Where the music came from

OpenAI says it collected training data from many sources. ClassicalArchives and BitMidi donated their large MIDI collections, the team found online collections covering jazz, pop and African, Indian and Arabic styles, and it also drew on a dataset called MAESTRO. To get more out of the data, the team shifted pieces up and down in pitch, varied their volume and timing slightly, and trained an internal "critic" that judges whether a passage came from the dataset or from the model itself. That critic helps choose better samples.

Try it: Open the MuseNet tool while it is online, pick a composer you know well, and generate three pieces from the same famous opening. Listen for what stays the same and what changes. Then pair that composer with an unusual instrument set and notice where the music starts to sound less natural.

What people are saying

Computer-generated music is not new; it has been around for decades. What stands out to TechCrunch is how flexible MuseNet is: one model handles many genres and can combine them, while many earlier systems focus on a single composer or style. The reviewer is less impressed by the plain synthesizer sounds used for playback, and finds the results strongest in jazz and classical improvisation, where an odd note is easier to forgive.

OpenAI is upfront about limits. The instruments you choose are strong suggestions, not guarantees: the model calculates probabilities across all notes and instruments and can still pick something else. It also struggles with unusual pairings, such as Chopin played by bass and drums.

How can you try MuseNet, and what is it good for?

The prototype co-composer is embedded in OpenAI's MuseNet announcement and is available through 12 May 2019. In simple mode you choose a composer or style and, optionally, the start of a well-known piece, and you hear pre-generated samples that were not hand-picked. In advanced mode you interact with the model directly; it takes longer, but you create a brand-new piece. OpenAI invites people to share pieces they like, and says it will decide on next steps after the trial based on feedback. According to TechCrunch, at least part of the project is expected to be open-sourced in the coming weeks.

As for uses, it is early. The most obvious ones are creative: sketching ideas, finding an unexpected continuation for a melody, or hearing a tune arranged in a different style. Musicians may treat it as a brainstorming partner rather than a finished composer. OpenAI also sees music as a useful test bed for its research, because it sits between text and images, and a listener can easily hear when a model loses the rhythm or the structure of a piece.