music generation
In plain wordsUsing AI to compose or produce music — from melodies written as notes to complete songs with vocals — often from a short text description like 'upbeat jazz with saxophone'.
The automatic creation of music by computer, either symbolically, as notes or MIDI events, or directly as audio. Symbolic systems, from rule-based and Markov-chain composers to recurrent and transformer models such as Music Transformer (2018), produce scores that must still be performed or rendered. Audio systems generate sound itself: OpenAI's Jukebox (2020) modeled raw audio with a multiscale VQ-VAEVQ-VAE, n.: A network that compresses an image or sound into a short list of entries from a learned 'codebook' — like describing a picture with a fixed set of tiles — which other models can then generate. and transformers; Google's MusicLM and Meta's MusicGen (2023) generated music from text with language models over neural audio codecneural audio codec, n.: A neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece. tokens; and diffusion and flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples. models generate spectrograms or latents. Commercial services such as SunoSuno, prop. n.: A popular AI service that writes and performs complete songs — music, lyrics and singing — from a short text description. and Udio (2023–2024) produce complete songs with lyrics and vocals in seconds. Evaluation relies on listening tests and metrics such as Fréchet Audio Distance. Training on copyrighted recordings led major record labels to sue Suno and Udio in June 2024; licensing settlements from late 2025 began moving the field toward licensed training data and controls over artists' likenesses.
The music generation tool gave us a passable jingle, but the vocalist slurred the brand name.
Origin
Algorithmic composition is old: Lejaren Hiller and Leonard Isaacson's computer-generated Illiac Suite dates from 1957, and David Cope's Experiments in Musical Intelligence from the 1980s. Neural music generation followed deep learning in the 2010s; Jukebox (April 2020) made raw-audio song generation credible, and text-to-music became mainstream in 2023–2024.
Related
- SunoSuno, prop. n.: A popular AI service that writes and performs complete songs — music, lyrics and singing — from a short text description.
- neural audio codecneural audio codec, n.: A neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece.
- text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps.
- voice cloningvoice cloning, n.: Using AI to copy a specific person's voice from recordings, so a computer can say new things that sound like them — useful for dubbing and accessibility, and abused in scams.
- AI artAI art, n.: Images, music or other artworks made with the help of AI — from early computer drawing programs to today's prompt-based image generators — and the debate over whether and how such work counts as art.
- AI copyright litigationAI copyright litigation, n.: The many lawsuits in which authors, artists, news organizations, record labels and others accuse AI companies of copying their work without permission to build or run AI models.
- diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.
Sources
- Dhariwal, P. et al. (2020). Jukebox: A Generative Model for Music. arXiv:2005.00341. arxiv.org/abs/2005.00341
- Agostinelli, A. et al. (2023). MusicLM: Generating Music From Text. arXiv:2301.11325. arxiv.org/abs/2301.11325
- Copet, J. et al. (2023). Simple and Controllable Music Generation (MusicGen). arXiv:2306.05284. arxiv.org/abs/2306.05284
Mentioned in
audio language modelaudio language model, n.: An AI language model that can take in sound, such as speech, music and everyday noises, and understand or generate it, much as text chatbots handle words., SunoSuno, prop. n.: A popular AI service that writes and performs complete songs — music, lyrics and singing — from a short text description., neural audio codecneural audio codec, n.: A neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece., source separationsource separation, n.: Pulling apart a mixture of sounds into its separate parts, such as isolating one voice in a noisy party or the vocals, drums and bass in a finished song.