low-rank adaptation
In plain wordsA cheap way to customize a big AI model: instead of changing all its billions of numbers, you train a small add-on of a few million numbers that slightly adjusts how it behaves.
A parameter-efficient fine-tuningparameter-efficient fine-tuning, n.: A family of tricks for customizing a large AI model by training only a tiny fraction of its numbers, or a small add-on, leaving the rest untouched to save memory and money. method that leaves a pretrained model's weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. untouched and instead learns a correction to selected weight matrices W (typically the attention and feed-forward projections of a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.) in factored form: the product of two thin matrices B and A of rank r, far smaller than the matrix dimensions, scaled by α/r. B starts at zero, so training begins from the unmodified model. After training, the product BA can be added into W, so the adapted model runs exactly as fast as the original; alternatively the small LoRA files are kept separate and swapped per task or user. With r between about 4 and 64, trainable parameters fall by orders of magnitude: the original paper reports 10,000 times fewer than full fine-tuning of GPT-3 175B, and a third of the GPU memory. LoRA often approaches full fine-tuningfull fine-tuning, n.: Fine-tuning in which every one of a model's internal numbers is allowed to change — the most flexible option, but also the most expensive in memory and storage. on narrow tasks but learns less, and forgets less, on large knowledge-heavy ones. Variants include QLoRAQLoRA, n.: A memory-saving version of LoRA that squeezes the big model into a compact 4-bit form while training small add-ons, letting very large models be customized on a single graphics card., DoRA and LoRA+; LoRA is ubiquitous for customizing image generators and open-weight language models.
We serve one base model with 200 customer LoRAs swapped in per request.
Origin
Introduced by Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen of Microsoft in June 2021 (ICLR 2022), motivated by evidence that fine-tuning updates have a low intrinsic dimension. It spread rapidly in 2023 with Hugging Face's PEFT library, Stable Diffusion LoRAs and QLoRA.
Usage note
Not to be confused with LoRa, a long-range radio protocol. Rank, the scaling factor α, learning rate and the choice of layers that receive adapters strongly affect results.
Related
- parameter-efficient fine-tuningparameter-efficient fine-tuning, n.: A family of tricks for customizing a large AI model by training only a tiny fraction of its numbers, or a small add-on, leaving the rest untouched to save memory and money.
- QLoRAQLoRA, n.: A memory-saving version of LoRA that squeezes the big model into a compact 4-bit form while training small add-ons, letting very large models be customized on a single graphics card.
- adapteradapter, n.: A small trainable add-on slotted into a large frozen AI model to teach it a new task or style, like a plug adapter that lets one device work in many countries.
- low-rank approximationlow-rank approximation, n.: Replacing a big grid of numbers with a slimmer one that keeps most of the same information, the way a few pencil strokes can capture a face. It saves memory and computation.
- full fine-tuningfull fine-tuning, n.: Fine-tuning in which every one of a model's internal numbers is allowed to change — the most flexible option, but also the most expensive in memory and storage.
- fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way.
- model mergingmodel merging, n.: Combining two or more trained AI models into one by blending their internal numbers directly, so the result shares their skills without any further training.
- full fine-tuningfull fine-tuning, n.: Fine-tuning in which every one of a model's internal numbers is allowed to change — the most flexible option, but also the most expensive in memory and storage.
Sources
- Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. arXiv:2106.09685. arxiv.org/abs/2106.09685
- Biderman, D. et al. (2024). LoRA Learns Less and Forgets Less. Transactions on Machine Learning Research.
Mentioned in
fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way., Stable DiffusionStable Diffusion, prop. n.: A widely used image-generating AI whose model files were released for anyone to download, so people could run it on their own computers and adapt it., adapteradapter, n.: A small trainable add-on slotted into a large frozen AI model to teach it a new task or style, like a plug adapter that lets one device work in many countries., catastrophic forgettingcatastrophic forgetting, n.: When a neural network trained on something new gets suddenly and badly worse at what it could already do, because the new training rewrites the internal settings the old skills relied on., ControlNetControlNet, prop. n.: An add-on for image generators that makes them follow a guide image — a sketch, a stick-figure pose or a depth map — so the layout comes out as intended., DreamBoothDreamBooth, prop. n.: A way to teach an image generator one specific subject — your dog, your face, a product — from a handful of photos, so it can draw that exact subject in new scenes., FLUXFLUX, prop. n.: A family of image-generating AI models from the German start-up Black Forest Labs, founded by researchers behind Stable Diffusion; some versions can be downloaded and run freely., full fine-tuningfull fine-tuning, n.: Fine-tuning in which every one of a model's internal numbers is allowed to change — the most flexible option, but also the most expensive in memory and storage., linear transformationlinear transformation, n.: A rule for moving every point in a space that keeps straight lines straight and the origin in place: rotating, stretching, reflecting or squashing everything evenly. Every grid of numbers describes one., low-rank approximationlow-rank approximation, n.: Replacing a big grid of numbers with a slimmer one that keeps most of the same information, the way a few pencil strokes can capture a face. It saves memory and computation., matrix factorizationmatrix factorization, n.: Breaking a big table of numbers, such as which users liked which films, into two smaller tables of hidden traits, so missing entries can be filled in by multiplying them back together., model mergingmodel merging, n.: Combining two or more trained AI models into one by blending their internal numbers directly, so the result shares their skills without any further training., parameter-efficient fine-tuningparameter-efficient fine-tuning, n.: A family of tricks for customizing a large AI model by training only a tiny fraction of its numbers, or a small add-on, leaving the rest untouched to save memory and money., rankrank, n.: The number of truly independent directions a grid of numbers contains. A low-rank grid can be rebuilt exactly from a few simple building blocks, however large it looks., singular value decompositionsingular value decomposition, n.: A way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data., Thinking Machines LabThinking Machines Lab, prop. n.: Mira Murati's AI company, which she started in San Francisco in 2025 after serving as OpenAI's technology chief. It makes tools for customizing AI models and releases some models openly., hypernetworkhypernetwork, n.: A neural network whose job is to produce the settings, or weights, of another neural network, so that one model can generate many specialized models on demand., prefix tuningprefix tuning, n.: Customizing a frozen language model by learning a short sequence of invisible 'virtual words' that is placed in front of every input at every layer, steering what the model produces., QLoRAQLoRA, n.: A memory-saving version of LoRA that squeezes the big model into a compact 4-bit form while training small add-ons, letting very large models be customized on a single graphics card., representation fine-tuningrepresentation fine-tuning, n.: A way of adapting a language model to a new task by training small, targeted edits to its internal signals instead of changing its weights, often with far fewer adjustable numbers than other methods., stability–plasticity dilemmastability–plasticity dilemma, n.: The basic tension any learning system faces: it must change enough to learn new things, yet stay stable enough not to lose what it already knows., task arithmetictask arithmetic, n.: Editing an AI model by adding or subtracting the changes that fine-tuning made to it, as if skills were arrows that can be added together or reversed., test-time trainingtest-time training, n.: Letting an AI model briefly update itself on each new problem it is given, learning from that problem's own clues before answering, instead of keeping its knowledge frozen after training., textual inversiontextual inversion, n.: Teaching an image generator a new 'word' for a specific object or style from a few example pictures, leaving the model itself unchanged — only one vocabulary entry is added.