LoRA in AI: what it is, explained simply

In plain words

Teaching an existing AI model a new skill or style, called fine-tuning, normally means adjusting all of its billions of numbers. That needs expensive hardware, and every customised copy is as big as the original.

LoRA, short for low-rank adaptation, takes a shortcut. It locks the original model so nothing in it changes, and bolts on a small set of new numbers alongside. Only those few are trained. When the model runs, the add-on's small corrections are combined with the frozen original.

The add-on is tiny, often a fraction of a percent of the model, so it trains on a single graphics card and saves as a small file. One base model can carry many such add-ons and swap them in seconds: one for legal writing, one for Hindi replies, one for a company's tone.

LoRA is hugely popular for open language models and for image generators, where people train and share LoRAs that teach a base model a particular character or subject.

Everyday example

Clip-on lenses for a phone camera

Nobody buys a new phone to take wide-angle shots. We clip a small lens over the existing camera. The phone's camera stays exactly as it was; the clip-on bends the light a little differently. Swap in a macro lens and it does something else. LoRA is a clip-on lens for an AI model: small, cheap, removable, and the main camera never changes.

Where the example breaks: A clip-on lens sits on the outside. A LoRA's corrections are added inside every layer of the model, and can even be merged in permanently. And unlike lenses, two LoRAs combined do not always work well together.

How it actually works

Microsoft researchers introduced LoRA in June 2021. The insight: the change a model needs for a new task can be described with far fewer numbers than the model itself.

Freeze the big matrices

A transformer's layers are made of large grids of numbers (matrices), for example 4,096 by 4,096, about 16.8 million parameters each. LoRA leaves these untouched.

Add two thin matrices

Beside a frozen matrix W, LoRA adds two thin ones: B, 4,096 tall and r wide, and A, r tall and 4,096 wide, where r (the rank) is small, such as 8. Multiplied together, B × A makes a full-size update. At rank 8 that is 65,536 trainable numbers, about 0.4% of the original.

Train only the thin ones

During fine-tuning, only A and B learn. Because there are so few, the memory needed for training drops sharply. On GPT-3, the paper reported 10,000 times fewer trainable parameters and 3 times less GPU memory than full fine-tuning.

No slowdown when running

After training, B × A can be added straight into W, so the tuned model runs exactly as fast as the original. Or the small adapter can be kept separate and swapped as needed.

QLoRA

In 2023, QLoRA combined LoRA with a 4-bit quantized base model, letting researchers fine-tune a 65 billion parameter model on one 48 GB graphics card.

Introduced

June 2021, Edward Hu and colleagues at Microsoft, "LoRA: Low-Rank Adaptation of Large Language Models"

Headline result

Matched or beat full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3 while training far fewer parameters

Family

One of several parameter-efficient fine-tuning (PEFT) methods; Hugging Face's PEFT library implements it

File size

An image-model LoRA is typically a few hundred MB (Hugging Face), a small fraction of the base model it plugs into

Questions people ask

What does LoRA stand for?

Low-rank adaptation. "Adaptation" means adjusting a pre-trained model to a new task. "Low-rank" is a maths term: the adjustment is built from two thin matrices, which can only express a limited, simple kind of change, and that turns out to be enough for most tasks.

What is the difference between LoRA and fine-tuning?

LoRA is a kind of fine-tuning. Full fine-tuning changes every number in the model. LoRA freezes the model and trains a small add-on, which is much cheaper and gives a small file instead of a whole new copy of the model.

What is a LoRA in Stable Diffusion?

A small add-on file that teaches an image model like Stable Diffusion a particular character, object or look. Hugging Face describes them as a few hundred MB, easy to store and share; loading one changes what the base model draws without retraining it.

Is LoRA as good as full fine-tuning?

Often close, and the original paper found it matched full fine-tuning on the models tested. But a 2024 study on code and maths found LoRA learned less than full fine-tuning, while forgetting less of what the model already knew. For big changes, full fine-tuning can do better.