LoRA: fine-tuning large models by training small low-rank updates

TL;DR

Full fine-tuning updates every weight in a model, which becomes impractical for very large models — especially when you need many task-specific copies. LoRA freezes the pre-trained weights and, for selected weight matrices, learns an update expressed as the product of two small matrices. Only those small matrices are trained, and they can be merged into the original weights for deployment. Compared with fine-tuning GPT-3 175B with Adam, the authors report that LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times, while matching or beating fine-tuning quality — with no additional inference latency.

The problem

The standard recipe is to pre-train on general data and then adapt to a task or domain. Full fine-tuning produces a complete new set of weights per task; the authors use GPT-3 175B to show how expensive it becomes to deploy many independent fine-tuned copies at that scale.

Existing parameter-efficient methods had trade-offs of their own:

• Adapter layers insert small extra modules into each Transformer block. They must be computed sequentially, which adds latency at inference time.
• Prefix and prompt tuning learn extra input tokens instead of weights. The authors found prefix tuning hard to optimise, and reserving part of the sequence for learned tokens reduces the length available for the actual task.

The method

The weight update is low-rank

Earlier work had found that pre-trained models can be adapted within a surprisingly low-dimensional subspace. The authors hypothesise that the change in weights during adaptation also has a low "intrinsic rank". For a pre-trained weight matrix $W_0 \in \mathbb{R}^{d \times k}$, LoRA constrains the update to

During training $W_0$ is frozen and only $A$ and $B$ receive gradients. The forward pass for an input $x$ becomes $h = W_0 x + BAx$.

Details that matter

• Initialisation. $A$ starts from a random Gaussian and $B$ starts at zero, so $BA = 0$ and training begins exactly at the pre-trained model.
• Scaling. The update $\Delta W x$ is scaled by $\alpha / r$, where $\alpha$ is a constant. The authors note that, with Adam, tuning $\alpha$ is roughly equivalent to tuning the learning rate, so they do not tune it separately.
• Where it is applied. The paper adapts only attention weight matrices and freezes the feed-forward (MLP) blocks, leaving other placements to future work. Most experiments adapt only the query and value projections.
• Merging. Because $BA$ has the same shape as $W_0$, it can be added into the weights before deployment, so inference costs exactly what the original model costs. Switching tasks means subtracting one $BA$ and adding another.

How much capacity is needed?

The paper also studies what the learned updates look like. The authors find LoRA competitive even at very small ranks, and that the update amplifies directions that are present in the pre-trained weights but not emphasised by them. They argue this helps explain why a low-rank update is enough for the tasks they tested.

What the paper reports

The headline results, as reported by the authors in the abstract:

These are the authors' own measurements on the models and benchmarks they chose. "On par or better" summarises many comparisons; individual tasks vary, so read the paper's tables for any specific result.

Why it mattered

The practical appeal is that one copy of a large base model can serve many tasks through small, cheap-to-store updates. Later work builds directly on the method:

• QLoRA (Dettmers et al., 2023 (external site)) backpropagates through a frozen, quantised base model into LoRA adapters, reducing memory further.
• DoRA (Liu et al., 2024 (external site)) splits pre-trained weights into magnitude and direction and uses LoRA for the directional update. Its authors describe LoRA and its variants as having gained considerable popularity among parameter-efficient fine-tuning methods.
• Hugging Face's PEFT library (external site) uses LoRA as the example method in its quick-start.

Limitations and open questions

• Batching across tasks. If $BA$ is merged into the weights to avoid latency, requests for different tasks cannot easily share one forward pass. Keeping adapters unmerged allows mixing, where latency is less critical.
• Heuristic choices. Which matrices to adapt, and at what rank, were chosen largely by heuristics. The authors list more principled selection as future work.
• Scope of evidence. The feed-forward layers, layer norms and biases were not adapted, so the paper does not show whether adapting them helps.
• Low rank is task-dependent. The authors note that a task far from the pre-training data — their example is a different language — could need much more capacity than a small rank provides.

Reading notes for practitioners

• Sweep rank and target matrices. The paper found very small ranks competitive, but that is an empirical result for its tasks. Check on your own validation set rather than defaulting to large values.
• Decide early: merge or serve unmerged. Merged adapters cost nothing at inference; unmerged adapters let one deployment serve many tasks.
• Compare against full fine-tuning when you can. "On par" is the paper's aggregate finding, not a guarantee for your task.
• Pin adapters to their base model. An adapter is only meaningful on the exact base weights it was trained against. Version them together.