LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method that adapts a large language model by freezing its original weights and training…
See why full fine-tuning is costly and how LoRA sidesteps it.
Full fine-tuning updates every weight in a model, which for billions of parameters demands enormous GPU memory and produces a full-size copy per task. LoRA (Low-Rank Adaptation) avoids this: it freezes the original weights and trains only small added matrices, updating roughly 0.1 to 1 percent of the parameters.
The result is dramatically less memory and compute, plus tiny adapter files — often just megabytes — that you attach to the base model at inference. You can keep many task-specific adapters for one shared base.
Understand the low-rank matrices LoRA trains instead of the full weights.
For a chosen weight matrix, LoRA adds a bypass: two small matrices, A and B, whose product has the same shape as the original but is built from a much smaller rank r. During training the original weight stays frozen and only A and B are learned; at inference their product is added to the frozen weight.
The insight is that the change needed to adapt a model to a task is 'low-rank' — it lives in a small subspace — so a few thousand parameters can capture it instead of millions. LoRA is usually applied to the attention projection layers.
After training you can keep the adapter separate (swap adapters per task on one base) or merge A and B back into the weights for a standalone model with no inference overhead. Both are common.
Set the hyperparameters that matter and reach large models with QLoRA.
Three settings dominate. Rank r sets the capacity of the adapter (common values 8 to 64) — higher fits more but trains more parameters and risks overfitting. Alpha scales the adapter's contribution. Target modules choose which layers get adapters, typically the attention query and value projections.
QLoRA combines LoRA with a 4-bit quantized base model: the frozen weights are stored in 4-bit precision while the LoRA adapters train in higher precision. This cuts memory enough to fine-tune very large models on a single consumer or cloud GPU.
from peft import LoraConfig, get_peft_model config = LoraConfig( r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"], lora_dropout=0.05, task_type="CAUSAL_LM", ) model = get_peft_model(base_model, config) # only adapters are trainable model.print_trainable_parameters() # ~0.1-1% of the total
Using Hugging Face PEFT, LoraConfig sets rank, alpha, and which layers get adapters; get_peft_model wraps the frozen base so only the small LoRA matrices train. The printout confirms just a tiny fraction of parameters are trainable.
Decide when fine-tuning with LoRA beats RAG or prompting.
Reach for LoRA when you need to bake in behavior — a consistent tone or format, a specialized classification skill, or a domain style — and you have hundreds to thousands of quality examples showing the pattern. LoRA makes that affordable where full fine-tuning would not be.
It is the wrong tool for supplying facts. Knowledge that changes or must be cited belongs in RAG; simple behavior shifts often need only prompting. The usual order is prompting first, RAG for knowledge, and LoRA when a pattern must be built into the model.
LoRA (Low-Rank Adaptation) fine-tunes a model efficiently by freezing its weights and training small low-rank matrices added to its layers, updating ~0.1-1% of parameters and yielding tiny, swappable adapters. Key knobs are rank, alpha, and target modules; QLoRA adds 4-bit base quantization to fit large models on one GPU. Use LoRA to bake in behavior with many examples — but reach for RAG for knowledge and prompting for simple shifts.
You want a model that always replies in your company's support tone and JSON format. Decide whether LoRA, RAG, or prompting fits, pick a starting rank and target modules for a LoRA run, and explain why the tone belongs in the weights while the product facts belong in RAG.
What is LoRA in LLM fine-tuning?
LoRA (Low-Rank Adaptation) adapts a model by training tiny added matrices while the original weights stay frozen, updating a small fraction of parameters.
How does LoRA reduce the cost of fine-tuning?
By freezing the base and learning only low-rank matrices, LoRA needs far less memory and yields small, swappable adapters instead of a full model copy.
What is QLoRA?
QLoRA stores the frozen base in 4-bit precision while training higher-precision LoRA adapters, making very large models fine-tunable on modest hardware.
When should I use LoRA instead of RAG or prompting?
LoRA teaches behavior efficiently; changing or citable knowledge belongs in RAG, and simple behavior changes often need only prompting.