LoRA Fine-Tuning Explained: Low-Rank Adaptation for LLMs

LoRA lets you fine-tune a large language model by training a few million parameters instead of billions. This guide covers how it works, which settings matter, and where it falls short.

By ForGen AI · Updated October 23, 2026

What is LoRA?

Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method. It freezes the pretrained weights of a model and learns a small, low-rank update next to them. It was introduced by Hu et al. in LoRA: Low-Rank Adaptation of Large Language Models (2021). On GPT-3 175B, the paper reports about 10,000× fewer trainable parameters and 3× less GPU memory than full fine-tuning, with comparable quality.

How the low-rank update works

Take a weight matrix W of shape d × k inside a transformer layer. LoRA leaves W untouched and adds a learned update ΔW = B·A, where B is d × r, A is r × k, and the rank r is much smaller than d or k:

h = W·x + (α / r) · B·A·x

A starts as small random values and B starts at zero, so the adapter begins as a no-op and the model initially behaves exactly like the base model. Only A and B get gradients. The optimizer state, which is the expensive part of fine-tuning, exists only for those small matrices.

How many parameters does LoRA train?

One adapted matrix adds r × (d + k) trainable parameters. For a 4096 × 4096 attention projection at r = 16, that is 131,072 parameters instead of 16,777,216, or about 0.8%. Across a whole model, LoRA adapters are typically well under 1% of the base model's size, which is why each one is only a few megabytes on disk.

The settings that matter

A minimal example with Hugging Face PEFT

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters()

When LoRA is the right choice

• Style, tone and format: teaching a model to answer like a brand, a character or a specific person.
• Narrow tasks: classification, extraction, or structured output for a fixed schema.
• Many variants of one model: one base model in memory with many small adapters swapped per user or per tenant.

LoRA is a weaker fit when you need the model to absorb a large body of new factual knowledge. Retrieval (RAG) or continued pretraining usually works better there.

FAQ

What does LoRA stand for?

LoRA stands for Low-Rank Adaptation. It was introduced by Hu et al. in 2021 (arXiv:2106.09685) as a parameter-efficient way to adapt large pretrained models.

What rank should I use for LoRA?

Ranks between 8 and 64 cover most use cases. Start at r=16 and raise it only if the adapter underfits a large or complex dataset.

Does LoRA slow down inference?

Not when the adapter is merged. The low-rank update can be folded into the base weights (W + BA), so inference costs the same as the original model. Unmerged adapters add a small overhead but can be swapped per request.