LLM Fine-Tuning Glossary: LoRA, QLoRA, PEFT and More

Short, plain-English definitions of the terms that come up when fine-tuning and personalising large language models.

By ForGen AI · Updated October 23, 2026

Adapter

A small set of trainable parameters added to a frozen pretrained model to specialise it for a task, user or style. LoRA adapters are the most common kind.

Base model

The large pretrained (and sometimes instruction-tuned) model whose weights stay frozen while adapters are trained on top of it.

Double quantization

Quantizing the scaling constants of a block-wise quantized model as well, saving about 0.37 bits per parameter. Introduced with QLoRA.

Full fine-tuning

Updating every weight of a pretrained model. It is the most flexible option and also the most expensive in memory, compute and storage. Compare in LoRA vs full fine-tuning.

GPTQ

A post-training quantization method for generative pretrained transformers, often used to compress a model to 3–4 bits for faster inference.

LoRA (Low-Rank Adaptation)

A PEFT method that freezes base weights and learns a low-rank update ΔW = B·A for selected layers. See the LoRA guide.

LoRA alpha

A scaling factor. The adapter's update is multiplied by α / r, which controls how strongly the adapter shifts the base model.

Multi-tenant fine-tuning

Training separate adapters for many customers, characters or users on one shared base model. See multi-adapter training.

NF4 (4-bit NormalFloat)

A 4-bit data type whose levels are placed at quantiles of a normal distribution. It suits pretrained weights, which are roughly normally distributed.

Paged optimizer

An optimizer whose state can move between GPU and CPU memory through unified memory, which avoids out-of-memory crashes during spikes.

PEFT (Parameter-Efficient Fine-Tuning)

The family of methods that adapt large models by training only a small number of parameters. It includes LoRA, prefix tuning, prompt tuning and adapters. It is also the name of Hugging Face's library for these methods.

QLoRA

LoRA trained on top of a 4-bit NF4-quantized base model, using double quantization and paged optimizers. See the QLoRA guide.

Rank (r)

The inner dimension of a LoRA update. It sets adapter capacity and size. Common values are 8–64.

Shared-pass training

Flora's patent-pending technique. When many adapters on a frozen base model share training prompts, each shared prompt runs through the base model once, and the cached output trains every adapter. See Flora.

Target modules

The layers of a model that receive LoRA adapters, for example attention query and value projections, or all linear layers.