Training Many LoRA Adapters at Once: Multi-Tenant LLM Fine-Tuning

Personalised AI products end up with one adapter per customer, character or user. Serving many adapters is mostly solved. Training them efficiently is the part most teams still pay too much for.

By ForGen AI · Updated October 23, 2026

The multi-tenant fine-tuning pattern

A lot of LLM products end up with the same structure: one base model, many small adapters. A support-bot platform trains an adapter per business. A game studio trains one per character. A personal-assistant app trains one per user. LoRA makes each adapter small and cheap to store, so the per-tenant model is practical.

Serving many adapters

Inference for this pattern is a well-studied problem. Systems such as S-LoRA and Punica (both 2023), along with multi-LoRA support in vLLM and LoRAX, keep one copy of the base model in GPU memory and batch requests for different adapters together. Thousands of adapters can share a single GPU.

Training many adapters: the duplicated work

Training usually doesn't get the same treatment. The typical pipeline launches one fine-tuning job per adapter, and every job runs the large base model over its whole dataset. Tenant datasets tend to come from the same onboarding template, so they overlap heavily:

The inputs are the same and the targets differ. With a frozen base model, the base model's output for "What are your business hours?" is identical for both tenants, yet a per-adapter pipeline computes it once per tenant.

Shared-pass training

Flora, a patent-pending technique from ForGen AI, splits training into two phases:

1. Shared phase: training sets are pooled by common inputs. Each shared prompt goes through the frozen base model once. The output (logits or embeddings) is stored and used to train every adapter whose dataset contains that prompt, each toward its own target.
2. Individualized phase: prompts unique to a single adapter train in the usual way.

For N adapters with S shared prompts and U unique prompts each, base-model forward passes per epoch drop from N·(S+U) to S + N·U. With 500 adapters, 40 shared and 10 unique prompts, that is 25,000 passes reduced to 5,040. Try your own numbers.

Design tips for adapter fleets

• Make onboarding templated. The more tenants answer the same questions, the more compute can be shared, and the more consistent the adapters are.
• Use an application-specific base model. Fine-tune the base once on your prompt structure (for example business FAQ fields or RPG character attributes), then train per-tenant adapters on top of it.
• Keep ranks small. Per-tenant adapters rarely need more than r=8–16, which keeps storage and serving cheap.
• Quantize the base. A 4-bit NF4 base (QLoRA) lowers the memory needed for training.

FAQ

Can multiple LoRA adapters share one base model?

Yes. Serving systems such as S-LoRA, Punica, vLLM and LoRAX keep one base model in GPU memory and apply a different adapter per request, so thousands of adapters can share a GPU.

Why is training many adapters expensive?

Each adapter is usually trained as its own job, so every job runs the full base model forward pass on its whole dataset. When datasets share prompts, much of that compute is duplicated.

How does shared-pass training help?

If the base model is frozen and a prompt is the same across adapters, the base model's output for that prompt is identical. Computing it once and reusing it for every adapter removes the duplicate forward passes.