Multi-LoRA Serving: Many Adapters on One GPU

When multi-LoRA beats one model per tenant

A platform with per-customer fine-tunes can merge each one into a full model and run a replica per tenant, or keep one shared base resident and apply a small low-rank adapter per request (multi-LoRA serving). A dedicated replica costs a full model's GPU memory whatever the tenant's traffic. A shared base costs that memory once, plus a small per-adapter increment and a throughput penalty for the adapter math. The rule:

• Many tenants, low or bursty per-tenant traffic: multi-LoRA wins. GPU count follows aggregate load instead of tenant count.
• Few tenants, each saturating at least one GPU: dedicated merged replicas win. A merged adapter adds no inference latency ^[1], and you avoid adapter-slot scheduling limits.
• Mixed: keep a merged replica for the one or two heavy tenants and a multi-LoRA pool for the long tail.

A worked break-even, using placeholder throughput figures you must replace with your own measurements:

• Dedicated replicas: 40 tenants x 1 GPU = 40 GPUs minimum, before any redundancy, and most of them idle.
• Multi-LoRA pool: effective capacity 10 x (1 - 0.20) = 8 req/s per GPU; 30 / 8 = 3.75, rounded up to 4, plus one for N+1 = 5 GPUs.

The crossover comes when per-tenant peak load approaches one GPU's capacity; then the dedicated replica stops idling and the adapter penalty dominates. Our model routing cost model applies the same fleet reasoning to whole models.

Partitioning rarely fits: NVIDIA lists H100 MIG as "up to 7 MIGS @ 10GB" ^[2], while a 7B BF16 model needs about 15 GB for weights alone (computed below). See GPU sharing options (MIG/MPS).

LoRA recap and why adapters are small

LoRA freezes the pretrained weight matrix W0 (d x k) and learns a low-rank update ΔW = BA, where B is d x r, A is r x k, and the rank r is much smaller than d or k ^[1]. The forward pass becomes h = W0x + BAx. The paper reports that, compared with GPT-3 175B fine-tuned with Adam, LoRA "can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times" ^[1]. The Punica authors summarize the serving consequence: each fine-tuned model "only adds 0.1% to 1% of the model weight", and LoRA is usually applied to all dense projections in the attention block and the MLP ^[3].

An adapted layer with input width k and output width d holds r x (k + d) adapter parameters; the rest is bookkeeping over layers and modules.

How batching across adapters works

Each projection is split into two paths:

1. Shared base path. Every request in the batch goes through one dense GEMM against W0.
2. Per-adapter path. Each token row is multiplied by its own adapter's A and B, a gather-style operation over mixed adapters.

Punica introduced a CUDA kernel for the second path, Segmented Gather Matrix-Vector Multiplication (SGMV). The paper reports "negligible performance differences between batching the same LoRA models and batching different LoRA models" ^[3]. Its abstract reports 12x higher throughput than state-of-the-art serving systems of the time, "while only adding 2ms latency per token" ^[4]. S-LoRA added three things: all adapters kept in host memory and fetched on demand, a "Unified Paging" pool that holds adapter weights and KV cache together, and custom kernels for heterogeneous ranks that avoid padding. It reports serving "thousands of LoRA adapters on a single GPU" and up to 4x the throughput of HuggingFace PEFT and vLLM with naive LoRA support ^[5]. In its experiments it served 2,000 adapters at once ^[6].

Both multiples were measured against 2023 baselines; they do not predict gaps between current releases. Today's vLLM uses a Punica-derived wrapper (vllm/lora/punica_wrapper/) for its LoRA kernels ^[7]. SGLang defaults to a chunked SGMV backend (csgmv), which its docs say achieves "20% to 80% latency improvements over the basic triton backend" ^[8].

GPU memory budget

A multi-LoRA GPU has four memory consumers:

1. Base weights. Parameters x bytes per parameter.
2. Adapter slots. GPU-resident adapter buffers, sized by the engine's slot configuration rather than by the adapters you actually load (see the rank section).
3. Activations, CUDA graphs and runtime overhead. Measure this on your stack; do not guess.
4. KV cache. Everything left over.

Worked example: Qwen2.5-7B-Instruct on an 80 GB H100

The model card lists 7.61B parameters, 28 layers and GQA with 28 query heads and 4 KV heads (Qwen2.5-7B-Instruct). Its config gives hidden_size 3584 and intermediate_size 18944 (config.json), so head_dim = 3584 / 28 = 128.

Base weights (BF16): 7.61e9 x 2 bytes = 15.22 GB.

Adapter parameters per layer, per unit of rank (our arithmetic, all seven projections targeted):

Across 28 layers: 90,112 x 28 = 2,523,136 parameters per unit of rank.

A q_proj/v_proj-only adapter at r = 16 is (7,168 + 4,096) x 28 x 16 x 2 bytes = 10.1 MB: target modules matter as much as rank.

KV cache per token (BF16): 2 (K and V) x 28 layers x 4 KV heads x 128 x 2 bytes = 57,344 bytes. Our KV cache memory calculator walks through the same formula for other models.

Usable budget: vLLM's --gpu-memory-utilization default is 0.92 in the current docs and in v0.32.0 source ^[9]. That gives 80 GB x 0.92 = 73.6 GB.

With 8 GPU adapter slots at a padded rank of 64 (2.58 GB, derived in the next section) and an assumed 4 GB for activations and graphs:

73.6 - 15.22 - 2.58 - 4 = 51.8 GB for KV, or 51.8e9 / 57,344 ≈ 903,000 tokens.

The slots cost 2.58e9 / 57,344 = 45,056 tokens of KV, about eleven concurrent 4K-token sequences. Cheap here; sharper on larger bases or generous slot settings. The KV cache capacity planner shows how to turn the remaining tokens into concurrency.

vLLM configuration

The relevant flags, checked against vllm/config/lora.py at tag v0.32.0 (released 2026-10-18) and the "latest" docs on 2026-10-28:

Sources: vLLM engine arguments, lora.py @ v0.32.0. The LoRA feature page adds that "when --lora-target-modules is not specified, LoRA will be applied to all supported modules in the model" ^[10].

Two defaults bite:

• --max-loras defaults to 1. In v0.32.0 the V1 scheduler counts distinct adapters among the running requests scheduled in the current step ^[11]; on main after the release (commit 8d3f6f7) it counts all running requests, including those not scheduled this step ^[12]. Either way, a waiting request for a new adapter is skipped while the cap is reached. Base-model requests (adapter ID 0) do not count. At the default, a long generation for tenant A blocks tenant B's first token.
• --max-lora-rank defaults to 16; higher-rank adapters need it raised.

Runtime loading. Beyond --lora-modules name=path at startup, POST /v1/load_lora_adapter and /v1/unload_lora_adapter work once VLLM_ALLOW_RUNTIME_LORA_UPDATING=True; load_inplace: true replaces an adapter of the same name. The docs warn: "This feature comes with security risks. It should not be used in production unless it is an isolated, fully trusted environment." Resolver plugins (local filesystem, Hugging Face Hub) need the same variable, and the docs call remote downloads "insecure and not intended for use in production environments" ^[10].

Observability. vLLM exports vllm:lora_requests_info. Its labels carry per-adapter running and waiting counts (as comma-separated strings) and the static max_lora value ^[13]. Rising waiting counts at moderate GPU utilization usually mean the max_loras cap is binding.

SGLang configuration

From the SGLang server arguments, LoRA docs and fields/lora.py @ v0.5.22 (v0.5.22 released 2026-10-19):

Differences from vLLM that matter in practice:

• Base traffic consumes a slot in SGLang. In vLLM it does not.
• Dynamic loading. For /load_lora_adapter and /unload_lora_adapter, SGLang recommends setting --max-lora-rank and --lora-target-modules explicitly at startup. Otherwise they are inferred from the startup adapters, and a later, larger adapter may not fit.
• Pinning. A pinned adapter "will not be evicted from GPU memory during runtime". The number of pinned adapters is capped at max-loras-per-batch - 1 to prevent starvation of unpinned requests.
• Separate pools. S-LoRA-style Unified Paging is listed under SGLang's "still under development" features. Adapter memory and KV cache are therefore budgeted separately, as in vLLM ^[8].

Rank and slot sizing

vLLM preallocates GPU adapter buffers when the model's LoRA layers are created. For each adapted linear layer, it allocates lora_a_stacked with shape (max_loras, 1, max_lora_rank, input_size) and lora_b_stacked with shape (max_loras, 1, output_size, max_lora_rank), in lora_dtype ^[14]. The formula below covers the seven decoder projections; adapters on embeddings, lm_head or MoE experts allocate additional buffers. Each slot is therefore sized for max_lora_rank, not for the adapter that occupies it. A rank-8 adapter in a rank-64 server uses a rank-64 slot. This is why the docs say to set the flag "to the maximum rank among all LoRA adapters you plan to use" and that a value much larger than needed "wastes memory and can cause performance issues" ^[10].

For the Qwen2.5-7B example, GPU slot memory = max_loras x max_lora_rank x 2,523,136 x 2 bytes (our arithmetic, single GPU, all seven projections):

The last row is the classic mistake: one rank-256 experiment raises the server-wide cap and turns over a quarter of usable HBM into adapter padding. Avoid it:

• Standardize ranks per pool. If most tenants train at r = 16 and a few at r = 64, run two pools rather than padding everyone to 64.
• Restrict target modules. If no adapter touches the MLP, --lora-target-modules q_proj v_proj style restrictions shrink both slots and per-token LoRA compute.
• Size max_loras to concurrent distinct adapters, not catalogue size. The catalogue lives in host memory.

Tensor parallelism. With TP, vLLM shards part of the LoRA computation by default and offers --fully-sharded-loras, which the docs say is "likely faster" at high sequence length, max rank or TP size ^[9]. Recompute per-GPU slot memory from the sharded shapes rather than dividing by TP degree.

Swap latency and cache policy

Adapter residency has three tiers:

1. GPU slot (max_loras / max-loras-per-batch): immediately usable.
2. Host RAM cache (max_cpu_loras / max-loaded-loras): one PCIe copy away.
3. Storage (local NVMe, shared filesystem, object store): file read, deserialization, then the copy.

PCIe floor (our arithmetic). NVIDIA lists H100 PCIe Gen5 at 128 GB/s ^[2]. Taking half of that as a one-direction figure (an assumption that the listed number is bidirectional), a rank-16 adapter (80.7 MB) needs at least 80.7e6 / 64e9 ≈ 1.3 ms to copy, and a rank-64 adapter (323 MB) about 5.0 ms. These are lower bounds; storage reads usually dominate cold loads.

Host RAM sizing (our arithmetic). At least the adapter file size times the cache count: 256 cached rank-16 adapters x 80.7 MB ≈ 20.7 GB of host memory. Cheap on a GPU server, hence S-LoRA's pattern of keeping all adapters in main memory ^[5].

Overlap loading is not free. SGLang's async loading needs pinned host memory, caps max-loaded-loras at 2x max-loras-per-batch, and can break up multi-adapter prefill batches. The docs' worked case: four adapters with 2 ms loads and 20 ms prefills take about 2 x 4 + 20 = 28 ms synchronously but up to 2 + 4 x 20 = 82 ms with overlap. Enable it only when loading is the bottleneck; the docs report "~35% reduction in median TTFT" under adversarial churn ^[8].

Measuring it. Send a request to an uncached adapter, then repeat it; the TTFT difference is the cold-load penalty. Repeat after evicting from GPU only to isolate the PCIe tier, and record p50/p99 per tier. A tenant's first request after idle pays this, so it belongs in the SLO.

Tenancy, isolation and security

• Runtime load endpoints are an admin plane. vLLM explicitly limits runtime updating to isolated, fully trusted environments ^[10]. Keep /v1/load_lora_adapter (and SGLang's /load_lora_adapter) off the tenant-facing gateway; drive it from a deployment controller pulling signed artifacts from a store you control.
• Adapter selection is just a model string. Requests choose an adapter through the model field ^[10]. Without a gateway that maps API keys to allowed adapter names, any client can call any tenant's fine-tune. Enforce the mapping before the engine.
• Noisy neighbours. A high-concurrency tenant holds slots and KV pages and delays others' admission (the max_loras rule above). Apply per-tenant rate and concurrency limits at the gateway.
• Data separation. Tenants share HBM, kernels and the scheduler; contracts requiring hardware isolation need dedicated replicas or MIG partitions.

Benchmark protocol

Single-adapter benchmarks overstate capacity. A representative test:

1. Adapter popularity: draw adapter IDs from a Zipf distribution over your real catalogue size, with at least two skews (head-heavy and near-uniform).
2. Rank mix: use the ranks and target modules you actually ship, with max_lora_rank set as in production.
3. Cache state: include a cold-start phase with empty GPU and host caches, then a warm steady state.
4. Slot sweep: vary slot counts (4, 8, 16, 32) and record startup KV capacity alongside throughput.
5. Metrics: TTFT and ITL p50/p99 per adapter, goodput under SLO, waiting counts. Unpopular adapters' tails fail first.
6. Baseline: the same load on the base model alone, giving the penalty used in the break-even.

Limits: when to merge or fine-tune fully

• Heavy single tenants. Once a tenant needs its own GPU anyway, merge (W = W0 + BA) to remove the extra matmuls and slot contention ^[1].
• Rank creep. Slots and copies grow linearly with rank; tenants needing very high ranks or full fine-tunes belong elsewhere.
• Quantized bases. An adapter trained against BF16 weights sees different numerics on an FP8 or INT4 checkpoint; validate quality per adapter. Our FP16/FP8/INT4 inference cost analysis covers the memory side of that choice.
• Unsupported modules and models. vLLM requires models that implement SupportsLoRA, and tower/connector LoRA for multimodal models is experimental ^[10]. SGLang lists embedding-layer LoRA as still under development, and LoRA with DP attention currently requires the Triton backend ^[8]. Check adapters that train embeddings or lm_head.

GPU Smith can help size a multi-tenant adapter pool against measured workloads.

Sources

1. Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models," arXiv 2106.09685. https://arxiv.org/abs/2106.09685
2. Chen et al., "Punica: Multi-Tenant LoRA Serving," arXiv 2310.18547 (abstract and PDF). https://arxiv.org/abs/2310.18547
3. Sheng et al., "S-LoRA: Serving Thousands of Concurrent LoRA Adapters," arXiv 2311.03285 (abstract and PDF). https://arxiv.org/abs/2311.03285
4. vLLM documentation, "LoRA Adapters" (latest, page dated October 21, 2026). https://docs.vllm.ai/en/latest/features/lora/
5. vLLM documentation, "Engine Arguments" (latest). https://docs.vllm.ai/en/latest/configuration/engine_args/
6. vLLM documentation, "Metrics" design notes. https://docs.vllm.ai/en/latest/design/metrics/
7. vLLM source at tag v0.32.0 (2026-10-18): vllm/lora/layers/base_linear.py, vllm/config/lora.py. https://github.com/vllm-project/vllm/blob/v0.32.0/vllm/lora/layers/base_linear.py
8. vLLM source, vllm/v1/core/sched/scheduler.py at v0.32.0 and at main commit 8d3f6f7 (2026-10-29 UTC). https://github.com/vllm-project/vllm/blob/v0.32.0/vllm/v1/core/sched/scheduler.py and https://github.com/vllm-project/vllm/blob/8d3f6f7e788a3bcbd73cae851e7717d222eb8bb9/vllm/v1/core/sched/scheduler.py
9. SGLang documentation, "Server Arguments," and python/sglang/srt/arg_groups/fields/lora.py @ v0.5.22. https://docs.sglang.io/advanced_features/server_arguments.html
10. SGLang documentation, "LoRA Serving." https://docs.sglang.io/advanced_features/lora.html
11. Qwen, "Qwen2.5-7B-Instruct" model card and config.json. https://huggingface.co/Qwen/Qwen2.5-7B-Instruct
12. NVIDIA, "H100 Tensor Core GPU" product page (specifications table). https://www.nvidia.com/en-us/data-center/h100/
13. PyPI release pages for vllm (0.32.0) and sglang (0.5.22), checked 2026-10-28. https://pypi.org/project/vllm/ and https://pypi.org/project/sglang/