LoRA and QLoRA GPU Memory: How Much VRAM Do You Need?

Why full fine-tuning memory math does not transfer

A full fine-tuning run and a LoRA run on the same model can differ in GPU memory by an order of magnitude, and the difference comes almost entirely from which tensors need gradients and optimizer state. The ZeRO paper gives the standard accounting for mixed-precision Adam: an fp16 copy of the parameters and of the gradients (2Ψ + 2Ψ bytes for a model of Ψ parameters), plus an fp32 copy of the parameters, momentum and variance (4Ψ + 4Ψ + 4Ψ bytes), for a total of 16Ψ bytes of model state before any activations ^[1].

Low-Rank Adaptation (LoRA) breaks that multiplier. The pretrained weight matrix W₀ is frozen and "does not receive gradient updates", while two small matrices B (d × r) and A (r × k) carry the trainable update ^[2]. Frozen weights need only their storage bytes; the 16-byte ledger applies only to the adapter. The LoRA authors report that for a large Transformer trained with Adam this cuts VRAM by up to two-thirds when r ≪ d_model, from 1.2 TB to 350 GB on GPT-3 175B, because optimizer states for the frozen parameters are no longer stored ^[2]. QLoRA then attacks the remaining dominant term, the frozen base, by storing it in 4 bits ^[3].

The full-training ledger, with sharding arithmetic, is in the LLM training memory calculator; this guide narrows it to parameter-efficient fine-tuning (PEFT) and maps results to GPU tiers.

The five memory buckets

A single-GPU PEFT step holds five categories of memory. The LoRA and QLoRA columns below are derived from the ZeRO accounting above.

Table 1. PEFT memory ledger

Buckets 2-4 together cost roughly 16 bytes per adapter parameter under the ZeRO accounting. That figure also holds if adapters are kept in fp32 (4 B weight + 4 B gradient + 8 B Adam moments), and drops to about 12 B/param if adapter weights and gradients are kept in bf16 with fp32 Adam moments (2 + 2 + 8). This guide uses 16 B/param as the conservative planning value.

Bucket 5 deserves emphasis. The QLoRA authors measured that for a 7B LLaMA model at batch size 1 with LoRA weights equal to 0.2% of the model, the LoRA input gradients occupied 567 MB while the LoRA parameters took only 26 MB; with gradient checkpointing the input gradients fell to an average of 18 MB per sequence, against 5,048 MB for the 4-bit base model ^[3]. Their conclusion was that "aggressively reducing the amount of LoRA parameter yields only minor memory benefits".

LoRA mechanics that matter for memory

Three configuration choices determine the adapter's size. In the Hugging Face PEFT library, r is "the rank of the update matrices", where lower rank means fewer trainable parameters; target_modules names the modules (for example attention blocks) that receive LoRA updates; and lora_alpha is the scaling factor ^[4]. Each adapted d_in × d_out linear layer adds r × (d_in + d_out) trainable parameters.

The original paper adapted only attention weights, and a GPT-3 configuration with r = 4 on the query and value projections shrank the checkpoint by roughly 10,000×, from 350 GB to 35 MB ^[2]. QLoRA found that this query/value-only practice did not match full fine-tuning on large base models, that LoRA on all linear transformer-block layers was required, and that the rank r did not affect performance once all layers were adapted ^[3].

Table 2. Derived LoRA trainable parameters for Qwen2.5 models

Counts are derived from the published configurations (hidden, intermediate and layer sizes, grouped-query KV heads) in the Qwen2.5-7B, Qwen2.5-14B and Qwen2.5-72B model cards and config files), summing r × (d_in + d_out) over the q, k, v, o, gate, up and down projections. Even the largest case, r = 64 on every linear layer of a 72B model, holds about 13.5 GB of adapter state at 16 B/param, roughly 9% of the 145 GB bf16 base.

For deployment, merging removes the adapter cost entirely: PEFT states that LoRA "does not add any inference latency because adapter weights can be merged with the base model", using merge_and_unload() ^[4].

QLoRA: a 4-bit frozen base

QLoRA "backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters", and its abstract claims it reduces memory "enough to finetune a 65B parameter model on a single 48GB GPU" while preserving 16-bit fine-tuning task performance ^[3]. Three components carry the memory savings:

• 4-bit NormalFloat (NF4). A data type the authors describe as "information theoretically optimal for normally distributed weights". Weights are stored in NF4 and dequantized to bf16 for every matrix multiplication, so compute runs in 16-bit while storage is 4-bit ^[3].
• Double Quantization. Blockwise quantization with 32-bit constants and a block size of 64 adds 32/64 = 0.5 bits per parameter. Quantizing those constants to FP8 with a block size of 256 reduces the overhead to 8/64 + 32/(64 × 256) = 0.127 bits, saving 0.373 bits per parameter, about 3 GB on a 65B model.
• Paged optimizers. Optimizer state is allocated in NVIDIA unified memory and "automatically evicted to CPU RAM when the GPU runs out-of-memory", then paged back for the update step. The authors call this "critical" for 33B/65B tuning on a single 24/48 GB GPU; at batch size 16 on 48 GB GPUs they found no 65B slowdown, though they did not measure paging overhead in general.

An NF4 weight with double quantization therefore costs 4 + 0.127 = 4.127 bits, or ≈0.516 bytes per parameter (derived), but only in quantized layers. The Transformers bitsandbytes integration replaces torch.nn.Linear layers with Linear4bit ^[5], and by default its quantizer leaves the output embedding (lm_head) and tied weights unconverted "for numerical stability reasons" (Transformers source, quantizers/base.py, release v5.21, 2026-10-26). Input embeddings are not linear layers and also stay 16-bit. This is not negligible: Qwen2.5-72B has 72.7B parameters of which 70.0B are non-embedding ^[6], so about 2.7B parameters remain in bf16.

The conditions behind "65B on 48 GB"

The paper's memory appendix reports a 45.0 GB footprint for the 65B LLaMA model at batch size 1, sequence length 512, with gradient checkpointing, and warns that "if one uses a larger batch size, or if a long sequence is processed, the activation gradient might consume a considerable amount of memory" ^[3]. The same appendix notes that the 33B model "does not quite fit" into 24 GB without paged optimizers. Reported hyperparameters were r = 64, α = 16, LoRA on all linear layers, NF4 with double quantization and bf16 compute (appendix B.2). The 48 GB claim is a statement about short sequences with paging as a safety valve, not about long-context fine-tuning.

The paper puts regular 16-bit fine-tuning of LLaMA 65B at "more than 780 GB" ^[3] without stating its accounting; 780 GB equals 65B × 12 bytes, whereas the ZeRO 16Ψ accounting gives 1,040 GB (derived).

Activations and gradient checkpointing

Gradient checkpointing stores selected layer outputs and recomputes the rest during backward. The original paper describes "an algorithm that costs O(√n) memory to train a n layer network, with only the computational cost of an extra forward pass per mini-batch", and measured roughly 30% additional runtime in its experiments ^[7]. ZeRO gives a transformer-scale example: a 1.5B GPT-2 at sequence length 1K and batch size 32 needs about 60 GB of activations, reduced to about 8 GB with checkpointing at a 33% recomputation overhead ^[1].

LoRA does not shrink activations much, because gradients still flow through every frozen layer to reach the adapters. For planning, this guide uses two derived terms with layer-boundary checkpointing at micro-batch b and sequence length s:

• Checkpointed layer inputs: layers × b × s × hidden × 2 bytes (bf16). For Qwen2.5-72B at b = 1, s = 2,048: 80 × 2,048 × 8,192 × 2 ≈ 2.68 GB.
• Output logits: b × s × vocabulary × 4 bytes if materialized in fp32. With Qwen2.5's 152,064-token vocabulary at s = 2,048: 2,048 × 152,064 × 4 ≈ 1.25 GB, independent of model size.

Both scale linearly with b × s. They are illustrative lower bounds: recomputed intermediates, attention workspace, temporary buffers and fragmentation (ZeRO's "residual states") come on top, hence the 15% reserve below. Real peaks must be measured on the target stack.

Worked examples

Assumptions (all illustrative): Qwen2.5 architectures from Table 2; LoRA on all linear layers at r = 16; adapter state at 16 B/param; micro-batch 1, sequence length 2,048; checkpointing on; bf16 compute; a 15% reserve on the subtotal; single GPU unless noted.

Table 3. Derived single-job memory estimates (GB)

The arithmetic for the 72B QLoRA row: non-embedding parameters 70.0B × 0.516 B ≈ 36.11 GB; the remaining 72.7B − 70.0B = 2.7B parameters (mostly the input embedding and output head, 2 × 152,064 × 8,192 ≈ 2.49B) kept in bf16 × 2 B ≈ 5.40 GB; adapters 210.5M × 16 B ≈ 3.37 GB; activations 3.93 GB; subtotal 48.8 GB; with reserve 56.1 GB.

Three readings follow. A 7B bf16 LoRA run fits a 24 GB card only at short sequences and micro-batch 1. A 14B model is a natural QLoRA job for 24 GB and a bf16 LoRA job for 48 GB. The 72B QLoRA estimate exceeds 48 GB at s = 2,048 because of the bf16 embeddings and longer sequence, so the paper's 45.0 GB at s = 512 is a best case; a 48 GB card would need paged optimizers, r = 8 (about 1.7 GB of adapter state) or shorter sequences, while 80 GB fits with margin. A 72B bf16 LoRA run exceeds a single 141 GB H200 and needs sharding.

For DeepSeek fine-tuning, the distilled checkpoints are dense: DeepSeek-R1-Distill-Qwen-14B is built on Qwen2.5-14B and the 70B distill on Llama-3.3-70B-Instruct ^[8], so the 14B row applies and the 72B row is a close proxy. The full R1 has 671B total parameters (37B activated), all of which must be resident: about 1,342 GB in bf16 or 346 GB at 0.516 B/param (derived), a multi-GPU job even under QLoRA. Serving-side hardware for R1 is covered in the DeepSeek R1 self-hosting cost analysis.

When ZeRO and FSDP still matter

ZeRO removes memory redundancy across data-parallel ranks in stages: partitioning optimizer state reduces model-state memory from 4Ψ + 12Ψ to 4Ψ + 12Ψ/N_d; adding gradient partitioning gives 2Ψ + 14Ψ/N_d; partitioning parameters as well gives 16Ψ/N_d ^[1]. Under LoRA, the optimizer and gradient terms are already tiny, so stages 1 and 2 save little. The dominant term is the frozen base, which only parameter sharding (ZeRO stage 3 or FSDP FULL_SHARD) divides across GPUs.

The Hugging Face PEFT documentation gives two measured reference points. LoRA (r = 8, all linear layers) on Llama-2-70B with FSDP on 8 × H100 80 GB, at sequence length 2,048 and per-device batch 8, consumed 72-80 GB per GPU. QLoRA with FSDP on 2 × 24 GB GPUs consumed 19.6 GB per GPU with CPU offloading and about 107 GB of CPU RAM, or 35.6 GB per GPU with offloading disabled; the guide summarizes that the job took 16 × 80 GB GPUs for full fine-tuning and 8 × 80 GB with FSDP plus LoRA ^[9]. The page requires fsdp_use_orig_params=false for the savings and, as of PEFT v0.21.2 (2026-10-01), notes that merging under FSDP is unsupported and that paged_adamw_8bit errors on checkpoint save with FSDP plus QLoRA.

GPU selection

Capacity decides whether a job runs; bandwidth and interconnect decide how fast.

Table 4. GPU memory tiers for PEFT (vendor specifications)

For single-GPU PEFT, SXM versus PCIe matters little because no inter-GPU traffic occurs; once FSDP shards a 70B base, parameter all-gathers run every step over NVLink or PCIe and the interconnect matters. QLoRA also dequantizes each weight tensor to bf16 at every use ^[3], so step time on a bandwidth-limited card should be measured rather than inferred from bf16 LoRA results.

Own vs rent for fine-tuning

PEFT jobs are typically short and bursty, which favors rental. The own vs rent GPU TCO analysis (last updated 2026-10-14, rechecked 2026-10-28) places the rent-favorable zone below roughly 40% sustained utilization and cites an H100 market median of about $2.29-$3.12 per GPU-hour as of May 2026. On those figures, an illustrative 20-hour single-H100 QLoRA run of a 72B-class model costs about 20 × $2.29 ≈ $46 to 20 × $3.12 ≈ $62 in GPU time, excluding storage, data preparation and evaluation. Provider-level rates for H100, H200 and consumer cards, dated July 2026, are compiled in the GPU cloud rental prices survey. Ownership becomes defensible when fine-tuning is continuous (frequent retraining, many adapters) or when data cannot leave a private environment.

Checklist and failure modes

When a PEFT job runs out of memory, the cheapest fixes come first, in this order:

1. Sequence length. Activation terms scale linearly with s; truncating or packing to a shorter maximum is the largest lever.
2. Micro-batch size. Reduce to 1 and recover the effective batch with gradient accumulation.
3. Gradient checkpointing. Enable it if off; expect roughly 30% extra step time.
4. Rank and target modules. Lowering r saves little (Table 2), but on 70B-class models it can free several GB.
5. Quantize the base. Move from bf16 LoRA to QLoRA (NF4 with double quantization) and enable a paged optimizer.
6. Shard. Use FSDP full sharding with use_orig_params=false, with CPU offload as a last resort.

Common failure modes: assuming the whole model is 4-bit when embeddings and lm_head stay 16-bit; out-of-memory spikes on the longest batch, which paged optimizers exist to absorb; ignoring fp32 logits on large-vocabulary models; and planning a 72B job against the "65B on 48 GB" headline without its 512-token sequences.

GPU Smith can help review a fine-tuning memory budget or GPU shortlist against a specific model and dataset.

Sources

1. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (full text) - Rajbhandari et al., arXiv; 16Ψ mixed-precision Adam accounting, K = 12, GPT-2 1.5B 24 GB example, activation checkpointing example, stage-wise partitioning formulas (accessed 2026-10-28)
2. LoRA: Low-Rank Adaptation of Large Language Models (full text) - Hu et al., arXiv; frozen W₀, BA decomposition, 1.2 TB to 350 GB VRAM, 10,000× checkpoint reduction, attention-only adaptation (accessed 2026-10-28)
3. QLoRA: Efficient Finetuning of Quantized LLMs (full text) - Dettmers et al., arXiv; 65B on 48 GB, NF4, double quantization arithmetic, paged optimizers, appendix G memory footprint, hyperparameters, >780 GB baseline (accessed 2026-10-28)
4. Training Deep Nets with Sublinear Memory Cost (full text) - Chen et al., arXiv; O(√n) checkpointing with one extra forward pass, ~30% runtime overhead (accessed 2026-10-28)
5. LoRA conceptual guide - Hugging Face PEFT docs; definitions of r, target_modules, lora_alpha; merge_and_unload and no added inference latency (accessed 2026-10-28)
6. Fully Sharded Data Parallel with PEFT - Hugging Face PEFT docs (text identical in the v0.21.2 release source); 70B LoRA on 8 × H100 and QLoRA on 2 × 24 GB measurements, required FSDP settings, caveats (accessed 2026-10-28)
7. Bitsandbytes quantization - Hugging Face Transformers docs; Linear4bit replaces nn.Linear layers, NF4 and nested quantization options (accessed 2026-10-28)
8. transformers/src/transformers/quantizers/base.py - Hugging Face, GitHub, release v5.21 (2026-10-26); get_keys_to_not_convert keeps lm_head and tied weights unquantized (accessed 2026-10-28)
9. Qwen2.5-7B model card - Qwen, Hugging Face; 7.61B parameters, 6.53B non-embedding, 28 layers, GQA 28/4, config dimensions (accessed 2026-10-28)
10. Qwen2.5-14B model card - Qwen, Hugging Face; 14.7B parameters, 13.1B non-embedding, 48 layers, GQA 40/8, config dimensions (accessed 2026-10-28)
11. Qwen2.5-72B model card - Qwen, Hugging Face; 72.7B parameters, 70.0B non-embedding, 80 layers, GQA 64/8, config dimensions (accessed 2026-10-28)
12. DeepSeek-R1 model card - DeepSeek, Hugging Face; 671B total / 37B activated parameters, distill base models (accessed 2026-10-28)
13. NVIDIA L4 Tensor Core GPU - NVIDIA; 24 GB, 300 GB/s, 72 W, low-profile PCIe (accessed 2026-10-28)
14. NVIDIA L40S GPU - NVIDIA; 48 GB GDDR6 with ECC, 864 GB/s, PCIe Gen4 x16, 350 W (accessed 2026-10-28)
15. NVIDIA RTX PRO 6000 Blackwell Server Edition - NVIDIA; 96 GB GDDR7 with ECC, 1,597 GB/s, up to 600 W (accessed 2026-10-28)
16. NVIDIA H100 Tensor Core GPU - NVIDIA; H100 SXM 80 GB / NVL 94 GB, bandwidth, NVLink figures (accessed 2026-10-28)
17. NVIDIA H200 Tensor Core GPU - NVIDIA; 141 GB, 4.8 TB/s, SXM and NVL form factors, NVLink (accessed 2026-10-28)