Host CPU and System RAM Sizing for GPU Servers

The short answer, and why it is usually guessed

GPU server quotes get scrutinised for the accelerators and filled in with the vendor's default CPU and DIMMs, leaving GPUs idling behind a starved host or a terabyte of DRAM nobody planned to use. There is no published law for host sizing, but there are two anchors. NVIDIA's own 8-GPU DGX systems ship with 14 to 16 CPU cores and about 256 GB of DRAM per GPU. vLLM documents a hard floor for inference: at least 2 + N physical cores for N GPUs, more with data parallelism. Size cores upward from that floor using measured CPU work (tokenization, multimodal preprocessing, data loading). Size RAM as a sum of named consumers: page cache for weights, a KV-cache offload tier, pinned buffers, checkpoint staging, plus headroom. The DGX ratios are one vendor's reference points, not requirements.

Reference ratios from NVIDIA systems

Specs are cited; the per-GPU and DRAM:HBM columns are our arithmetic.

Sources: DGX H100/H200 user guide; DGX B200 user guide; DGX B300 user guide, which names the CPUs but not their core count, taken from Intel's 6776P specification page (64 cores, 8 memory channels, 88 PCIe lanes, 350 W); GB200 NVL72 page ("1 Grace CPU | 2 Blackwell GPU" per superchip).

DRAM/GPU treats 2 TB as 2,048 GB (the H100/H200 and B200 guides specify 32 DIMMs, so 64 GB each by our arithmetic; the B300 guide gives no DIMM count, only 128 GB DIMMs for the 4 TB upgrade). DRAM:HBM is 2,048 / 640 = 3.2 for H100 and 2,048 / 2,304 = 0.89 for B300.

1. Cores per GPU have barely moved: 14 to 16 across three DGX generations, while GPU FLOPS rose sharply.
2. DRAM per GPU is flat at about 256 GB in the default configs, so DRAM:HBM has collapsed from 3.2x on H100 to under 1x on B300. At the default 2 TB, a B300 host cannot hold one full copy of its HBM. Designs that mirror HBM contents into DRAM (KV offload, checkpoint staging) should price the 4 TB option.
3. Grace changes the shape: 36 cores and up to 240 GB per GPU on GB200.

The inference floor: vLLM's process model

vLLM's optimization and tuning guide (v0.32.0, released 2026-10-18) lists one API server process (which handles "HTTP requests, tokenization, and input processing"), one engine core process (which "runs the scheduler and coordinates GPU workers"), and one worker per GPU. "The minimum is 2 + N physical cores", and "Using fewer physical CPU cores than processes will cause contention and significantly degrade throughput and latency."

• Physical cores, not vCPUs. With hyperthreading "1 vCPU = 1 hyperthread = 1/2 physical CPU core", so the minimum becomes 2 x (2 + N) vCPUs. A request of 10 CPUs for an 8-GPU pod is half the floor on an SMT host.
• Data parallelism adds processes: "Minimum physical cores = A + DP + N + (1 if DP > 1 else 0)", where A is the API server count (default DP).

A DGX-class host has 112 to 128 cores, so the floor uses 10 to 15 percent of them. The floor is where vLLM says performance falls apart, not a target; everything above it is budget for CPU work that scales with traffic.

Where inference CPU time goes, and how to measure it

The same vLLM guide names the scaling CPU work: "tokenization, chat template rendering, and multi-modal data loading all run on CPU", and notes that more cores "and even the clock speed can significantly improve end-to-end performance." For multimodal requests, "By default, 8 CPU threads are used in each API server to load media items", and the multimodal processor cache defaults to 4 GiB (mm_processor_cache_gb), with total cache memory of mm_processor_cache_gb * data_parallel_size.

We do not quote a "tokens per second per core" figure for tokenization: it depends on tokenizer, prompt lengths, chat template and image sizes, and we found no source we would stand behind. Measure it:

1. Load-test at the target request rate with production-like prompts and media.
2. Watch per-process CPU (top -H or pidstat -t) for the API server and engine core. An API server pinned at 100 percent of a core while GPU utilization sags is the pattern vLLM describes: "CPU contention may be the bottleneck."
3. Add API servers (the guide's scale-out pattern) and re-measure. If throughput rises, input processing was the bottleneck. Scale-out disables multimodal IPC caching, and VLLM_MEDIA_LOADING_THREAD_COUNT may need lowering to avoid CPU exhaustion.
4. Record cores in use at target load and add burst headroom.

Image-heavy and long-document workloads are where 14 cores per GPU earn their cost.

Training hosts: data loaders, pinned memory, checkpoint staging

Each PyTorch DataLoader worker is a separate process, so 8 GPUs with 8 workers each is 64 decode-and-augment processes plus the ranks. Image and video augmentation can consume every core a DGX has; pre-tokenized text needs few. The signal to watch is step time stretching while loader workers are saturated.

Pinned memory is the second consumer. The PyTorch pinned-memory tutorial explains that pageable data must first be staged into "a page-locked copy", and warns that large pinned allocations "can monopolize a substantial portion of RAM" and "cannot be swapped out to disk." Count them.

Checkpoint staging is the third and usually the largest. The PyTorch async checkpoint recipe says async checkpointing copies models "into internal CPU-buffers" and "does raise CPU memory by a factor of checkpoint_size_per_rank X number_of_ranks", and that with pinned buffers "the same peak memory pressure being sustained" between steps.

Worked example: checkpoint staging for a 70B training run (hypothetical).

• Assumption: mixed-precision Adam. The ZeRO paper counts 2 (FP16 parameters) + 2 (gradients) + 12 (FP32 copy, momentum, variance) = 16 bytes per parameter. Checkpoints skip gradients: assume 14 bytes.
• Full checkpoint: 70 x 10^9 x 14 bytes = 980 GB.
• Sharded across 8 nodes (64 GPUs), each node stages 980 / 8 = 122.5 GB.
• On a 2 TB node that is about 6 percent of RAM. Unsharded state would stage 980 GB per node, nearly half of RAM, held continuously if buffers persist. Check your real checkpoint size before trusting the 14-byte assumption.

System memory sizing: a worked inference example

Scenario (hypothetical). One 8 x H100 80 GB node serving Qwen2.5-72B-Instruct in BF16 on vLLM as two TP=4 replicas (DP=2), with a CPU KV tier for prefix reuse.

Step 1: model facts. The Qwen2.5-72B-Instruct config gives 80 layers, 8 KV heads, hidden size 8,192 and 64 attention heads (head dim 8,192 / 64 = 128). Its safetensors index reports total_size 145,412,407,296 bytes, so weights are 145.4 GB.

Step 2: KV bytes per token (BF16). 2 (K and V) x 80 layers x 8 heads x 128 dims x 2 bytes = 327,680 bytes, about 0.33 MB per token.

Step 3: GPU-side KV, for scale. Assume 90 percent of 4 x 80 GB usable: 288 - 145.4 = 142.6 GB, or 142.6 x 10^9 / 327,680 = about 435,000 tokens per replica (an upper bound; activations ignored).

Step 4: CPU KV tier. vLLM's KV offloading guide sets the tier with cpu_bytes_to_use, "Total bytes of host memory reserved for the CPU tier across all workers (not per-worker)", and copies blocks "into pinned host memory". Target one million cached tokens per replica, a bit over twice the GPU tier: 10^6 x 327,680 = 327.7 GB per replica, 655.4 GB for both. This is pinned memory.

Step 5: weight staging and page cache. Loading from local NVMe typically goes through the Linux page cache; keeping the shards cached speeds restarts. Both replicas read the same files, so count 145.4 GB once.

Step 6: everything else. OS, runtime, Python heaps, CUDA host allocations, agents. We have no sourced figure; measure a loaded, idle node and call it X.

Total: 655.4 + 145.4 + X = 800.8 GB + X.

Decision. 1 TB leaves about 200 GB for X and headroom: workable but tight, since pinned pages cannot be reclaimed and page cache can. 2 TB allows doubling the KV tier or caching a second model. Without the KV tier the node needs about 145 GB plus X, and 512 GB is plenty. The KV tier, not GPU count, drives the decision.

Transfer cost check: a 32,768-token cached prefix is 32,768 x 327,680 = 10.7 GB, or 2.7 GB per GPU under TP=4. The H100 spec page rates PCIe Gen5 at 128 GB/s (both directions combined). At an assumed 50 GB/s achieved one-way, reloading takes about 54 ms per GPU. Whether that beats recomputing the prefix depends on your prefill speed; measure it.

Memory channels and bandwidth

The Xeon 6776P supports 8 DDR5 channels at 6,400 MT/s ^[1]. Theoretical peak per socket is 8 x 6,400 x 10^6 x 8 bytes = 409.6 GB/s. Populate every channel on both sockets with identical DIMMs; empty channels lose bandwidth that KV offload and checkpoint staging use. Check the vendor's population guide for speed at two DIMMs per channel before choosing 16 large or 32 small DIMMs.

PCIe lanes and topology

Lane arithmetic explains why 8-GPU servers do not attach everything directly to the CPUs. Eight GPUs at x16 is 128 lanes; eight 400G NICs at x16 is another 128; eight NVMe drives at x4 is 32. That totals 288 lanes, against 2 x 88 = 176 on a dual 6776P host. Devices therefore share upstream links through PCIe switches, so which GPU sits under which switch and socket matters for KV offload. Get the block diagram before ordering, and verify negotiated link width and speed with our PCIe link troubleshooting guide.

Grace versus x86 hosts

The Grace CPU Superchip page lists 144 Neoverse V2 cores, LPDDR5X options of 240, 480 and 960 GB, bandwidth "Up to 1024 GB/s for 240GB, 480GB" (768 GB/s for 960 GB), and NVLink-C2C "Up to 900 GB/s". In GB200 each 72-core Grace serves two GPUs with up to 480 GB LPDDR5X at up to 512 GB/s ^[2].

• For Grace: more cores per GPU, NVLink-C2C rather than PCIe between CPU and GPUs (per the GB200 page), and up to 512 GB/s of memory bandwidth per Grace against 409.6 GB/s theoretical for a 6776P socket.
• Against Grace: on-module memory fixes capacity at purchase (480 GB per GB200 superchip at most), and every host-side dependency needs a validated Arm build.
• For x86: up to 4 TB on DGX B200/B300, with on-site expansion using 128 GB DIMMs per the B300 guide.

Needing more than about 240 GB of host memory per GPU, for a large KV tier say, points to x86 with 4 TB. NVIDIA's next host CPU is covered in our Vera CPU specs article.

NUMA and binding

Two sockets means two NUMA domains; a KV tier allocated on the wrong socket crosses the inter-socket link on every transfer. Binding is covered in GPU/CPU/NIC affinity and rank binding; on Kubernetes, see pod-level resource managers and NUMA for making CPU and memory requests NUMA-aligned. For sizing: budget cores and RAM per socket, not per node.

Cost trade-offs

GPU idle time is paid at the GPU rate, so weigh the quoted price step to a larger CPU SKU against even a few percent of GPU utilization lost. Do not trim cores below measured need plus headroom, but do not buy 4 TB by default either: DRAM without a named consumer is waste. Reasonable savings: fewer, higher-clocked cores for text-only inference, and 1 TB instead of 2 TB when there is no KV tier and checkpoints are sharded. Get DRAM pricing at order time; it has been volatile.

Sizing worksheet

Outputs. Cores per socket = (vLLM floor + measured preprocessing + data loaders) / 2, plus burst headroom, doubled for SMT vCPUs. RAM = page cache + KV tier + staging + caches + X + headroom, rounded up to an evenly populated channel configuration. Compare with the 14 to 16 cores and about 256 GB per GPU reference, and write down why you differ.

GPU Smith helps teams run this kind of host sizing against their own traffic before a purchase order is signed.

Sources

1. DGX H100/H200 User Guide: Introduction, NVIDIA (accessed 2026-10-28)
2. DGX B200 User Guide: Introduction, NVIDIA (accessed 2026-10-28)
3. DGX B300 User Guide: Introduction, NVIDIA (accessed 2026-10-28)
4. Intel Xeon 6776P Processor specifications, Intel (accessed 2026-10-28)
5. NVIDIA GB200 NVL72, NVIDIA (accessed 2026-10-28)
6. NVIDIA Grace CPU Superchip, NVIDIA (accessed 2026-10-28)
7. NVIDIA H100 Tensor Core GPU, NVIDIA (accessed 2026-10-28)
8. Optimization and Tuning, vLLM documentation, v0.32.0 (accessed 2026-10-28)
9. KV Offloading Usage Guide, vLLM documentation, v0.32.0 (accessed 2026-10-28)
10. A guide on good usage of non_blocking and pin_memory() in PyTorch, PyTorch (accessed 2026-10-28)
11. Asynchronous Saving with Distributed Checkpoint (DCP), PyTorch (accessed 2026-10-28)
12. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, Rajbhandari et al., arXiv, 2019 (accessed 2026-10-28)
13. Qwen2.5-72B-Instruct config.json, Qwen on Hugging Face (accessed 2026-10-28)
14. Qwen2.5-72B-Instruct model.safetensors.index.json, Qwen on Hugging Face (accessed 2026-10-28)