NVIDIA H200 SXM
The H200 SXM is the memory-upgraded revision of Hopper: the same GH100 compute die as the H100, but with 141 GB of HBM3e across six memory stacks instead of 80 GB across five, and 4.8 TB/s of bandwidth instead of 3.35 TB/s. Compute throughput per GPU is identical to H100, so the H200 exists to fix the memory ceiling, not to add FLOPS: bigger KV caches, larger single-GPU model shards, and fewer GPUs needed to hold a given model. NVIDIA markets it as drop-in compatible with H100-qualified systems at the same 700 W envelope, which is why it is the common upgrade path for operators with an existing Hopper fleet who are not ready to requalify for Blackwell. It sits above H100 and below Blackwell (B200/B300) in NVIDIA's current lineup, positioned as the memory-bound-workload answer within Hopper rather than a new architecture.
- Compute per GPU is identical to H100: the entire case for H200 is memory capacity and bandwidth. If your bottleneck is FLOPS, not memory, H200 will not move your throughput.
- 141 GB per GPU changes sharding math for large models: workloads that needed 2 H100s per shard for memory reasons often fit on 1 H200, which can net out cheaper per unit of served capacity despite the higher unit price.
- The bandwidth jump (4.8 TB/s vs 3.35 TB/s) matters most for memory-bound inference (large batch, long context, high KV-cache pressure) and less for compute-bound training steps; benchmark your actual workload before assuming H200 is worth the premium.
- 'Drop-in compatible' is a vendor claim about power and socket, not a guarantee: verify your specific HGX baseboard and cooling loop are qualified for HBM3e's tighter PCB routing tolerances before assuming a straight swap.
- Given the tighter and more volatile lead times reported for H200 versus H100, lock allocation early if the design depends on 141 GB per GPU; do not assume H200 supply behaves like the now-easing H100 market.
How much does an H200 cost?
Reported street prices run roughly $30,000 to $45,000 per GPU, with an 8-GPU HGX H200 server commonly quoted in the $300,000 to $500,000 range. Prices vary significantly by channel, volume, and how tight HBM3e supply is at the time of quote.
H100 vs H200 for inference?
H200 wins for inference workloads with large KV caches, long context windows, or big batch sizes, because the extra 61 GB and 43 percent more bandwidth reduce the number of GPUs needed per model instance. For short-context, compute-bound inference, the two parts perform the same and H100 is usually the cheaper choice.
Is H200 a new architecture or just more memory on H100?
It is the same GH100 Hopper die and the same compute throughput as H100; the change is entirely in the memory subsystem, 141 GB of HBM3e at 4.8 TB/s instead of 80 GB of HBM3 at 3.35 TB/s.
Can I upgrade an existing H100 HGX server to H200?
NVIDIA markets H200 as drop-in compatible with H100-qualified systems at the same 700 W envelope, but this needs to be verified against your specific baseboard and cooling loop, since HBM3e imposes tighter PCB substrate routing requirements than HBM3.