GPU server buying guide: H100 vs H200 vs L40S vs RTX PRO 6000 for AI

The common mistake is buying the flagship for a job that does not need it. Before you compare part numbers, be honest about what the machine will spend most of its life doing, because each workload has a different bottleneck.

Training a model from scratch is memory-bandwidth and interconnect bound, it wants the fastest HBM and the fattest GPU-to-GPU links you can get. Fine-tuning an existing model is lighter but still benefits from large VRAM so you can fit the model plus optimiser states. Inference, serving a trained model to users, cares about latency, throughput-per-dollar and VRAM to hold the weights, not raw training FLOPS. And graphics, CAD or visualization is a different axis again, where display features and driver certification matter.

Two mechanical details cut across the whole table. First, SXM vs PCIe: SXM modules sit in a dense baseboard with the highest power and full NVLink for the fastest GPU-to-GPU communication, while PCIe cards drop into a standard server slot and are easier to source and cool, at lower interconnect bandwidth. Second, NVLink is the high-speed fabric that lets multiple GPUs behave more like one big GPU, decisive for large-model training, largely irrelevant for single-GPU inference.

The chassis around the GPU sets your ceiling. There are three tiers worth knowing.

At the top, HGX and DGX 8-GPU platforms package eight SXM GPUs with NVLink/NVSwitch on one baseboard, this is the rack-scale building block for serious training, and it is what clusters are made of. For a desk, DGX Spark (the GB10 desktop system) puts a slice of that stack in front of a developer for local prototyping without booking cluster time. And for inference or lighter training, a single- or dual-socket server such as a Dell PowerEdge R760 or HPE ProLiant DL380 takes two to four PCIe GPUs, a pragmatic, easy-to-cool, easy-to-service box that covers most production serving.

Start sizing with VRAM: the model has to fit in memory first. A useful rule of thumb: model weights in FP16 need roughly the parameter count × 2 bytes, then add overhead for activations, KV cache and the framework. So a 70-billion-parameter model in FP16 is about 140GB just for weights, which is why it does not fit on a single 80GB H100 and either needs two GPUs, an H200/141GB, or quantization to a smaller precision.

Once it fits, GPU count is about throughput: more GPUs process more tokens or more training steps in parallel. And once you cross one node, networking becomes the bottleneck, multi-node training wants InfiniBand NDR or 400GbE so the GPUs are not starved waiting for gradients. Under-spec the fabric and expensive GPUs sit idle.

Not every GPU box has to be brand new. Refurbished and previous-generation parts, a well-tested A100 node, for example, can deliver most of the value for inference or budget training at a fraction of the price, and we test and warranty them the same way as new.

Two operational realities to plan for: power and cooling, and lead time. An 8-GPU training node is a serious electrical and thermal load, think in kilowatts per rack, and confirm your facility can feed and cool it before it arrives. Lead times swing with allocation: mainstream inference cards are usually quick, while the newest flagship SXM parts can carry a wait, so tell us your deployment date early and we plan the build around it.

Every GPU server we ship is factory CTO: you choose the GPUs, CPUs, memory, storage and networking, and we assemble and burn the machine in before it leaves. We sell EXW Hangzhou and ship every server by air. If you are importing for the first time, we help arrange a forwarder who delivers to your door in 1–2 weeks. Everything carries our 1-year warranty with cross-border advance-replacement RMA. If you are not sure which card fits your model and budget, send us the workload and we will spec it with you.

The specifications quoted above come from NVIDIA's own product documentation. Confirm memory, form factor and platform details against the primary pages before you commit to a card: