TL;DR: Server Edition is a candidate when the supported fine-tuning jobs fit on one card or MIG instance and its measured cost per completed job is attractive. SXM offers a different memory-bandwidth and scale-up system, and H100 also supports MIG. Neither “under 30B” nor “needs two cards” settles the choice. Account for the training method, sequence length, activations, software support and communication pattern before benchmarking the actual job mix.
How to approach it
Define the product's supported jobs in bytes and runtime requirements. Separate adapter training from full-parameter training, calculate a memory budget, and test whole-card and partitioned configurations. Then compare completion time, queueing, isolation and total cost.
A strong answer
Suppose customers mostly run low-rank adaptation on models up to 30B, with occasional full-parameter runs on smaller models. That describes two different memory and communication workloads. The large frozen base in an adapter job need not carry optimizer state, while a full-parameter job updates the base itself.
The RTX PRO 6000 Blackwell Server Edition provides 96 GB GDDR7 and a configurable power limit up to 600 W. Compare it with the exact H100 variant: the H100 SXM specification is an 80 GB HBM device with NVLink. A PCIe server card and an SXM system are different deployment configurations, so include chassis, cooling, interconnect and support in the comparison. Accelerator Selection: H100 to and RTX PRO 6000 explains those gates.
Calculate the job before assigning a card
An illustrative 30B adapter-training budget is:
Frozen BF16 base: 30 billion × 2 bytes = 60.0 GB
Trainable adapters: assume 0.5% of base = 150 million parameters
Adapter training state: assume 16 bytes/parameter = 2.4 GB
Activations and workspace: assume 15.0 GB
Total: 77.4 GB, before any omitted framework or allocation overhead
Nominal remaining capacity:
96 GB card: 18.6 GB
80 GB card: 2.6 GB
sanity: 77.4 is below both nominal capacities. It suggests more headroom on 96 GB,
not proof that the job fails on 80 GB or succeeds on 96 GB.
The adapter fraction depends on rank and targeted layers. Activations depend on sequence length, batch size, checkpointing and kernels. Measure the peak through forward, backward and optimizer steps; loading the weights is not a fit test. Quantization or checkpointing can change the budget, with quality, support and runtime tradeoffs.
For full-parameter training, a common illustrative mixed-precision state budget is 16 bytes per parameter: two-byte weights, two-byte gradients, four-byte master weights and eight bytes for two optimizer moments. Other implementations differ. Model Memory Footprint explains the accounting.
| Model | State at 16 bytes/parameter | Minimum 96 GB cards from state alone | Minimum 80 GB cards from state alone |
|---|---|---|---|
| 8B | 128 GB | 2 | 2 |
| 13B | 208 GB | 3 | 3 |
| 30B | 480 GB | 5 | 6 |
These are lower bounds assuming the chosen strategy shards the state. Activations and buffers add memory, and data parallelism that replicates the full state does not obtain these bounds. Five 96 GB cards holding 480 GB would leave no room for anything else. There is no universal 13B transition: even the 8B example exceeds one card under these assumptions.
Illustrative 30B model; method and sequence settings determine the allocation.
- Frozen base with trainable adapters
60 GB base + 2.4 GB adapter state + assumed 15 GB activations/workspace = 77.4 GB.
- Full-parameter training
At an assumed 16 bytes per parameter, state alone is 480 GB before activations.
- Map the actual job to hardware
Measure peak memory and completion time on the whole card or intended MIG profile.
Partitioning changes the service unit
The MIG profile guide lists up to four 1g.24gb instances for RTX PRO 6000 Blackwell. Each has a fraction of the card's memory and compute. Four instances do not each receive 96 GB. H100 has its own supported MIG profiles and must be included in the same scheduling comparison.
A 7B adapter job may fit in 24 GB for a particular sequence length and batch, but the parameter count alone does not prove it. Test peak memory and completion time inside the intended profile. Four simultaneous jobs can improve packing without multiplying useful throughput by four: they divide resources, and shorter whole-card runs may sometimes complete more work.
MIG, MPS and Time-Slicing distinguishes resource partitioning from other sharing mechanisms. Hardware partitions provide important isolation, but tenant security still includes the host, driver, storage, credentials and orchestration. GPUs on the same physical card also retain shared failure dependencies.
Measure communication without inventing a slowdown ratio
RTX PRO 6000 has no NVLink scale-up fabric. Multi-card communication therefore depends on the server's PCIe topology, supported peer access and software path. That can be a substantial constraint for tightly coupled work, but it is not a prohibition on multi-GPU training.
Do not divide H100's advertised 900 GB/s bidirectional aggregate by roughly 63 GB/s of one-way Gen5 x16 capacity and call the answer a 14× application penalty. The directions, topology, payload efficiency and collective algorithm differ. Many small tensor-parallel collectives can be latency-sensitive; a sharded training job has a different traffic pattern from data-parallel gradient reduction. Adapter training may communicate only the trainable state under a suitable strategy.
Run collective tests over the relevant message-size range, confirm the actual peer path, and then measure the full training job. Include communication overlap and end-to-end completion time. No hardware benchmark is claimed here.
Choose a supported job mix
If single-card or partitioned adapter jobs dominate, fit with margin, and finish more cheaply on RTX PRO 6000 at the required queueing and completion targets, choose that pool. Reverse the choice if the supported jobs need a scale-up path, H100 finishes them more economically, or a required kernel or feature is unavailable on the candidate stack.
A mixed fleet may help when both job classes have enough demand. It also adds scheduling, capacity and support work. Compare that cost with a simpler fleet; buying a few NVLink nodes is not automatically a cheap hedge. Base future capacity on a concrete product forecast and an upgrade path rather than a fixed eighteen-month rule.
What interviewers probe next
- “Does a job fit because its weights fit?” No. Include trainable state, activations, buffers and peak allocation.
- “Does MIG guarantee four times the throughput?” No. It divides resources; measure packing and job completion at the service level.
- “Does a two-card job force H100?” No. The communication pattern and measured total cost decide whether the PCIe design is adequate.
Common mistakes
- Saying 77.4 GB cannot fit in 80 GB without examining actual overhead.
- Forgetting H100's own MIG support.
- Using a decode communication estimate as a full-training benchmark.
- Treating lower GPU power as proof that a server is easier or cheaper to deploy.
Key takeaways
- The example 30B adapter budget is 77.4 GB; the nominal headroom is 18.6 GB versus 2.6 GB.
- At the illustrative 16-byte state budget, even an 8B full-parameter job needs more than one card before activations.
- Choose by supported jobs completed at the required service level and total cost.