mediumNewBasetenModal

nvidia-smi shows 30 percent utilization on our serving fleet. Is that a problem, and what would you look at instead?

The number nvidia-smi calls utilization measures whether any kernel was running, not whether the chip was busy. A decode step that saturates HBM can read as thirty percent. The right dashboard has four other numbers on it.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Core

🩺 Fleet Reliability & ObservabilitySign in

DCGM and GPU TelemetryNVIDIA's Data Center GPU Manager reads a GPU's counters, runs its diagnostics and exports both to the monitoring stack, and nearly every fleet's dashboards and alerts are built on it. The skill is knowing which of its hundreds of fields carry signal: the profiling metrics that say whether the tensor cores are busy (not the utilization number everyone reads first), the error counters that predict a failure, the throttle reasons that explain a slow step, and the diagnostic levels that decide whether a node returns to the pool. This page walks those fields, derives an MFU estimate from them, and gives a fleet's alert thresholds.

Advanced

⚡ Kernels & Compilers🔒 Premium

Profiling with NsightNsight Systems answers where wall-clock time goes across CPU, kernels and copies; Nsight Compute answers why one kernel is slow, from hardware counters. The skill interviewers test is the order: timeline first, then the Speed of Light section, then the two or three metrics that name the bottleneck, so that a memory-bound kernel is recognized from its profile in under a minute and the fix is bytes, not occupancy.

Foundational

🚀 Inference & Serving

Prefill vs DecodeAn LLM request runs in two phases with opposite hardware profiles: prefill reads the whole prompt in one compute-bound pass and decides time to first token, decode emits one token per forward pass and is bound by memory bandwidth. Every serving decision, from batch size to which GPU to buy to whether to split the two phases across machines, follows from that split.

Foundational

🖧 Hardware & Cluster Build-Out

Cluster Bring-Up: Firmware, Drivers and the StackBring-up is an ordered dependency chain and skipping a step produces a symptom that points somewhere else. Firmware first, then the operating system and kernel, then the GPU driver, then the fabric manager, then the network stack, then GPUDirect, then CUDA and NCCL, then the container and scheduling layer. On Blackwell HGX systems the fabric manager reaches the NVSwitches through a bridge device and therefore depends on the InfiniBand stack being present, which is a dependency that surprises almost everyone the first time.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWhat changes when you serve a vision-language model instead of a text-only one?Next in this trackWhen does offloading the KV cache to CPU memory or NVMe beat recomputing it?Next in this trackA customer sends the same prompt twice at temperature zero and gets different answers. Explain why, and what you can promise them.

FEDITOR'S NOTE

Scored on explaining what the utilization counter actually measures, on the bandwidth argument for why decode looks idle, and on naming the DCGM fields and engine metrics that say whether the fleet is actually saturated.

DISCUSSION · 0

No comments yet — be the first to share your approach.