hardNew

Half the weights are zero. Why did serving not get faster?

Separate valid 2:4 structure, packed representation and sparse dispatch from model quality and the fraction of request time that can improve.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Separate valid 2:4 structure, packed representation and sparse dispatch from model quality and the fraction of request time that can improve.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

Structured Weight SparsityZeros accelerate inference only when a supported representation and kernel skip their work. Separate pruning quality, compressed storage, dispatch and end-to-end benefit.

Advanced

🚀 Inference & Serving🔒 Premium

Reasoning Budgets and Answer LatencyReasoning tokens consume generation time and retained state before an answer becomes usable. Separate the first generated token from the first answer token, reserve output capacity, and evaluate reasoning limits against task success.

Advanced

📐 AI Systems Design🔒 Premium

Model Routing and CascadesChoose a model against measured task quality, latency and cost. Distinguish routing before generation from cascades that pay for a first answer before escalating.

Advanced

⚡ Kernels & Compilers🔒 Premium

GEMM Work Partitioning and Wave QuantizationOutput tiles determine how much independent matrix work is available. Split-K and work-centric schedules can reduce an underfilled final wave, while adding partial-result reduction and workspace costs.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Closest to this topicIf you could change one thing about GPU architecture for LLMs, what would it be, and what argues against it?Closest to this topicFirst-token latency is fast, but the reasoning model's answer takes seconds. What do you check?Closest to this topicAn Nsight summary reads DRAM 90%, SM 30%, occupancy 45%, L2 hit rate 20%. What is the bottleneck?

DISCUSSION · 0

No comments yet — be the first to share your approach.