Advanced

torch.compile and CUDA Graphs

torch.compile captures Python into a graph with Dynamo, fuses it into Triton kernels with Inductor, and can wrap the result in a CUDA graph so a whole forward pass is one launch. CUDA graphs are what make batch-1 decode fast in every serving engine, and graph breaks, recompiles and static-shape rules are what make both bite in production. Interviewers ask when compile helps, when it hurts, and how you would know.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Kernel Fusion CUDA Programming Model Triton Programming Model Bandwidth-Bound Decode Throughput Serving Engines: vLLM, SGLang and TensorRT-LLM Profiling with Nsight

PRACTICE THIS IN REAL QUESTIONS

CUDA, Triton & Kernel EngineeringWhat does torch.compile actually do to your model, and when does it fail to help?→CUDA, Triton & Kernel EngineeringWhy do serving engines capture decode steps into CUDA graphs, and what does capture require of the rest of the code?→Open-Weights Models & Serving EnginesThe engine logs say CUDA graphs are disabled for your deployment. Does it matter?→CUDA, Triton & Kernel EngineeringWhy fuse kernels, how much does it save, and what can fusion not fix?→Open-Weights Models & Serving EnginesYour model decodes at a tenth of its bandwidth bound at batch one. Explain the gap.→Distributed Training & ParallelismFSDP or DeepSpeed ZeRO-3: which would you pick for a new training codebase today, and why?→