Advanced

Forward Pass, Backpropagation and Optimizer Steps

Follow one parameter through prediction, loss, gradient and update. Understand why backward does not itself train the weights, why training needs extra memory, and how microbatches differ from optimizer steps.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Tensors, Shapes and Dtypes Model Memory Footprint Training FLOPs: 6ND Data Parallelism and DDP Activation Checkpointing Pretraining, Fine-Tuning and Distillation

LESSONS THAT TEACH THIS

Training at ScaleA training step follows a critical path through compute, exchange and update18 min→Foundations of AI InfrastructureA request becomes tokens, and tokens become two different workloads30 min→

PRACTICE THIS IN REAL QUESTIONS

Coding for InfraImplement scalar autograd and prove that a shared input receives every gradient contribution→CUDA, Triton & Kernel EngineeringA training step runs at 20 percent model FLOPs utilization. Profile it and find where the missing time goes.→Distributed Training & ParallelismIn data-parallel training, what actually gets communicated between GPUs, and how much is it per step?→Napkin Math, Cost & CapacityEstimate how long it takes to write a checkpoint for a 405B training run→Kubernetes, Slurm & GPU SchedulingOne fleet: training that wants every idle GPU, and inference with a p99 SLO. Separate pools, or one pool with preemption? Show the numbers.→Napkin Math, Cost & CapacityHow long does an all-reduce of a 70B model's gradients take on eight GPUs?→