memory

Compare GPU memory, decode traffic and measured useful throughput. Work out a capacity ceiling without mistaking it for a performance forecast.

Track which state remains replicated at each ZeRO stage. Compute a declared 70B memory recipe, then add live gathers, activations and buffers before deciding fit.

Separate the 100B memory floor from the training deadline, then compare two 1,024-GPU layouts with their actual state ownership and communication schedules.

Derive the weighted gradient, test a short micro-batch, and separate activation savings from total memory and communication. Matching the optimizer step is only part of matching a larger batch.

Follow saved and recomputed values through backward. Compare complete update time and peak memory instead of promising a fixed memory saving or slowdown.

Follow a token into retained attention state, derive bytes per position for a 70B model, and distinguish present cache usage from the room needed to finish a response.

Trace one cache position through a block table, calculate the memory recovered from oversized reservations, and explain why sharing still needs careful writes and matching.

Draw query-head sharing, count actual stored state and compare payloads with explicit model geometry and binary units.

Follow adapter-specific token rows through a shared base, calculate stored adapter bytes, and separate the resident set from the active batch.

Find which startup step runs out of memory, count the allocations on that GPU, and change the setting that controls the shortage.

Compare actual encoded tensors, per-rank runtime fit, supported kernels and held-out task quality. Four-bit weight-only kernels can accelerate wider arithmetic, and smaller weights need not reduce the GPU count.

Confirm that requests are waiting for usable state capacity. Separate allocated memory, live blocks and evictable prefixes, then compare admission, lower-precision state, prefix sharing and hardware changes with explicit assumptions.

A context-window limit is not a latency or concurrency guarantee. Budget retained state, prefill and shared service time; distinguish cache reuse from eliminating work; choose isolation only when measured interference justifies its cost.

Apply the project’s offline tuning guidance in context, derive the memory tradeoff, and use workload-specific acceptance criteria instead of universal queue and headroom targets.

Group compatible adapters by pinned base, derive their size from target matrices and ranks, and benchmark mixed batches. Catalogue size, CPU residency, GPU slots and active adapters are separate limits; base sharing does not make fifty variants operationally free.

Compare raw weight sizes at 16, 8 and 4 bits, then budget KV cache and runtime memory for Llama 3.1 70B. Follow the worked per-GPU example.

Calculate Llama 3.1 70B KV cache memory at 128k context using eight KV heads. Worked BF16 and FP8 payloads, GB versus GiB, and per-GPU capacity checks.

Separate frozen weights from trainable state, count rank-16 adapters from matrix shapes, and check peak memory before choosing a fine-tuning layout.

Separate resident parameters, active work and the union of experts touched by a batch. Compare toy dense and MoE budgets without promising a fixed speedup.

Estimate activation memory from saved tensor shapes and lifetimes. Use a published reference formula carefully, then check checkpointing and parallel ownership against the real peak.

Derive adapter bytes from the target matrices, share compatible base weights, and keep each request's cache identity separate. A worked pool design avoids applying the mixed-adapter cost twice.

Follow a video job from prompt to downloadable artifact. Work the latent-shape arithmetic, compare batching with actual delivery times, and design bounded queues, cancellation and safe retries.