Together AI Napkin Math, Cost & Capacity interview questions

Why decode speed is a division of bandwidth by weight bytes, why the answer is about 24 tokens a second regardless of how fast the tensor cores are, and the two ways to double it.

Fleet dollars per second over tokens per second, times a million: a worked $2.78, the utilization term that turns it into $6.94, and why batch is the lever that moves it by 50x.

Two constraints per replica, memory and throughput, each derived from the model and the traffic; whichever binds sets the users per replica, and the replica count follows with headroom. The full chain from 1,000 users to a number of nodes.

fp8 halves the bytes per decode step and doubles the compute peak, but the all-reduces and launch overhead do not shrink, so the measured gain is 1.5x at a fixed batch and about 2x once the freed memory raises the batch. The chain, with the step-time table.

For bf16 weights the decode intensity equals the batch, so the ridge is at batch 295 on an H100. Then the KV cache enters and the answer changes shape: the projections go compute-bound at 300 while the attention reads never do, and at 8k context the step as a whole stays memory-bound at any batch.

Decode is bandwidth-bound, so cost per token tracks dollars per terabyte per second, and on spec the MI300X wins by 2x. The table, the memory-capacity term that changes the replica shape, and the software-efficiency discount that decides whether the spec ranking survives a benchmark.

Two bounds, bandwidth and compute, and five questions about what the number counts. For a 70B on an H100 the claim exceeds the bf16 compute peak and is impossible; for an 8B at high batch it is routine; for input tokens it is easy. The chain that tells the cases apart.

The first number every serving interview asks for, derived from parameter count and bytes per parameter, in three precisions, with the part people forget to add.

Four numbers from the config file, one formula, and a per-sequence result that is a third of the model's own weights. Plus the mistake that makes the answer eight times too big.

The 6ND rule, where the 6 comes from, and the 6.3e24 that every training-time and GPU-count question in the loop is built on.

The fleet equation applied to 6.35e24 FLOPs: 11 days, and how the answer swings from 9 to 15 with the one parameter the interviewer wants you to state.

The fleet equation solved for GPU count: about 6,200 H100s at 40% MFU, why it rounds up to a power of two, and the reasons a good answer adds 15% before naming a number.

Sixteen bytes per parameter is 1.1 TB of static state before a single activation; LoRA keeps the 141 GB of frozen weights and shrinks the rest to a few gigabytes. The derivation, the GPU counts, and the activations that both still pay.

Two divisions decide it: the kernel's FLOPs per byte against the card's ridge point. The worked case, the profiler counters that give the inputs, and what to change once you know which wall you hit.

Gradient bytes per step over step time, with the ring all-reduce factor that nearly doubles it: the arithmetic for a 7B and a 70B, the link speeds that cover it, and the step-time threshold where overlap stops hiding it.

Prefill is the compute-bound half of serving: 2 × N × tokens FLOPs over the effective TFLOPS of the replica. The chain that gives a 0.24 s floor on a node, the single-card version, and what it means for time to first token.

6ND on 15.6 trillion tokens gives 3.8e25 FLOPs; converting to GPU-hours needs an MFU, and Meta's published 30.8 million hours lets you solve for it. The chain, the check, and what the implied MFU tells you.

Memory follows total parameters and compute follows active ones, so the two models pull the GPU count in opposite directions. The chain for a 235B-total, 22B-active MoE against a dense 70B, for serving and then for training.

Sixteen bytes per parameter is 6.5 TB per checkpoint; over a 100 GB/s filesystem that is about a minute, which at a 30-minute cadence is 3% of the run. The chain, the storage tiers that meet it, and why the write is made asynchronous.

700 W per GPU is the number everyone knows and it is only 40% of the answer. The chain from GPU TDP to node, to IT load, to facility power with PUE: about 1.8 MW, and what that means in racks and dollars.

Capex plus two years of power and operations, minus what the cards are worth at the end, against a rental rate at a utilization. The chain that produces a break-even utilization, and the two inputs that swing it from 60% to over 100%.

The ring all-reduce formula, 2(n−1)/n times the buffer over the link bandwidth, applied to 141 GB of bf16 gradients on an NVLink node: 0.27 s by the formula, and the two reasons the measured number is different.