← Open-Weights Models & Serving Engines / 26

hardNewFireworks AITogether AIBaseten

You upgraded the serving engine and throughput fell 15 percent. Find the cause.

Engine upgrades change defaults, kernel selection and scheduling at once, so attribution is the whole problem. The bisection that works on a version rather than a commit, the four categories of cause, and the pre-upgrade baseline without which none of this is possible.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

Engine upgrades change defaults, kernel selection and scheduling at once, so attribution is the whole problem. The bisection that works on a version rather than a commit, the four categories of cause, and the pre-upgrade baseline without which none of this is possible.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

🧮 Open Weights & Serving Engines

Serving Benchmarks That Do Not LieMost published serving numbers are not comparable to each other and not predictive of production, because they differ in the input distribution, the concurrency, whether the cache was warm, and which of several very different metrics is being reported. A benchmark that supports a decision has to fix all four, report a distribution rather than a mean, and be run against the traffic shape you actually serve. The single most useful discipline is to compute the bandwidth bound first, so you know what fraction of the possible you achieved.

⚡ Kernels & Compilers🔒 Premium

FlashAttention InternalsStandard attention writes the N x N score matrix to HBM and reads it back, which makes it memory-bound and quadratic in memory. FlashAttention tiles Q, K and V through shared memory, keeps a running max and sum so the softmax never needs the full row, and recomputes scores in the backward pass. Knowing the online-softmax rescale, why FlashAttention-2 flipped the loop order, and what FlashAttention-3 overlaps on Hopper is the difference between naming the paper and being able to write the kernel.

⚡ Kernels & Compilers

CUDA Programming ModelCUDA splits a program into a host that allocates, copies and enqueues work, and a device that runs thousands of identical threads organized as a grid of blocks. Getting the split right, and knowing that a launch returns before the kernel runs, decides whether your first live-coding kernel produces a correct number or a silent zero.

⚡ Kernels & CompilersSign in

Triton Programming ModelTriton replaces CUDA's thread with a program that owns a whole block of data, and replaces manual shared-memory staging and coalescing with a compiler that derives them from block shapes. Pointer arithmetic on vectors, masks for the tail, and program-id swizzling for L2 reuse are the three idioms every Triton kernel is built from, and the live exercise at Anthropic, OpenAI and the serving startups is usually one of a fused softmax, a LayerNorm or a matmul in exactly this style.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackA model interleaves linear-attention and full-attention layers. What changes about serving it?Next in this trackYou serve a small, a medium and a large open-weights model. How do you route requests between them?Next in this trackYou need a released bf16 model at half the footprint. Walk through quantizing it yourself.

FEDITOR'S NOTE

Scored on needing a pre-upgrade baseline, on defaults changing as the most common cause, and on bisecting by configuration rather than by code.

DISCUSSION · 0

No comments yet — be the first to share your approach.