TL;DR: The number is the least interesting part and the interviewer knows it. What they are testing is whether you can defend a performance claim, which means four things. A baseline that was warm, correctly configured and running the same work, because most large speedups turn out to be against something misconfigured. A method: how you timed it, how many runs, what you reported, and whether the variance was small enough for the difference to mean anything. A ceiling: what the theoretical limit was, so the interviewer knows whether 2x was most of what was available or a tenth of it. And the scope of what changed, meaning whether the win survived contact with production traffic or only held for the shape you measured. Lead with the mechanism rather than the number. "The kernel was memory-bound at 12% of peak bandwidth, and fusing the three elementwise passes removed two round trips to HBM" is a claim someone can check, and "we got a 3x speedup" is not.
How to approach it
Say what was slow and how you knew, which is usually a profile rather than a complaint. Then the mechanism you found, stated in terms of the resource that was saturated. Then the change. Then the measurement, including the baseline and how you made it fair. Then the number, and then immediately what it does not include. Ending on the caveat rather than the number is a deliberate move and it reads as confidence.
A strong answer
A typical situation: an engineer speeds up an inference path and reports 3.1x. The interviewer asks what the baseline was doing, and it turns out the baseline was running without a warmup, so the first iterations included compilation. The real number was 1.4x. The engineer is not dishonest and the number was still worth having, but the story now has to be rebuilt in the interview, which is the worst place to discover the problem.
What makes a claim survive, in the order to say it:
| Element | Weak | Strong |
|---|---|---|
| Diagnosis | "It was slow." | "Nsight showed 12% of peak DRAM bandwidth with high L2 traffic, so it was fetching bytes it did not use." |
| Mechanism | "I optimized the kernel." | "Three elementwise ops were separate kernels, each reading and writing the full tensor. Fusing them removed two HBM round trips." |
| Baseline | "Compared to before." | "Same input distribution, warmed up, clocks recorded, best-of-median over 50 runs, same dtype." |
| Number | "3x faster." | "1.9x on the fused path, median 2.4 ms to 1.26 ms, interquartile range under 3%." |
| Ceiling | Not mentioned. | "The roofline says this shape is memory-bound at about 295 FLOP per byte on H100, so the remaining headroom is the one unavoidable read." |
| Scope | Not mentioned. | "It holds for sequence lengths above 512. Below that the launch overhead dominates and the win is under 10%." |
Profiling with Nsight covers the diagnosis half and which counter answers which question. Roofline Model covers the ceiling, and having a ceiling in your answer is the single fastest way to signal that you optimized deliberately rather than by trying things. Kernel Fusion covers the specific mechanism in the example above, and the general point holds regardless of which mechanism your story uses.
The four ways a real win shrinks under questioning, all of which are better to raise yourself:
1. the baseline was not warm
compilation, autotuning, cache population and clock ramp all land in the first
iterations. A cold baseline against a warm candidate inflates everything
2. the inputs were cache-resident
a benchmark that reuses one tensor measures L2, not the memory system the production
path uses. Flushing between iterations changes the number, sometimes by a lot
3. the shape was favorable
a fast path that handles powers of two and falls back otherwise looks excellent on a
benchmark of powers of two. Report the production shape distribution
4. the win did not reach the user
a kernel 3x faster inside a step that is 8% kernel time moves the step by 5%. Say what
fraction of end-to-end time the thing you optimized occupied before you touched it
That fourth one is where most interview answers fall apart, and it is worth preparing explicitly. Amdahl's argument is simple arithmetic and interviewers apply it immediately: if the optimized part was 8% of the total and you made it 3x faster, the whole is 5.3% faster, which is a real result and a very different claim.
The reversal condition: a win that does not show up end to end can still be the right work, and there is an honest version of that story. Removing a bottleneck that was about to bind, cutting memory so a larger batch fits, or making a path predictable rather than faster are all legitimate, and each is defended differently. What fails is presenting a local speedup as an end-to-end one and being caught by arithmetic the interviewer does in their head.
What interviewers probe next
- "What was the baseline configured like?" The most common follow-up, and the one that ends weak stories. Know the flags, the warmup and the clocks.
- "How much of the end-to-end time was this?" Have the fraction. Then the end-to-end effect follows in one line.
- "What was the theoretical limit?" A roofline, a bandwidth number, or a serial fraction. Without one, you cannot say whether you finished.
- "Did it regress anything?" Memory, compile time, numerical accuracy or a different shape. Every real optimization trades something, and naming it is more credible than a clean win.
Common mistakes
- Leading with the number instead of the mechanism, which invites the interviewer to attack the number.
- A baseline that was not warmed up, which is the single most common source of inflated speedups.
- No ceiling, so there is no way to say whether the work was finished or barely started.
- Quoting a local speedup as an end-to-end one, which the interviewer converts in their head.
- Claiming no regressions, which is rarely true and is easy to test with one follow-up.
Key takeaways
- Mechanism first, number last, caveat after the number.
- A fair baseline is warm, correctly configured, same dtype, same input distribution, with variance reported.
- Know the fraction of end-to-end time you optimized; a 3x on 8% of the total is 5.3% overall.
- Bring a ceiling from a roofline or a bandwidth limit, so "how much is left" has an answer.
- Raise the four shrinkage causes yourself: cold baseline, cache-resident inputs, favorable shapes, and local versus end-to-end.