TL;DR: GPUs suit workloads with enough parallel work, regular data access and reuse to feed their execution units. CPUs often handle short, irregular or control-heavy work efficiently. Compare the same operation and numerical requirements, including movement and launch costs. A GPU kernel can be faster while the complete request is slower.
How to approach it
Start with the workload's structure: independent work, dependency chains and memory traffic. Explain why matrix multiplication offers reuse. Then include transfers between CPU and GPU and show how keeping data on the GPU spreads that cost across several operations. Use the complete request time to decide where the work belongs.
A strong answer
A small service moves a tensor to a GPU for one operation and brings the result back. The device kernel is quick, yet latency rises. The missing part of the comparison is everything around that kernel.
CPU and GPU designs expose different ways to use parallelism. A CPU can devote substantial resources to making a small number of instruction streams progress through dependencies and branches. A GPU keeps many threads resident and can issue eligible work while other work waits. Modern CPUs also have vector and matrix facilities; GPUs also execute control flow. “One is smart and the other just has cores” does not explain a performance result.
Matrix multiplication is favorable because many output elements can be computed in parallel and input tiles can be reused. An M×K input times K×N weights needs approximately 2MKN floating-point operations under the multiply-add convention. Tensor Cores and Matrix Units provide specialized matrix paths, but supported dtype, layout and numerical behavior matter. Compare like with like rather than dividing a reduced-precision GPU peak by a CPU scalar peak.
For example, a 256×256 product with K=256 has 2×256³ = 33,554,432 FLOPs. Counting one read of each BF16 input and one BF16 output write gives 3×256²×2 = 393,216 bytes, or about 85.3 FLOPs/byte. That is a deliberately simple boundary model: additional reads, cached inputs or wider outputs change it. The Roofline Model explains why a compute peak alone is insufficient.
NVIDIA's matrix performance guide relates reuse, problem dimensions and hardware limits. A large matrix has more opportunities than a single dependent scalar operation; it still needs a suitable implementation.
The pictured times are hypothetical, not measurements of named processors. Assume the same acceptable numerical result, two milliseconds per CPU operation, 0.2 milliseconds per GPU operation, and three milliseconds of total movement paid once for a resident chain. Include launch costs in those assumed operation times. Then:
CPU time for k operations = 2k ms
GPU time with reuse = 3 + 0.2k ms
GPU is faster when 3 < 1.8k, so integer k must be at least 2.
k=1: CPU 2 ms; GPU 3.2 ms
k=10: CPU 20 ms; GPU 5 ms
sanity: moving data for every operation instead costs 3.2k ms.
The mechanism is amortization: keeping intermediate data on the GPU avoids repeatedly paying the same boundary cost. The reversal is clear too. If results must return to the CPU after every operation, the assumed benefit disappears. Real transfers may overlap with computation, so trace the dependency path instead of blindly adding every duration.
| Workload property | Placement experiment |
|---|---|
| Large regular matrix work | Compare supported accelerator kernels at matching accuracy |
| One tiny operation with host data | Measure transfer and launch overhead before moving it |
| Long dependent chain | Inspect available parallelism; more lanes may not help |
| Data already on the GPU | Include the cost of moving it away in the CPU alternative |
Memory access is another part of the explanation. Adjacent accesses from warp lanes can use memory transactions efficiently; scattered accesses may fetch many unused bytes. Reusing an on-chip tile reduces traffic across the HBM boundary. Neither property makes every deep-learning operation compute-bound. Decode at low concurrency can repeatedly read weights with little reuse, while a wider batch can change that balance.
For the placement decision, record end-to-end latency, throughput, batch/shape distribution and numerical error against the chosen reference. Use Nsight Systems to locate transfers, host gaps and kernels; CPU utilization alone does not establish the bottleneck. Compare a steady-state resident path and a cold request separately. If preprocessing or launch gaps dominate, replacing the GPU with a faster SKU may leave most of the delay intact.
What interviewers probe next
- “Why not simply add CPU cores?” It can help parallel CPU work, but memory channels, interconnects and the operation's scaling determine the result. Count total system cost and power for the measured workload.
- “Are reductions and sampling always CPU work?” No. Data location and available implementations matter. Sending a large resident tensor to the host can cost more than a GPU reduction.
- “Does high GPU activity prove efficiency?” No. A device can stay active moving bytes or doing redundant work; measure useful progress.
- “Does quantization change this?” It changes payload and supported arithmetic, and may change quality. Re-run the comparison with the same acceptance criteria.
Common mistakes
- Quoting a universal CPU/GPU speedup without matching work or precision.
- Timing only the kernel when the user waits for transfers too.
- Assuming every matrix multiply reaches compute peak.
- Moving work between devices without tracking its intermediate data.
Key takeaways
- Parallelism, reuse and memory access explain the opportunity; measured end-to-end work decides placement.
- In the synthetic example, 10 resident GPU operations take 5 milliseconds versus 20 on CPU.
- Paying the three-millisecond boundary cost on every operation reverses that result.