Inference Engineer

Your route into inference engineering

Start with basic Python, HTTP requests and command-line tools. You can learn the memory arithmetic and run the CPU exercises without a GPU. Device performance claims require actual hardware runs; each hands-on resource states its execution status.

If model serving is new to you, begin with From Model to Service. Otherwise, use these checkpoints to find your starting point. Work through the linked course modules in their published order; the checkpoints connect the parts of the curriculum needed for the role. Continue when you can produce the stated evidence, rather than after a fixed number of weeks.

Free resources need no account. Free account resources require sign-in. Premium resources require paid access.

1. 01

   Follow a request and define the clocks

   Learn how prompt processing differs from token generation. Distinguish the wait for first content, later token gaps and the time until a complete answer.

   Before moving on: Draw one request timeline and explain where queueing belongs. Include a failed request when calculating service success.

   • Begin with the request lessonFree
   • Prefill and decode referenceFree
   • Practice defining latency and goodputFree
2. 02

   Calculate memory before choosing concurrency

   Account for weights, runtime allocations and growing request state on each device. Learn why a model that loads can still run out of room while serving.

   Before moving on: Produce a per-device memory budget and show which device limits admission. Recalculate it when the output allowance doubles.

   • Build the concurrency budgetFree account
   • KV cache sizing referenceFree
   • Defend a concurrency estimateFree
3. 03

   Choose an engine and establish a baseline

   Separate execution engines from gateways and fleet controllers. Compare supported configurations under the same task, traffic and quality requirements.

   Before moving on: Write an engine shortlist and a reproducible workload manifest. Record unsuccessful requests alongside latency and throughput.

   • Design the benchmarkFree
   • October engine comparison and responsibilitiesPremium
   • Practice selecting a serving engineFree account
4. 04

   Control cache growth and scheduled work

   Work through paging, shared prefixes, eviction, continuous batching and chunked prefill. Tie each change to the resource or waiting time it should improve.

   Before moving on: Draw shared and private cache blocks, then propose an admission policy for mixed short and long requests. State when cache locality should lose to queue delay.

   • Work through prefix reuseFree account
   • Chunked prefill referencePremium
   • Choose a replica under competing constraintsFree account
5. 05

   Evaluate precision, speculation and distributed execution

   Test lower-precision formats and speculative decoding against task quality and complete execution cost. Then study tensor parallelism, expert communication and prefill/decode separation.

   Before moving on: Keep or reject one optimization using a controlled comparison. For a multi-device design, account for both retained state and transferred bytes.

   • Separate weight, activation and cache precisionPremium
   • Evaluate a complete speculative roundPremium
   • Extend the design to expert parallelismPremium
6. 06

   Serve reasoning and tool outputs correctly

   Measure the wait for useful answer content and leave room for it in the generation budget. Follow streamed tool arguments through completion, validation and authorization.

   Before moving on: Build an answer-latency budget and run the CPU output-contract exercise. Reject a truncated call even when its buffered JSON parses.

   • Start the reasoning and output-contract moduleFree account
   • Reasoning budget referencePremium
   • Diagnose a slow answer behind a fast first tokenPremium
7. 07

   Operate the service under load and failure

   Connect queue limits, request routing, cold starts, scaling and cancellation. Learn which metrics expose capacity loss and how state handoffs fail.

   Before moving on: Write a runbook for overload and one worker failure. Preserve tenant boundaries and account for replicas that are starting but cannot yet serve.

   • Prepare the SLO review and runbookPremium
   • Inference autoscaling referencePremium
   • Review the deployment lifecyclePremium
8. 08

   Defend a measured deployment

   Join the compatibility record, workload, raw outcomes and cost calculation into an engineering decision another person can reproduce.

   Before moving on: Deliver a serving comparison with quality and latency gates, an explanation for one measured difference, and the condition that would change your choice.

   • Build the reproducible serving teardownPremium
   • Attach the output compatibility recordPremium
   • Check the cost denominatorPremium

For an example of how performance work has been assessed, read Anthropic's account of its performance engineering take-home. It describes optimizing code for a simulated accelerator and explains why the assessment changed. Treat it as practice material; it does not establish the current interview process for an inference role.

Research checked October 16, 2026. The Fireworks careers page and Anthropic careers page list roles across inference, performance and infrastructure. These roles help motivate the skills here. The practice questions are editorial exercises, not claims about a company's current interview rounds.