Trace a request before choosing a platform

TL;DR: Draw the path from a user request to the model process, then attach evidence to each boundary. Start with one simple service whose behavior you can observe. A remote timeout is insufficient evidence to change its accelerator or serving engine.

Where you are. You have seen what a model and GPU do in Start here. Now turn that picture into a service boundary map, without writing application code.

Begin with the request that fails

Our teaching scenario is a small documentation assistant. It answers a test request on its host but times out from another machine. The host is a virtual machine, or VM: a provisioned computer with CPU, memory, networking and storage resources. A model process runs inside it. Later it may move into a container or onto an orchestrated platform; the request still has to reach a listening process.

Write down the requested hostname, destination port, API path and caller identity. A name is resolved to an address. A route carries packets toward that address. Firewall policy decides whether traffic is permitted. The process must listen on an interface that the caller can reach. A loopback-only listener explains local success without implying a broken model.

The diagram separates reachability from permission and readiness. A valid encrypted connection can end in an authorization denial. A running process can be reachable while it is still loading weights. You need evidence for the specific request, not just a green machine-health indicator.

Build a small boundary table

For an illustrative trace, setup takes 40 ms, gateway handling 10 ms, engine waiting 50 ms, prompt processing 200 ms and first-token return 15 ms. If these are non-overlapping intervals, the first token reaches the caller after 315 ms. The engine contributes 250 ms of that interval. Label both boundaries before comparing dashboards.

If the client reports a timeout before any gateway request ID exists, inspect the earlier path first. If the gateway accepted the request and its upstream connection failed, inspect the gateway-to-worker path. Avoid restarting every worker merely because one boundary is uncertain; that can replace an observable failure with a cold-start backlog.

Decide what to operate first

Choose a single VM or a managed endpoint for the first controlled exercise. Record who owns patching, process recovery, logs, storage permissions and capacity limits. A managed service can take responsibility for some of those tasks, but you still own the workload, data permissions and evidence of useful output.

A container packages application dependencies and starts processes with isolation mechanisms. It is not a promise that a model is loaded or that accelerator software is compatible. A cluster scheduler adds placement and reconciliation. Those become useful when the service needs their capabilities; they do not replace the boundary table.

SSH belongs to the administrative path, which should have its own access policy. Do not expose it broadly to solve a model API problem. Similarly, a successful local notebook run demonstrates one execution context. Preserve its model identity and input before asking a remote service to reproduce it.

Do this before moving on

Produce a five-row boundary table for the toy service. Attach the 315 ms trace and mark which evidence would distinguish a bad hostname, an authorization denial and an engine queue. Then remove setup cost through connection reuse: the new illustrative TTFT is 275 ms. Explain why the engine's processing time did not change.

Your deliverable is a map and a diagnosis order. It is not proof that a real endpoint achieves these timings. If you have a service available, collect one real trace and replace every assumed interval with observed evidence from the matching boundary.

Explain the diagnosis aloud

An interviewer says, “It works on the VM, but users time out. Would you add GPUs?” Start with the difference between the working and failing paths. Ask whether the caller resolved the name, connected and reached the gateway. Only after that boundary is established should you use queue and execution measurements to consider serving capacity. If both paths reach the same worker and its queue grows under load, the evidence now supports a capacity investigation.

Go deeper

• Request Networking: DNS, TCP and TLS explains the connection boundaries behind this diagnosis order.
• Inference Platform Architecture connects this small request path to larger serving pools.
• Latency metrics separates first-token timing from throughput and useful outcomes.

Key takeaways

• Local success leaves remote reachability and authorization untested.
• The worked first-token trace totals 315 ms across distinct boundaries.
• Select platform capabilities after identifying the responsibilities you need.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 1The model responds locally, but the remote caller cannot resolve its hostname. What do you inspect first?

   Name resolution from the callerGPU memory capacityThe quantization format

2. 2What remains of the 315 ms trace if 40 ms of setup is avoided?

   Answer it, then reveal