Trace a request before choosing a platform

TL;DR: Follow a failing request from the caller to the model, checking how far it gets. A timeout can happen before the request reaches the GPU, so establish where it stops before changing the hardware or serving engine.

Where you are. In Start here, you saw what a model and GPU do. Now follow the network path that lets someone on another machine use that model.

Begin with the request that fails

Imagine a small documentation assistant that answers questions on the machine where it runs. You send the same request from your laptop, but this time it times out. The model has already worked once. What changed is the path the request must take to reach it.

In this example, the assistant runs on a virtual machine, or VM: a computer allocated to you with its own CPU, memory, networking and storage resources. The model runs as a process inside that machine. Even if you later package it in a container or move it to a cluster, callers still need a network connection to the process accepting their requests.

Start by recording the hostname, port and API path used by the failing caller, along with the identity it uses to sign in. The hostname must resolve, or translate, to the intended address, and the network must carry traffic to that address. A firewall may block that traffic even when the address is correct.

The service must also accept connections on an interface reachable from outside the VM. If it listens only on the loopback address, it accepts connections only from its own machine. That would explain why the local test succeeds while your laptop cannot connect.

Each step in the diagram answers a different question. Can the caller connect? Is it allowed to use the service? Is the worker ready to run the model? An encrypted connection can succeed even when permission is denied, and a reachable worker may still be loading model weights. A green machine-health indicator cannot tell you which of these happened to your request.

Build a small boundary table

In this example, the gateway is the component that checks the caller's identity, admits the request and selects a worker to run it. Each row below is a boundary where you can save evidence about one request.

Consider a made-up trace in which connection setup takes 40 ms, gateway handling takes 10 ms, the request waits 50 ms inside the engine, prompt processing takes 200 ms, and returning the first token takes 15 ms. Assuming these intervals do not overlap, the caller waits 315 ms for the first token. This is its time to first token, or TTFT.

The engine accounts for only 250 ms of that wait: 50 ms in its queue plus 200 ms of processing. A client dashboard showing 315 ms and an engine dashboard showing 250 ms can therefore both be correct. Check where each timer starts and stops before comparing them.

Use the request ID to connect the caller's result to the gateway and worker logs. If the caller times out and the gateway has no record of the request, investigate the path to the gateway first. If the gateway accepted it but could not connect to a worker, you have narrowed the search to that connection. Restarting every worker at this point may remove useful evidence and leave incoming requests waiting for models to load again.

Decide what to operate first

Choose a single VM or a managed endpoint for the first controlled exercise. Record who owns patching, process recovery, logs, storage permissions and capacity limits. A managed service can take responsibility for some of those tasks, but you still own the workload, data permissions and evidence of useful output.

A container packages the application and its dependencies, then runs its processes with some isolation from the host. You still have to check whether the accelerator software is compatible and the model has finished loading. A cluster scheduler can choose where workers run and replace them when they fail. Add those capabilities when you need them; the request-path checks remain useful on either platform.

SSH belongs to the administrative path, which should have its own access policy. Do not expose it broadly to solve a model API problem. A successful run in a local notebook has the same limit: it demonstrates one execution context. Record which model it loaded and the input it used before asking a remote service to reproduce the result.

Do this before moving on

Produce a five-row boundary table for the toy service. Attach the 315 ms trace and mark which evidence would distinguish a bad hostname, an authorization denial and an engine queue. Then assume connection reuse removes the setup cost: the new illustrative TTFT is 275 ms. Explain why the engine's processing time did not change.

You should now have a map of the request path and an order for investigating failures. The timings above are exercise inputs. If you have a running service, collect a real trace and replace them with measurements taken at the same points.

Explain the diagnosis aloud

An interviewer says, “It works on the VM, but users time out. Would you add GPUs?” Start with the difference between the working and failing paths. Ask whether the caller resolved the name, connected and reached the gateway. Only after that boundary is established should you use queue and execution measurements to consider serving capacity. If both paths reach the same worker and its queue grows under load, the evidence now supports a capacity investigation.

Go deeper

• Request Networking: DNS, TCP and TLS explains the connection boundaries behind this diagnosis order.
• Inference Platform Architecture connects this small request path to larger serving pools.
• Latency metrics separates first-token timing from throughput and useful outcomes.

Key takeaways

• Local success leaves remote reachability and authorization untested.
• The worked first-token trace totals 315 ms across distinct boundaries.
• Select platform capabilities after identifying the responsibilities you need.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 1The model responds locally, but the remote caller cannot resolve its hostname. What do you inspect first?

   Name resolution from the callerGPU memory capacityThe quantization format

2. 2What remains of the 315 ms trace if 40 ms of setup is avoided?

   Answer it, then reveal