AI Infra Interviews logo

Build a prefix-aware routing proxy and test its failure paths

Compare round-robin, prefix affinity and a load-aware policy. Stream responses through a bounded local proxy and account for busy or failed replicas.

PremiumDistributed service project

AI Infra Interviews editorial · Updated

Does hashing a prompt tell a router where its KV cache lives?

No. A stable hash can encourage repeated prefixes to reach one replica, but it cannot observe eviction, tokenization or cache residency. This project compares that affinity hint with round-robin and a queue-aware heuristic. Local HTTP checks cover admission and stream failures. Measuring cache benefit still requires compatible model replicas and engine metrics.

Read the primary sources
What you build
A tested local routing proxy, policy experiment records and a cache-residency limitation analysis.
Environment
CPU · Python 3.12 · two compatible model replicas for cache measurements
Plan your session
2–4 hours plus optional two-replica GPU runs. A planning estimate, not measured runtime.
Before you begin
Complete the streaming client and controlled serving experiment. For cache measurements, supply two compatible replicas with the same model, tokenizer and prefix-cache settings.
HTTP routing executed · GPU cache experiment pending. Authentication, admission, affinity, streaming and failure release checked with local HTTP servers. No two-replica GPU cache run or TTFT reduction is claimed.
  1. 01Prefix hint

    Hash the leading system messages and a deployment identity.

  2. 02Admission

    Apply the routing policy, replica cooldown and in-flight limit.

  3. 03Stream

    Relay output once and release the reservation even after failure.

Follow the changing state through the project. Each boundary has a check in the walkthrough.

Continue with Premium

Premium includes this guide’s complete walkthrough and downloadable project files. The technical overview, prerequisites and verification notes above are public.

View Premium access

Start with Stream vLLM responses with Python and detect incomplete answers (Free to read).

Primary sources

Checked . Source review and execution checks are described separately above.