← AI Infrastructure System Design / 35

expertNewNVIDIAGoogle CloudMicrosoft Azure

Design an inference service whose operator cannot read customer prompts or the model owner's weights.

Separate customer, model-owner and operator trust. Bind encrypted requests and model-key release to an approved CPU/GPU workload, then account for retrieval, cache, logs, scaling and recovery.

Updated Nov 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Separate customer, model-owner and operator trust. Bind encrypted requests and model-key release to an approved CPU/GPU workload, then account for retrieval, cache, logs, scaling and recovery.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 324 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

📐 AI Systems Design🔒 Premium

Remote Attestation and Key ReleaseAttestation supplies evidence about a running environment. A key-release policy must also bind freshness, approved software, every required device and the recipient channel before releasing a model or data key.

🚀 Inference & Serving🔒 Premium

Confidential InferenceConfidential inference keeps prompts, model state and responses inside an attested CPU/GPU boundary during execution. Its privacy claim depends on where requests decrypt, which code receives keys, and where plaintext can leave.

📐 AI Systems Design

The AI Infra Design Round PlaybookA design interview is a chance to explain requirements, capacity and tradeoffs clearly. Use a flexible structure that leaves time for mechanisms, failures and follow-up questions.

📐 AI Systems DesignSign in

GPU Job Scheduler DesignA GPU scheduler decides which jobs run, where they run and when resources can be reclaimed. Its placement rules must account for whole jobs, topology and tenant policy.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Closest to this topicHow would you measure confidential inference overhead without confusing encryption, cold starts and batching?Closest to this topicDeploy a confidential inference service at an air-gapped site. What must work after the internet is disconnected?Closest to this topicGPU attestation fails after a driver update. How do you restore serving without bypassing the key-release policy?

FEDITOR'S NOTE

A credible design identifies every plaintext exit and tests denied launches. A green device report is only part of the release decision.

DISCUSSION · 0

No comments yet — be the first to share your approach.