One fabric link goes down and comes back once an hour. What does that do to a 1,024-GPU training job, and how would you find it?

Trace a link flap through transport and job recovery. Calculate detection, restart and replay costs, and learn what continuous counters and timed tests can actually establish.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Trace a link flap through transport and job recovery. Calculate detection, restart and replay costs, and learn what continuous counters and timed tests can actually establish.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

💻 Coding for Infra🔒 Premium

Batching Queues and BackpressureA request batcher groups work to reduce processing overhead. Queue bounds, dispatch timing and failure handling keep that efficiency from becoming uncontrolled waiting.

Foundational

🗂️ Scheduling & Orchestration

Node Lifecycle: Drain, Upgrade and ReturnFleet maintenance moves nodes through drain, repair and validation before returning them to service. Deadlines and concurrency limits keep that work within the available capacity.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Stragglers and HangsTrace how a late worker reaches the critical path, distinguish longer steps from lower throughput, and separate ordinary jitter from a stalled job. Worked examples show when delay is hidden, how shared jitter changes the math, and what evidence supports a repair.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Incident Response for GPU FleetsGPU incident response restores useful service while preserving enough evidence to diagnose the failure. Mitigation, repair and return-to-service checks have different purposes.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWhy does object storage behave badly for random reads, and what does a training pipeline do about it?Next in this trackDeduplicate and quality-filter a multi-petabyte web corpus. What does that pipeline cost and where does it bottleneck?Next in this trackA mixture-of-experts model does an all-to-all twice per layer. What does that demand of the fabric, and what changes on a rack-scale system?

FEDITOR'S NOTE

Separate collective dependencies from barrier semantics, physical links from rank membership, and outage duration from lost job progress. Include detection in restart cost.

DISCUSSION · 0

No comments yet — be the first to share your approach.