29Roll out a driver and firmware upgrade across a live 2,048-GPU fleet without losing a training run.▼hardMetaCoreWeaveMicrosoft4 replies◆ premiumProtect running work, validate a supported upgrade bundle, and expand through measured cohorts with a tested recovery path.Open full answer →
19You have one week to get a newly released trillion-parameter model serving production traffic. Plan it.▼hardTogether AIFireworks AIBaseten4 replies○ sign inBuild a week around release gates and dependencies. Budget artifact transfer, validate the exact runtime path, measure the intended workload, and size the canary evidence before promising full rollout.Open full answer →
23The model authors publish a new revision. How do you roll it out without a quality regression?▼mediumBasetenTogether AIModal4 replies◆ premiumA minor revision can change weights, encoding or response behavior. Compare pinned deployment bundles, evaluate task slices, monitor a representative canary and keep enough capacity to restore the old service.Open full answer →
29Upgrade the GPU driver across 2,000 live nodes without breaking running jobs. Walk me through the plan and what can go wrong.▼hardCoreWeaveLambda4 replies◆ premiumTreat the upgrade as a tested change to a compatibility set. Drain safely, canary representative workloads, size waves from usable spare capacity and rehearse restoration of the previous working configuration.Open full answer →
18Error rate on an inference fleet tripled ten minutes after a deploy. What do you do first, and what should have caught it?▼mediumBasetenOpenAI4 replies○ sign inStop rollout expansion, preserve evidence and use the safest available mitigation. Compare versions under representative load, and size canary exposure from the regression you need to detect rather than a fixed traffic percentage.Open full answer →