13You have one GPU and a synchronous API that receives 100 documents at once. Design the batching, and show the latency math.▼medium★ EssentialNewAnthropic4 replies○ sign inSequential is fifty seconds; one batch is four. In between are the questions the interviewer is holding: how you pack ragged inputs, what a batch window costs, and where the memory stops you.Open full answer →
24Design a platform that serves twenty open-weights models of varying size to internal teams.▼hardNewBasetenModalTogether AI4 replies◆ premiumTwenty models of different sizes is a packing problem with a cold-start penalty, and the two pull in opposite directions. What stays resident, what loads on demand, the arithmetic that decides which is which, and the interface that stops every team from asking for a dedicated replica.Open full answer →
