06Write a fused row softmax in Triton, explain why it is one HBM pass, and say where it stops scaling.▼mediumNewOpenAIFireworksTogether AI4 repliesunlockedOne program per row, the row in registers, max then exp then sum then divide, one read and one write of HBM. The runnable kernel with its launch, the numerics (subtract the max, accumulate in fp32), the block-size rule and the wide-row limit, and the arithmetic that says the kernel is done at 90% of copy speed.Open full answer →
18Implement sampling from a model's logits with temperature, top-k and top-p. What are the numerical traps?▼easyNewOpenAIAnthropic4 replies○ sign inFour lines of arithmetic with three ways to get it wrong: dividing by a temperature of zero, exponentiating without subtracting the maximum, and applying the filters in an order that changes the result. The implementation, the verification against the analytic distribution, and the overflow that is one logit away.Open full answer →
