Connect request delays to scheduler and device evidence

Request delays need scheduler and device evidence to explain them. At a steady 20 requests/s, growing prompts from 1,000 to 4,000 tokens quadruples offered prompt work; high cache occupancy may also include state that can be reclaimed.

19 MIN · PREMIUM