Allocated, active and placeable are different capacity measurements

TL;DR: Allocation measures assigned resources. Device activity measures a particular hardware behavior; useful throughput measures delivered work. Placeable capacity asks whether a specified job fits the free resources and constraints. A gap between these numbers needs attribution before buying hardware, reclaiming sessions or changing placement policy.

Where you are. First lesson of the fleet course. A dashboard shows 90% allocated while a device metric averages 30%. Jobs are still queued. Start by identifying the units, population and time window behind each number, then inspect what is preventing useful work or a valid placement.

Define each denominator

For exclusive devices, allocation can be expressed as assigned device-hours divided by available device-hours. A sampled activity average may instead describe time with a kernel executing. It is not automatically the percentage of peak arithmetic achieved, the percentage of memory in use, or the percentage of jobs doing useful work.

Suppose a synthetic ten-device fleet is observed for 100 hours: 1,000 available device-hours, 900 allocated device-hours and 300 kernel-active device-hours. Allocation is 90% and fleet-average sampled activity is 30%. If every active interval lies within an allocation, activity within allocated time is 300/900 = 33.33%. A dashboard averaging only allocated devices would have a different denominator from one averaging the whole fleet.

The 600 allocated-but-inactive device-hours need explanation. They may include input stalls, intended service headroom, checkpoint/recovery time or abandoned sessions. High activity also does not guarantee efficient work: a memory-bound kernel or communication polling can look busy. Fragmentation primarily describes free resources that cannot satisfy a request; it does not by itself explain why an allocated GPU is inactive.

The pair of percentages alone establishes neither that hardware should be bought nor that it should be removed. Compare forecast demand, queue objectives, recoverable inefficiency and the cost of each intervention.

Calculate placeability for the actual request

Assume 32 nodes with eight exclusive devices each, for 256 devices total. Ignore CPU, memory and topology restrictions temporarily. The synthetic free-device distribution is:

Free capacity is 40/256 = 15.625%. For jobs requiring k devices on one node, eligible nodes count those with at least k free; simultaneous identical-job slots sum floor(free_on_node/k):

The two four-free-device nodes each accommodate two two-device jobs, so eleven eligible nodes provide thirteen slots. The columns answer different questions. Nor can all rows be used at once: they describe alternative allocations of the same 40 devices.

Real placement must also satisfy CPU/RAM, device type, health, storage, security and communication constraints. A job permitted to span nodes may fit when a same-node job does not, with a potential communication cost. That is a workload decision, not a reason to silently ignore its requested shape.

Resource granularity is a policy and platform choice

Whole-device allocation is common, but hardware partitions and supported sharing mechanisms can expose smaller or multiplexed units. Their memory, compute, fault and security guarantees differ. Count the schedulable resource actually advertised; eight time-sharing slots are not necessarily eight independent devices of capacity.

Departures of differently sized jobs can leave scattered holes, but fragmentation does not inevitably worsen forever. Arrival mix, packing, job lifetimes and periodic emptying determine its evolution. Measure the shape distribution over time rather than assuming a universal trend.

Choose a recovery policy with its costs

Packing small jobs onto nodes with enough remaining resources can preserve whole nodes for larger jobs. A reservation for a particular shape can improve start-time predictability while leaving capacity idle. Backfilling with jobs that finish before the reservation can recover some of that time if runtime estimates and enforcement are credible.

Consolidation is more disruptive: moving a GPU job may require checkpoint/restart rather than transparent migration. In the diagram's small example, moving three compatible single-device allocations from C to the one free slot on A and two on B creates four free devices on C. That works only under the stated compatibility and restart assumptions. Count lost work, data reload, interference and user impact before claiming a saving.

Idle reclaim needs a phase-aware policy. A training job writing a checkpoint or an inference replica held for failover is not equivalent to an abandoned notebook. Establish thresholds, ownership, notice and state-preservation behavior. Keep sharing restricted to workloads whose isolation and latency requirements are met.

Choose the intervention that improves queue and useful-work objectives at acceptable cost. Reverse aggressive packing if it causes unacceptable contention or recovery concentration; revise reservations if the large-job benefit does not justify the idle time. Hardware expansion can still be justified when the validated workload exceeds the capacity that policies can recover.

Do this before moving on

For the 32-node example, compute same-node slots for jobs needing three devices. Only the two nodes with four free qualify, giving two slots. Then add a CPU requirement that one of those nodes cannot satisfy: only one placement remains, despite unchanged free GPU count.

Design a fleet report with allocation, defined activity metrics, delivered throughput, reserved resources and job-specific placeability/queue delay. Keep the denominator visible. For one queued job, name the exact unsatisfied constraint and the least disruptive action that would make it feasible. The next lesson, Gang Scheduling with Kueue and Volcano, examines why a feasible distributed job also needs coordinated admission.

Go deeper

• Kubernetes GPU Scheduling explains how schedulable device resources are exposed and assigned.
• Gang Scheduling with Kueue and Volcano connects resource shape with coordinated admission of workers.
• Topology-Aware Scheduling adds communication constraints to the placeability calculation.
• MIG, MPS and Time-Slicing compares the capacity and isolation implications of sharing mechanisms.
• The cluster shows 90% allocated and 30% utilised practices attribution from an ambiguous pair of metrics.
• Bin packing GPUs: fragmentation and defrag weighs consolidation benefits against disruption and restart costs.

Key takeaways

• 90% allocation and 30% fleet activity become 33.33% activity within allocated time only under the stated population assumptions.
• Fragmentation concerns unusable free shapes; low activity in allocated devices needs separate attribution.
• Forty free devices can provide zero eight-device same-node placements.
• Eleven eligible nodes provide thirteen two-device slots in the worked distribution.
• Judge packing, reclaim and sharing by queue outcomes, useful throughput and their actual costs.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 1The example has nine nodes with two free devices and two nodes with four free. How many simultaneous two-device same-node jobs fit?

   Zero, because no node is empty11, because eleven nodes qualify20, because all forty free devices can be paired13, because the four-free nodes each hold two jobs

2. 2What does 90% allocation and 30% fleet-average sampled kernel activity prove?

   Fragmentation is the only causeThe metrics differ; their populations and workload phases need attributionExactly 60% of the fleet can be reclaimed safelyThe fleet must be expanded

3. 3Can device sharing eliminate every packing problem?

   Answer it, then reveal