TL;DR: Account for declared bytes, keep recently used entries easy to find and protect in-use entries with reference-counted pins. Plan enough eviction before changing the cache so a failed admission leaves existing entries intact. Distinguish an oversized item from temporary pin pressure. Also state which storage layer the cache manages; a local file hit is not a guarantee that weights are already ready on the GPU.
How to approach it
Define the byte budget, key identity and ownership rules first. Implement lookup, acquire/release and admission, then test recency, shared pins and failed insertion. Explain which concurrency, loading and durability concerns remain outside the exercise.
A strong answer
An LRU (least recently used) cache evicts the entry that has gone longest without being used when it needs room. Suppose a serving node reuses weight artifacts of different sizes. Counting entries does not predict their total footprint. At an illustrative two bytes per parameter, 7 billion parameters occupy 14 GB and 70 billion occupy 140 GB before metadata or runtime state. Four artifacts could therefore represent 56 GB or 560 GB. Quantization and file representation change those figures.
Containers, Images and GPU Cold Starts explains the different loading stages. The code here is an in-memory directory of declared-size values. It demonstrates admission and lifetime rules; it does not allocate HBM, download files or measure actual filesystem occupancy.
Plan admission before evicting
Use immutable versioned keys, such as a model revision plus representation. An existing key cannot be silently replaced while another user holds it. acquire increments a pin count and returns the value; release decrements the count. Ordinary get updates recency but does not grant a protected lifetime.
from collections import OrderedDict
class PinnedCapacityError(RuntimeError):
pass
class WeightCache:
"""Single-threaded metadata model; positive declared byte sizes."""
def __init__(self, capacity_bytes):
if type(capacity_bytes) is not int or capacity_bytes <= 0:
raise ValueError("capacity must be a positive integer")
self.capacity = capacity_bytes
self.items = OrderedDict() # key -> (size, value), oldest first
self.pins = {} # key -> active holders
self.used = 0
self.evictions = 0
def get(self, key):
value = self.items[key][1] # KeyError distinguishes absence from None
self.items.move_to_end(key)
return value
def acquire(self, key):
value = self.get(key)
self.pins[key] += 1
return value
def release(self, key):
if key not in self.pins or self.pins[key] == 0:
raise ValueError("no pin to release")
self.pins[key] -= 1
def put(self, key, value, size):
if type(size) is not int or size <= 0 or size > self.capacity:
raise ValueError("invalid or oversized entry")
if key in self.items:
raise ValueError("key already present; use a new artifact version")
remaining = self.used + size - self.capacity
victims = []
for candidate, (nbytes, _) in self.items.items():
if remaining <= 0:
break
if self.pins[candidate] == 0:
victims.append(candidate)
remaining -= nbytes
if remaining > 0:
raise PinnedCapacityError("admission blocked by pinned entries")
# All expected admission failures occurred before this mutation.
for candidate in victims:
self.used -= self.items.pop(candidate)[0]
del self.pins[candidate]
self.evictions += 1
self.items[key] = (size, value)
self.pins[key] = 0
self.used += size
Python's OrderedDict supports moving a touched key to the recent end. Lookup and reordering are expected constant-time operations; admission can scan all N entries and remove multiple victims, so it is O(N) in the worst case. Do not describe the whole put as constant-time.
Work through the pin-pressure case
These numbers are arbitrary budget units; treating each as a GB gives the same accounting example:
c = WeightCache(300)
c.put("A", "weights-a", 140)
c.put("B", "weights-b", 90)
assert c.get("A") == "weights-a" # B is now oldest
c.put("C", "weights-c", 145)
assert list(c.items) == ["A", "C"]
assert c.used == 285 and c.evictions == 1
c.acquire("A")
c.acquire("A")
c.release("A") # one other holder still protects A
before = (list(c.items), c.used, c.evictions)
try:
c.put("D", "weights-d", 200)
except PinnedCapacityError:
assert (list(c.items), c.used, c.evictions) == before
else:
raise AssertionError("expected pin-pressure failure")
c.release("A")
c.put("D", "weights-d", 200)
assert c.used == 200 and list(c.items) == ["D"]
After A is pinned, evicting C would leave 140 units in use. The new 200-unit item would need 140 + 200 = 340, exceeding the 300-unit budget. The planned admission fails without pointlessly deleting C. A 400-unit item fails earlier because no eviction plan can fit it at all.
Capacity is 300 units. A140 and C145 occupy 285; A has a remaining pin.
- Try to insert D200
The cache needs to free at least 185 units.
- Only C145 is evictable
A140 + D200 = 340, still above capacity after removing C.
- Reject before removing C
The existing A and C entries remain intact. Retry only when policy or pin state permits.
Separate logical ownership from physical storage
For device allocations, releasing memory while a kernel uses it can violate correctness. For filesystem artifacts, unlinking a file is not automatically an immediate crash for an existing open mapping; operating-system and application semantics matter. The cache still needs a lifetime policy to avoid losing a required path or reclaiming storage that remains physically referenced.
Use try/finally or a context manager around real holder lifetimes, including error paths. A set of pins is insufficient when two replicas share one artifact: the first release must not unpin the second. Concurrent use needs synchronization across lookup-and-acquire and admission; the example deliberately requires serialized calls.
Downloads also need space before publication. Reserve temporary bytes, verify completeness and integrity, then publish the artifact according to the storage system's atomicity and durability rules. The code's expected admission failures leave the directory unchanged, but it is not a crash-transaction implementation or protection against arbitrary allocation failure mid-mutation.
Inference Autoscaling and Cold Starts explains why these layers affect readiness. For an assumed 140 GB payload and sustained 6 GB/s local reads, the transfer floor is 140 / 6 ≈ 23.3 seconds; at an assumed 1 GB/s remote path it is 140 seconds. A local cache hit can avoid the remote transfer while still requiring local reads, validation and transfer to GPU memory. It is not automatically free.
Routing affinity can improve reuse, but random placement does not necessarily make cache hits vanish. Measure the working set, popularity, placement and time between uses. Choose more complex eviction only when the workload supports its added cost.
What interviewers probe next
- “What if everything is pinned?” Report admission pressure and wait, reject or choose another node according to policy; never silently revoke holders.
- “Can I update an existing artifact?” Use a new immutable key and an explicit transition, rather than overwriting a pinned entry.
- “Does this enforce disk usage?” No; it accounts declared values and needs physical storage, temporary-file and concurrency controls around it.
Common mistakes
- Mutating recency or deleting victims before an admission that will fail.
- Using one boolean pin for several independent holders.
- Treating a filesystem cache hit as GPU readiness.
- Calling a scan-and-evict insertion constant-time.
Key takeaways
- The 300-unit example cannot hold a pinned 140-unit item and a new 200-unit item; failure preserves the existing cache.
- Track bytes and holder counts, with a defined admission result.
- A 23.3-second transfer floor is not a complete cold-start time.