Oracle Cloud Infrastructure AI Infrastructure Engineer interview questions

192 GB and 5.3 TB/s per card beat the H100 on both memory numbers at a lower hourly price. Whether that wins in production is a question about achieved bandwidth and ROCm kernel coverage, and there is a short list of numbers to run before you sign the order.

The first number every serving interview asks for, derived from parameter count and bytes per parameter, in three precisions, with the part people forget to add.

Four numbers from the config file, one formula, and a per-sequence result that is a third of the model's own weights. Plus the mistake that makes the answer eight times too big.

The 6ND rule, where the 6 comes from, and the 6.3e24 that every training-time and GPU-count question in the loop is built on.

The fleet equation applied to 6.35e24 FLOPs: 11 days, and how the answer swings from 9 to 15 with the one parameter the interviewer wants you to state.

The fleet equation solved for GPU count: about 6,200 H100s at 40% MFU, why it rounds up to a power of two, and the reasons a good answer adds 15% before naming a number.

Why decode speed is a division of bandwidth by weight bytes, why the answer is about 24 tokens a second regardless of how fast the tensor cores are, and the two ways to double it.

Fleet dollars per second over tokens per second, times a million: a worked $2.78, the utilization term that turns it into $6.94, and why batch is the lever that moves it by 50x.

Sixteen bytes per parameter is 1.1 TB of static state before a single activation; LoRA keeps the 141 GB of frozen weights and shrinks the rest to a few gigabytes. The derivation, the GPU counts, and the activations that both still pay.

Two divisions decide it: the kernel's FLOPs per byte against the card's ridge point. The worked case, the profiler counters that give the inputs, and what to change once you know which wall you hit.

Gradient bytes per step over step time, with the ring all-reduce factor that nearly doubles it: the arithmetic for a 7B and a 70B, the link speeds that cover it, and the step-time threshold where overlap stops hiding it.

Prefill is the compute-bound half of serving: 2 × N × tokens FLOPs over the effective TFLOPS of the replica. The chain that gives a 0.24 s floor on a node, the single-card version, and what it means for time to first token.

6ND on 15.6 trillion tokens gives 3.8e25 FLOPs; converting to GPU-hours needs an MFU, and Meta's published 30.8 million hours lets you solve for it. The chain, the check, and what the implied MFU tells you.

Two constraints per replica, memory and throughput, each derived from the model and the traffic; whichever binds sets the users per replica, and the replica count follows with headroom. The full chain from 1,000 users to a number of nodes.

Memory follows total parameters and compute follows active ones, so the two models pull the GPU count in opposite directions. The chain for a 235B-total, 22B-active MoE against a dense 70B, for serving and then for training.

Sixteen bytes per parameter is 6.5 TB per checkpoint; over a 100 GB/s filesystem that is about a minute, which at a 30-minute cadence is 3% of the run. The chain, the storage tiers that meet it, and why the write is made asynchronous.

700 W per GPU is the number everyone knows and it is only 40% of the answer. The chain from GPU TDP to node, to IT load, to facility power with PUE: about 1.8 MW, and what that means in racks and dollars.