Crusoe AI Infrastructure Engineer interview questions

Normalize link directions, count collective bytes, and see why a gradient reduction can outlast its real overlap window.

Derive a 16-leaf, 8-spine fabric from 512 endpoints, trace local and cross-leaf routes, and check what rail placement and oversubscription actually guarantee.

Compare host-staged and direct GPU-to-NIC transfers, count traffic at each boundary, and check transport selection before blaming a slow collective on missing GPUDirect.

Follow a host buffer across a NUMA boundary, calculate the cost with declared bandwidths, and align CPU workers and memory with the GPU that consumes the data.

Compare the same benchmark on a healthy node, then follow the evidence through NCCL transport choices, device topology and fabric health. Half the measured bandwidth is a symptom, not a unique diagnosis.

Specify a cluster from workload and facility constraints, then derive node counts, network ports, storage bursts and acceptance tests.

Compare rack-level power, cooling, structural and connectivity requirements with the offered site, then evaluate quoted deployment alternatives.

Build a cluster power budget with clear equipment boundaries, facility overhead and failover requirements. Distinguish energy use from reserved electrical capacity.

Derive cooling flow for a 4.8 MW hall, separate liquid and air loads, and test CDU capacity and redundancy at the actual design conditions.

Localize a slow fabric rail with matched tests, path comparisons and counter deltas. Separate an independent-bandwidth sum from collective completion time.

Define acceptance with component and system tests, absolute targets, peer comparisons and documented retests. Interpret burn-in exposure without treating it as a reliability guarantee.

Investigate rack power events at the correct electrical boundary and time scale. Test load reductions without assuming GPU caps control every watt or transient.

Count one complete rail, then distinguish logical links from cages, optical modules and cable assemblies. Derive power from the same physical inventory.

Design a management path that survives specified production failures. Count all endpoint classes, restrict device authority and test the complete recovery route.

Compare cooling methods from the supported hardware and heat path. Calculate airflow or liquid flow, then verify capacity, residual air load and failure response.

Build a deployment schedule from dependencies and committed lead times. Calculate float and price the delay only when an action moves usable capacity earlier.

Estimate replacement demand over replenishment time, then size for shortage risk and failure correlation. Distinguish shelf parts from runnable spare nodes.

Investigate widespread acceptance failures using matched controls, valid thresholds and cohort denominators. Preserve hardware and shared causes as testable hypotheses.

Separate compute, storage and management responsibilities, then decide which need independent physical paths and count their endpoints correctly.

Count endpoints, allocate every switch port once, and compare a common fabric with independent rails before adding a third tier.

Compare ownership, colocation and rental using the same useful work, full annual costs and realistic demand. Work a break-even from explicit assumptions.

Budget the full megawatt, fit whole servers, and compare useful training throughput under measured power limits. A worked cap example shows when more GPUs help.

Choose a supported tenant boundary, secure the surrounding services, and verify cleanup before reassignment. Calculate fragmentation from actual requests.

Map stable endpoints to rails, choose supported cables from measured routes, and verify topology before interpreting collective performance.

Reproduce the baseline, compare equivalent work, and trace the missing time. Acceptance, production and the original plan may differ in more than one way.

Plan a GPU-generation migration around usable workload capacity, site readiness, compatibility and a tested rollback path.

Test the vendor claim on your workload, including quality, latency, scaling and operating costs. A lower peak or lower price alone does not settle it.

Gate admission with fresh, platform-aware evidence. Use absolute and peer baselines, budget test overhead and keep automated cordons under control.

Survey real facility headroom, phase work around shared dependencies, and compare retrofit with relocation using useful capacity and total transition cost.

Define who owns each facility and hardware action, model restoration time, and verify power, cooling, telemetry and growth commitments before deployment.

Confirm the event definition and exposure, compare cohort rates, then test whether the cause follows a module, location or optical path.

Compare complete fabric proposals on workload performance, support and total cost. Translate any measured regression into application impact before pricing the discount.

Compare the old fleet's useful value with operating costs and resale, verify its health, and complete data handling before reassignment or transfer.

Diagnose a rack outage through independent evidence, preserve safety interlocks, and restore capacity only after its failed paths pass acceptance checks.

Use per-rank arrival times and controlled substitutions to explain a slow step without mistaking collective wait for network transfer.

Treat recovery as a new worker generation, then compare spare replacement with a legal shrink and its batch consequences.

Read a watchdog timeout without blaming the reporting rank. Compare group-scoped traces, separate missing evidence from mismatched calls, and distinguish CPU submission from GPU completion.

Count gradient traffic with the actual dtype, convert one-direction rates, and expose the bucket-readiness constraint that an average bandwidth division misses.

Build a power budget from complete servers to the facility meter, then check rack limits and distinguish a design maximum from an annual electricity bill.

Convert useful tokens into input traffic, then calculate separate checkpoint and restore budgets. Learn why an asynchronous writer can keep up with the cadence and still miss the recovery deadline.

Choose between supported InfiniBand and RoCE designs using workload tests, congestion behavior and a complete cost model.

Trace registered-memory access and GPU staging, then compare RDMA with sockets using measured CPU and application performance.

A bandwidth regression narrows the search only after the experiment is defined. Work through rank-count normalization, controlled path tests and the difference between collective time and training progress.

Count endpoint, leaf, aggregation and core ports for an explicit 4,096-GPU design. Compare 1:1 and 2:1 without confusing switch ports, links or optics.

Design a durable dataset and optional working-set cache around access semantics, request overhead and versioned readiness.

Separate host-buffer, GPU-buffer and collective tests, with explicit message-size models and bandwidth metric definitions.

Derive a complete rail topology and distinguish physical links, switch ports and transceivers. Size storage and management from their own requirements.

Can one storage system serve datasets and checkpoints? Follow a save from staging to recovery, calculate its budget beside protected reads, and learn when separate paths actually solve the contention.

Derive the two-tier endpoint limit, audit explicit 8,192-endpoint constructions and compare switches, links, optical modules and failure paths.

Two tenants can fit the average storage budget and still miss both checkpoint deadlines. Work through reservations, borrowing and bounded admission while protecting training reads, then test the design under failure.

Choose the control plane from job admission, placement, recovery and service needs, with a concrete partial-allocation failure.

Use a short admission check for recent changes and a deeper acceptance suite for new or repaired nodes. Set thresholds from matched healthy baselines, and make quarantine and return-to-service decisions explicit.

Follow a GPU request from node inventory through allocation, device confinement and task launch. A single-node, eight-task example shows how resource shape and per-task device numbering fit together.

Calculate traffic across the actual inter-cloud boundary before deciding. Whole-job placement often avoids repeated transfers, but communication volume, overlap, convergence requirements and available links determine whether a split job is viable.

Find the critical-path delay using local phase timing and controlled comparisons, rather than diagnosing a node from utilization or clock ratios.

Build an acceptance suite around the node’s intended workload and supported hardware limits. Combine configuration, correctness, communication and sustained-load evidence, then record a baseline and an explicit return-to-service decision.

Correlate clock-event reasons, temperature, configured limits and workload timing. Use the scope of the slowdown to choose the next investigation, and distinguish intentional power limits from unexpected thermal constraints.

Measure throughput at several power limits, translate the facility target into a GPU budget, and account for slower ranks before changing the fleet.

Derive a 10,240-GPU layout from model work, then account for collective traffic, durable checkpoints, power and replacements. A nested diagram connects tensor, pipeline and data parallel groups to the fleet.

Write a bounded Go downloader that propagates cancellation, joins every worker and publishes files under explicit unique names.

Evaluate a proposed 2,000-GPU purchase using compatible capacity, dated demand and complete alternatives. Make any gap left by reclaim visible.

Move a vendor case forward with controlled comparisons, a faithful reproduction and a safe mitigation. Escalate by impact while evidence improves.

Explain a career move using the actual role, team and your priorities. Test assumptions about ownership, hardware access and support.

Follow the weight bytes through a small decode batch and a long prefill. A scoped roofline calculation explains the difference without pretending to predict actual latency.

Follow a token into retained attention state, derive bytes per position for a 70B model, and distinguish present cache usage from the room needed to finish a response.

Work from each GPU's cache budget, reserve room for generated tokens, then check latency separately. A limiting-rank example shows why fitting today's prompts is not enough to admit tomorrow's work.

Separate quota, borrowing, reclaim latency and fairness, with a demand-conserving example of when pooling improves utilization.

Work from token timestamps to first-token wait, average pace and individual stalls. Then count requests meeting both latency limits, keeping failures and measurement boundaries explicit.

Draw query-head sharing, count actual stored state and compare payloads with explicit model geometry and binary units.

Follow a four-GPU request from advertised capacity to concrete devices, and see why free GPUs across nodes cannot satisfy one pod.

DRA gives the scheduler device claims and attributes to work with. Compare that with extended-resource counts, then check the deployed driver and controller support before promising sharing, partitioning or gang placement.