TL;DR: The GPU Operator manages configured parts of the NVIDIA software stack, such as drivers, container integration, device discovery and monitoring. Node Feature Discovery describes detected node features through labels. Labels guide configuration; the device plugin advertises resources; validation checks behavior. These are related signals with different meanings.
How to approach it
Follow what the application needs: a working driver, container access, a resource allocation and successful execution. Identify which components the operator owns and which are preinstalled. Then explain discovery, reconciliation and the fleet's actual admission policy.
A strong answer
In Kubernetes, an operator is a controller that installs a piece of software and keeps it in its declared state, repairing drift. A new node has a GPU product label but advertises no GPU resource. The label is not a reservation and does not prove successful device-plugin registration. Another node may advertise devices while failing a compute test. Keeping those signals separate makes operator incidents easier to reason about.
Node Feature Discovery exposes detected hardware and software features according to its sources and configuration. PCI-derived labels can include vendor and class information. It is not accurate to reduce all discovery to one vendor-presence label or to say it cannot identify GPU-related features. GPU-specific discovery adds richer information from the NVIDIA stack.
The GPU Operator overview lists managed components and supported deployment modes. The exact set varies by configuration and platform. A host-installed driver or toolkit may be intentionally retained; do not run two independent managers that both try to own the same layer.
| Responsibility | What its evidence establishes |
|---|---|
| Driver and fabric services | The relevant host/device stack can initialize |
| Container integration | The workload can receive supported device/library access |
| Device plugin | Kubernetes can advertise and allocate its resource type |
| Feature discovery | Labels describe detected attributes for selection |
| DCGM exporter | Supported GPU telemetry reaches the monitoring system |
| MIG manager, when enabled | Requested supported partition configuration is reconciled |
| Validation | The configured checks passed at that time |
These dependencies do not form one universal serial startup list. Several operands can progress independently once prerequisites are ready. Kubernetes GPU Scheduling covers the resource side; Cluster Bring-Up: Firmware, Drivers and the Stack covers the broader platform dependencies. Firmware, BIOS and every host component are not automatically managed by a GPU Operator installation.
Reconciliation means comparing desired configuration with observed state and taking managed actions to converge. It does not eliminate failed downloads, unsupported kernels, bad hardware or rollout planning. Version pinning, support-matrix checks, canaries and rollback remain operating work. Read status and operand logs to find the dependency that is preventing convergence.
Do not assume a validator automatically blocks every user pod through a standard taint. Verify the installed version and your fleet's readiness/admission integration. If production admission requires a burn-in test, implement and test that policy explicitly. A green validator result should have a timestamp and a known scope; it does not guarantee the node stays healthy forever.
A synthetic rollout budget illustrates automation without inventing engineer-hours saved:
Fleet: 200 nodes
Per-node maintenance duration: 30 minutes, assumed
Parallel wave size: 10 nodes
Waves: 200/10 = 20
Ideal elapsed rollout time: 20 × 30 = 600 minutes = 10 hours
Node maintenance time: 200 × 0.5 = 100 node-hours
Sanity: 100 node-hours is not automatically 100 hours of human labor.
Canary observation, draining long jobs, failures and rollback can lengthen elapsed time. Increasing wave size shortens the ideal schedule but enlarges the failure blast radius. Decide the wave size from service headroom and recovery evidence, not just the fastest possible update.
Use the operator when its supported management model fits the fleet and reduces configuration drift. A controlled immutable-image workflow can also provide convergence by replacing nodes, and can coexist with selected operator components. Reverse a full-stack operator choice when ownership conflicts or unsupported requirements make it less reliable than the tested image path.
What interviewers probe next
- "Who writes allocatable GPU count?" The kubelet exposes device-plugin resources; feature labels are a different interface.
- "Does an image lose all reconciliation?" No; an image replacement controller can reconcile by replacing nodes.
- "What should version alerts compare against?" The declared desired version and rollout phase, not only the fleet's modal version.
Common mistakes
- Treating labels, resource count and readiness as interchangeable.
- Assuming one component list applies to every configuration.
- Counting automated node-minutes as manual engineer-minutes.
Key takeaways
- Discovery describes; allocation reserves; validation tests.
- Twenty ten-node waves at thirty minutes each take ten ideal hours.
- Assign one clear owner to each managed layer.