Your first steps in AI infrastructure
AI infrastructure is the hardware and software that trains models and runs them for people. Start with a single request. The bigger systems will make more sense once you can explain that one.
No AI background or GPU purchase needed. This whole introduction is free to read without an account.
You ask a chat application a question. Its server needs software to handle your request and hardware to calculate an answer. A CPU (central processing unit), the general-purpose processor, coordinates the application. A GPU (graphics processing unit) can execute many suitable calculations in parallel.
Small models can also run on CPUs. A GPU is a workload choice, not a requirement for every inference task.
Run the application. Prepare inputs and launch work on the GPU.
Apply model operations to many numbers.
Holds model weights and working state while computation runs.
How much data can be held. A model can fail to load when memory is full.
How quickly bytes can move across an interface. Moving data can limit the computation.
Two GPUs have the same memory capacity. Must they run a model at the same speed?
No. Their memory bandwidth, compute capability and software support can differ. Capacity answers whether the allocations fit; speed requires checking the workload.
A model’s architecture describes its operations and connections. Its weights are learned numerical values, also called parameters. Training adjusts those values against an objective; inference uses a trained model to produce an output.
Examples → predictions → error signal → weight updates
A saved model state. The same weights can serve many requests.
New input + trained model → output
8B means roughly eight billion parameters. It does not mean 8 GB. The memory needed also depends on how each value is represented. Quantization uses lower-precision representations to reduce storage; test whether the converted model still handles your task well.
An open-weight model makes trained weights available to download under its license. Running it also needs compatible software, usually called a runtime. You can use a hosted service instead of operating the hardware yourself.
If ten people use the same model, do you always need ten separate copies of its weights?
No. A serving process can reuse shared weights across requests. Each request still needs its own working state, and extra replicas may be needed for throughput or reliability.
A tokenizer converts text into token IDs, the units the model processes. A token might represent a word fragment, punctuation or another encoded piece. Word count is not token count.
For ordinary autoregressive text generation, prefill processes the prompt and produces scores used to select the first output token. Decode repeatedly uses the previous output to produce the next. A KV cache retains attention state for reuse as the sequence grows.
A millisecond (ms) is one thousandth of a second. The example below produces four output tokens.
Illustrative request · times measured from arrival
400 ms here: initial waiting plus prompt processing. State the measurement boundary; a client can also see network delay.
100 ms here after the first token: (700 − 400) ÷ (4 − 1). Averages can hide a long pause.
The first token arrives quickly, but the rest of the answer pauses. Which part would you inspect first?
Inspect the gaps during generation and the serving load. Faster prompt processing alone does not fix slow subsequent tokens. Throughput is a separate count of work completed per unit of time, often across many requests.
Start with a hypothetical dense 8B model. A 16-bit value uses two bytes, so its uniform weight payload is 8 billion × 2 bytes = 16 GB. A device with 16 GB available has no space left for request state or software buffers.
Predict what happens when you change the weights to 4 bits, then try it below. Eight bits make one byte. Keep the same model and memory allowance so only one choice changes.
Over budget by 4 GB. This example does not fit.
8 billion × 16 bits ÷ 8 = 16 GB of weights
16 GB + 4 GB assumed other memory = 20 GB
Illustrative dense 8B model; decimal GB throughout. The fixed 4 GB is a teaching assumption for quantization metadata, request state and runtime buffers, not a recommended allowance. Actual formats can mix precision; actual workloads need their own budget.
The lower-precision version fits the example. Have you established that it is ready to serve users?
You have passed one memory screen. Check support for the exact model and format, measure quality, and test realistic input lengths and simultaneous requests. Memory fit alone establishes neither latency nor throughput.
You can now distinguish the machine from the model, explain the two waits in a request, and check a simple memory budget. Choose the work that interests you. You do not need to study all three paths at once.
Learn inference: loading models, scheduling requests and keeping answers responsive. Begin by following how a serving engine shares time and memory.
Start the inference lesson →First lesson free · later course lessons have access labels
Learn what happens during one training step before splitting it across devices. Some matrix and Python knowledge will help with the later material.
Start the training lesson →First lesson free · later course lessons have access labels
Learn how threads execute and access memory. This path suits readers comfortable with programming who want to understand CUDA and kernel performance.
Start the GPU programming lesson →First lesson free · later course lessons have access labels
Still deciding? Continue with the Foundations course syllabus. Its lessons show their access requirements before you open them. Save this page or bookmark a step’s link to return to it.
Both references are public. Downloadable guides have separate access requirements shown on their report pages.
This introduction uses simplified examples. The linked lessons provide the deeper mechanisms and their limits.
