Deploy Qwen3.5-4B with vLLM on an NVIDIA L4

TL;DR: Serve the pinned Qwen3.5-4B checkpoint with vLLM 0.29.0, a 4,096-token context cap and at most four sequences. Bind the host port to loopback, send a real chat request and record the container digest. This is a source-reviewed NVIDIA L4 starting configuration; its GPU execution and memory fit have not yet been measured here.

1. Choose the deployment and its limits

You will create an HTTP endpoint that accepts a conversation and returns a generated answer. The serving engine, vLLM, loads the model and schedules its work on the GPU. The endpoint follows a familiar chat-completions request format; an application does not need to load model weights itself.

This guide uses Qwen/Qwen3.5-4B, a specific checkpoint from the 2026 Qwen3.5 family. Its model card describes a language model with a vision encoder and a hybrid attention design. We disable media input for this first project. The recipe does not demonstrate image understanding, long-context quality or the family's published benchmark results.

Use a Linux x86-64 host with one NVIDIA L4, which NVIDIA lists with 24 GB of GPU memory. BF16 stores each weight in a 16-bit floating-point format. Language weights alone suggest a rough 4 billion × 2 bytes ≈ 8 GB payload. The actual model package also includes other tensors, and runtime allocations include recurrent/attention state, workspaces and framework overhead. This calculation cannot prove that a particular service configuration fits. See model memory footprint before increasing context or concurrency.

The initial 4,096-token cap is a deliberate short-request experiment. It is far below the model's advertised context capacity and should not be used to reproduce long-thinking benchmark claims. The NVIDIA L4 specifications describe the device, not measured results for this model.

2. Verify the host before fetching weights

Docker and the NVIDIA Container Toolkit must already be configured. Follow NVIDIA's installation instructions for your distribution if needed. Driver and container-runtime installation can affect other workloads; use a dedicated learning host for a first attempt.

nvidia-smi --query-gpu=name,memory.total,memory.free,driver_version --format=csv
docker version

Check that the reported GPU is the one you intended to use and that another process is not consuming the planned memory. The presence of the nvidia-smi executable is insufficient: it must successfully talk to the driver.

You also need enough host RAM and disk for the container, downloaded model files and logs. Reserve tens of GB of disk rather than just the approximate weight payload. Download time depends on the network and cache; the session estimate above does not promise a startup duration or cloud cost.

3. Pin the engine and record its identity

Create a working directory, save serve.sh and smoke.py from Project files inside it, then pull the selected version:

mkdir qwen-service
cd qwen-service
docker pull vllm/vllm-openai:v0.29.0
export VLLM_IMAGE="$(docker image inspect vllm/vllm-openai:v0.29.0 --format '{{index .RepoDigests 0}}')"
printf '%s\n' "$VLLM_IMAGE" > image-digest.txt
docker run --rm --gpus all --entrypoint nvidia-smi "$VLLM_IMAGE"

The last command checks GPU visibility inside this container. Stop if it fails. A functioning host driver does not prove that the container runtime is passing the GPU through correctly.

A version tag identifies a release name; the resolved digest identifies the pulled image content. The start script requires the digest form so restarting the experiment cannot silently pick up a changed tag. Keep image-digest.txt with the model revision and launch script. The vLLM Docker documentation describes the serving-image interface.

4. Start a bounded local service

Generate a project API key in the current terminal. Keep it out of shared logs and source control:

export VLLM_API_KEY="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
bash serve.sh
docker logs --follow hands-on-qwen

The complete serve.sh listing is in the source section below. Run the downloaded script as a file so the exact arguments are retained with your experiment.

The model and tokenizer both use revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. The cache mount retains those files across container restarts. --served-model-name gives the API a stable experiment name, while the release record retains the exact underlying model.

Several arguments deserve explanation:

These are reviewed against the vLLM 0.29.0 server arguments. A memory fraction is not an operating-system isolation boundary and not a reservation for the cache alone. Start with exclusive use of the GPU. vLLM server arguments covers the budgeting decisions in more detail.

The container listens on all of its own interfaces, but Docker publishes it only at 127.0.0.1:8000 on the host. Do not replace that host binding with a public address merely to make a laptop request work. The engine API key does not turn every operational endpoint into a hardened public service.

5. Prove a response completes

Use a second terminal on the GPU host. Export the same API key there using your shell's normal secret-handling method. curl should first reach the model listing:

curl --fail --silent --show-error --max-time 10 \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  http://127.0.0.1:8000/v1/models
python3 smoke.py > smoke-result.json
python3 -m json.tool smoke-result.json

The smoke.py source below shows the request body and acceptance check. Run it from the folder containing the downloaded file.

The smoke script supplies enable_thinking=False in the chat-template parameters, caps generation at 32 tokens and checks that a completion is nonempty, reports output tokens and terminates with stop. This is stronger than seeing a listening port or HTTP 200 from a health endpoint. It still does not establish quality or capacity.

There is no measured sample answer in this guide because the GPU run is pending. After your run, save the actual JSON output beside image-digest.txt. A length finish reason means the output limit stopped generation; do not count that as an ordinary completed answer for this check.

From a laptop, an SSH tunnel can carry requests to the host's loopback port:

ssh -N -L 8000:127.0.0.1:8000 YOUR_GPU_HOST

Replace YOUR_GPU_HOST with the host alias you already use. Run this in a laptop terminal, keep it open, and run the client from another laptop terminal with the same API key. If local port 8000 is occupied, choose another local tunnel port and adjust the client URL deliberately. Closing the tunnel does not stop the GPU server.

6. Diagnose the first failing stage

Read the startup logs from the beginning and identify the last stage that completed. A model-download error, an unsupported architecture and a cache allocation failure have different remedies.

The configuration is intentionally modest, but it is not an L4 fit guarantee. Preserve a failure log and change one variable at a time. The startup OOM walkthrough explains why blindly raising the memory fraction can make another stage fail.

7. Record the release, then stop the resource

Save the GPU name and driver version, image digest, both model revisions, serve.sh, the smoke result and the actual peak memory observed during your request. These describe what ran. A note containing only “Qwen on vLLM” leaves too many moving parts unidentified. Model onboarding connects those identities to an evaluation record.

To stop this experiment:

docker stop hands-on-qwen
docker rm hands-on-qwen
unset VLLM_API_KEY

The cache directory remains so you can repeat the run. Inspect its size before removing it; deleting cached weights changes the next startup's download behavior. A stopped container does not terminate a rented GPU machine. Save your artifacts, release the machine through its provider and inspect retained disk resources separately.

Before serving other users, add a managed authentication boundary, TLS, request limits and monitoring appropriate to that environment. First follow the release benchmark guide, which requires Premium, or read the free serving benchmark concepts to choose what to measure. A completed local response establishes one boundary in that process. The recipe remains GPU-pending until that response has actually been observed on the named hardware.

Check your understanding

The model listing returns HTTP 200. Has generation been verified? No. A chat request must complete with content and the expected finish reason. A ready HTTP process can still fail when the first model request executes.

Why can a laptop fail to reach port 8000 even when the host's smoke test passes? The host publishes that port on loopback. Use the documented SSH tunnel to reach it; changing the container's internal listen address does not change the host-side binding.

Complete project source

These are the same files offered under Project files. Run the downloaded files; the listings let you inspect each implementation.

serve.sh

#!/usr/bin/env bash
set -euo pipefail
: "${VLLM_API_KEY:?Set VLLM_API_KEY before starting the server}"
: "${VLLM_IMAGE:?Set VLLM_IMAGE to the digest recorded after docker pull}"
case "$VLLM_IMAGE" in
  vllm/vllm-openai@sha256:*) ;;
  *) echo 'Use the resolved vllm/vllm-openai image digest.' >&2; exit 1 ;;
esac
mkdir -p model-cache
# Bind only to the host's loopback interface. The container listens internally.
docker run --detach --name hands-on-qwen --gpus all --shm-size 2g \
  --publish 127.0.0.1:8000:8000 \
  --volume "$PWD/model-cache:/root/.cache/huggingface" \
  --env VLLM_API_KEY --env HF_HUB_DISABLE_TELEMETRY=1 \
  --entrypoint vllm "$VLLM_IMAGE" serve Qwen/Qwen3.5-4B \
  --revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a \
  --tokenizer-revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a \
  --served-model-name hands-on-qwen --host 0.0.0.0 --port 8000 \
  --dtype bfloat16 --max-model-len 4096 --max-num-seqs 4 \
  --gpu-memory-utilization 0.8 --language-model-only \
  --reasoning-parser qwen3 --generation-config vllm

smoke.py

"""Send one bounded non-streaming request; fail if the API contract is incomplete."""
import json
import os
import urllib.request

body = {'model': 'hands-on-qwen', 'messages': [{'role': 'user', 'content': 'Reply with the word ready.'}],
        'max_tokens': 32, 'temperature': 0, 'chat_template_kwargs': {'enable_thinking': False}}
request = urllib.request.Request('http://127.0.0.1:8000/v1/chat/completions',
    data=json.dumps(body).encode(), headers={'Content-Type': 'application/json',
                                            'Authorization': 'Bearer ' + os.environ['VLLM_API_KEY']})
with urllib.request.urlopen(request, timeout=120) as response:
    result = json.load(response)
choice = result['choices'][0]
assert choice['finish_reason'] == 'stop', result
assert isinstance(choice['message']['content'], str) and choice['message']['content'].strip(), result
assert result['usage']['completion_tokens'] > 0, result
print(json.dumps({'answer': choice['message']['content'], 'finish_reason': choice['finish_reason'],
                  'usage': result['usage']}, indent=2))