Secure a model endpoint with SSH tunnels and HTTPS

TL;DR: Keep the model server on loopback, use an SSH tunnel for a single operator, and test an HTTPS reverse proxy locally before extending access. This guide builds an isolated Nginx configuration with certificate verification, a small route allowlist, preserved upstream authentication and unbuffered streamed responses. It then checks unauthorized requests, oversized bodies and a dead upstream. The local lab uses a synthetic server, so it needs no GPU and makes no model-performance claim.

A model responds on the GPU host, but your laptop cannot reach it. Binding the model to every interface appears to fix the problem. It also changes who can reach the API. Before changing the listener, trace the request through the laptop, SSH connection, host port and container network. Each has its own address space and its own failure signals.

You will keep a reproducible proxy configuration and a test program that exercises real TLS and Nginx. The reference lab was run with Python 3.12, OpenSSL 3 and Nginx 1.24.0 on Linux. The directives are deliberately modest; preserve the tests when changing your Nginx release. Remote SSH forwarding and the final connection to a real GPU service remain steps for your own environment, rather than executed results from this page.

1. Prepare the tools and an isolated directory

Use Linux, or a Linux environment in WSL. You need Python 3.12, an Nginx binary with the HTTP SSL module, OpenSSL, and an OpenSSH client for the remote extension. Check them before downloading model weights or opening any ports:

python3 --version
nginx -v
nginx -V
openssl version
ssh -V

nginx -V prints build options to standard error. Look for --with-http_ssl_module. Use your distribution's supported packages if a tool is absent; the Nginx installation documentation describes the packaging choices. Installing a distribution package may start its default system service. This guide uses a separate process and configuration on high loopback ports, so you should not replace a machine's existing web-server configuration to follow it.

Create hosting-lab and download nginx-config.txt, prepare_proxy.py, upstream_fixture.py and test_proxy.py from this page into it. Keep all four together. Work in that directory for the remaining local commands.

mkdir -p hosting-lab
cd hosting-lab
python3 test_proxy.py

If your Nginx binary is outside PATH, pass its absolute path with --nginx /absolute/path/to/nginx. The test starts only the process it owns and removes its temporary TLS files afterward. It refuses to proceed if port 18080 or 18443 is already occupied. It never kills a process to free a port.

A passing run prints eight checks: accepted configuration, rejected untrusted certificate, model-list 401 and 200, blocked metrics route, first event delivered before upstream completion, rejected oversized body and failed unavailable upstream. These are actual network and proxy observations against a synthetic upstream. They do not prove that a model can load or fit in GPU memory.

If configuration checking fails, read the Nginx error printed before the Python traceback. A missing SSL module is a tool prerequisite problem. A port already in use is a local ownership problem. Resolve the named condition before continuing.

2. Run the local path yourself

The automated test is a first success. Now keep the processes visible in separate terminals so you can inspect each boundary.

In terminal A, prepare a private runtime directory:

python3 prepare_proxy.py --directory proxy-runtime

The directory must not already exist. The program creates a one-day, self-signed certificate valid for localhost and 127.0.0.1, then copies the configuration. Its private key stays inside the directory with restrictive permissions. This is a disposable lab identity, not a certificate for a public hostname.

Choose a fresh synthetic API key and start the fixture in terminal A:

export MODEL_API_KEY="$(openssl rand -hex 24)"
python3 upstream_fixture.py --port 18080

Terminal A now owns the upstream listener. Use a password manager or another local secret-transfer method to make the same key available as MODEL_API_KEY in the client terminal. Do not publish the value in a screenshot, repository or shared command transcript. The random key here has no relationship to a real provider credential.

In terminal B, from hosting-lab, check and start the independent proxy:

nginx -t -p "$PWD/proxy-runtime/" -c nginx.conf
nginx -p "$PWD/proxy-runtime/" -c nginx.conf -g 'daemon off;'

Leave it in the foreground. In terminal C, use the same working directory and the fixture key:

curl --fail --silent --show-error \
  --cacert proxy-runtime/cert.pem \
  --header "Authorization: Bearer $MODEL_API_KEY" \
  https://localhost:18443/v1/models

The response lists synthetic-fixture. Without the header it returns 401. Without --cacert, normal certificate validation rejects this self-signed lab identity unless you previously added it to your trust store. The exercise adds trust for this command only; it does not disable identity checking.

To watch streaming, send a small request and ask curl to avoid buffering its displayed output:

curl --fail --silent --show-error --no-buffer \
  --cacert proxy-runtime/cert.pem \
  --header "Authorization: Bearer $MODEL_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{"model":"synthetic-fixture","stream":true,"messages":[{"role":"user","content":"READY"}]}' \
  https://localhost:18443/v1/chat/completions

You should see a READY data event before the stop event and [DONE]. The fixture waits between these writes to make delivery visible. For machine-readable completion and timing evidence, use the client from Stream vLLM responses with Python, passing this URL and --ca-file proxy-runtime/cert.pem.

3. Trace the HTTPS request

TLS protects the client-to-proxy connection and lets the client verify the proxy's identity. The API key answers a different question: whether the upstream accepts this caller. Both are exercised independently in the tests. A successful TLS handshake with a 401 response is therefore a useful intermediate result, not an inexplicable contradiction.

The upstream connection in this configuration uses host loopback. If you later move the model to another machine, that assumption is gone. Configure and verify protection for the new proxy-to-model path instead of retaining plaintext HTTP merely because the front door uses HTTPS.

Only /v1/models and /v1/chat/completions are forwarded. Other paths, including /metrics, return 404 at the proxy. This matters because a serving framework's API key does not necessarily protect every operational endpoint. Read the selected release's vLLM security guidance before extending the route list. Do not expose administrative, dynamic loading or debugging routes as a side effect of a catch-all proxy rule.

The fixture's authorization check stands in for the serving endpoint's check. Nginx forwards the Authorization header unchanged. This lab does not implement tenant identity, individual key revocation or billing, and a shared key should not be described as any of those features.

4. Keep streaming behavior through the proxy

The important response directive is proxy_buffering off. Nginx can then forward response bytes as it receives them rather than waiting to accumulate a larger response buffer. gzip off avoids another transformation in this lab. See the Nginx proxy module reference.

The test proves more than eventual response equality. Its upstream sends one event, then waits for the test to release it. The client must receive that first event while upstream completion is still pending. Only then does the test let the upstream send the remainder. Re-enabling response buffering makes this test fail under the supplied fixture.

proxy_read_timeout 5s is an inactivity limit between upstream reads. It is not a five-second maximum generation duration. A stream that produces bytes regularly can run longer. Choose timeouts from the request contract and enforce a total request deadline separately where needed. A model doing a long prefill may produce no bytes for a while; blindly shortening this timeout can turn legitimate slow starts into interrupted streams.

The configuration also sets proxy_next_upstream off. A hidden retry of an inference POST can create duplicate work, and after response bytes have reached a user it cannot reconstruct the original stream. If you later add several upstream replicas, define retry eligibility and cancellation explicitly rather than assuming ordinary web-page retry behavior fits generation.

Request buffering remains at its normal setting. The request in this exercise is a small, bounded JSON document; the response is the stream. Do not confuse proxy_request_buffering with proxy_buffering when debugging delivery.

5. Understand the two loopbacks in an SSH tunnel

For one operator connecting to a GPU host, an SSH tunnel is usually the simpler first path. Keep the model's host listener private and create a local port on the laptop:

ssh -N \
  -L 127.0.0.1:18000:127.0.0.1:8000 \
  -o ExitOnForwardFailure=yes \
  -o ServerAliveInterval=30 \
  -o ServerAliveCountMax=3 \
  [email protected]

Replace gpu-user and gpu-host.example with the account and host you control. Establish and verify the SSH host identity through your provider or administrator before relying on the tunnel. Keep normal host-key checking enabled.

The first 127.0.0.1 is the laptop's listener. The second is resolved from the remote end of the SSH connection: it means the GPU host's loopback. Requests to http://127.0.0.1:18000 on the laptop reach port 8000 on the GPU host. The OpenSSH forwarding manual defines this boundary.

Leave SSH running in the foreground and make a real authenticated request through port 18000. ExitOnForwardFailure catches failure to establish the requested forwarding listener; it does not certify that the destination model is healthy. The SSH configuration reference makes that distinction explicit.

For the existing adapter deployment, change only the remote destination port to 8001 and use incident-labeler in the request. You can also forward the proxy's remote port 18443 to a local port, but then the client's TLS hostname must still match the certificate. This lab certificate is only valid for localhost and expires after one day; it is unsuitable as an ongoing remote-service identity.

6. Connect the proxy to the actual model

Complete the Qwen deployment guide first, including a successful authenticated smoke request on the GPU host. Stop the synthetic fixture when you switch to a model; a fixture response must never become a false readiness result for a GPU service.

Create a new copy of the lab runtime for the model experiment. In its nginx.conf, change both upstream destinations from 127.0.0.1:18080 to the existing model listener, 127.0.0.1:8000, or to port 8001 for the adapter. Keep the same API key required by the model server in the client environment. The proxy should forward it; it should not replace it with the fixture key.

Check the new configuration with nginx -t, then start that named runtime in its own foreground process. Verify all of these observations again: unauthenticated model listing fails, authenticated listing returns the actual model identifier, a complete generation succeeds, and an unknown route stays closed. The synthetic test cannot establish the engine's actual authorization behavior.

If the model runs in Docker, distinguish the host mapping from the listener inside the container. The companion guide publishes 127.0.0.1:8000:8000 on the host while vLLM listens on 0.0.0.0:8000 inside the container. Binding only the container's loopback can make the published mapping unusable. Publishing without a host address can expose the port more widely than intended. Consult Docker's port-publishing documentation for the installed Engine release and network mode, then verify reachability from a separate machine rather than trusting the command's appearance alone.

On a Linux host, inspect the listener and Docker's recorded mapping:

ss -ltn '( sport = :8000 )'
docker port hands-on-qwen

Neither command proves model readiness. Follow it with the authenticated application request. If a host firewall or cloud network rule is involved, scope changes to the exact host, port and intended client network; keep the separate SSH management path reachable. This project does not require opening port 8000 to the Internet.

7. Set limits with the right meaning

The lab limits request bodies to 16 KiB, permits four concurrent requests per client-address key and configures a request-rate bucket at 10 requests per second with a small burst. These are test settings. They are not a capacity recommendation for your GPU.

The request-rate module and connection-limit module control different populations. A short request-rate burst can leave several long streams active. Conversely, a concurrency limit does not tell you how many prompt tokens each request carries. Tune both alongside server-side context, output and workload limits.

Every user behind the same NAT can share one apparent source address. An IP bucket is therefore not a per-tenant quota. If a trusted load balancer is added, configure the real-client-address trust boundary carefully; do not accept an arbitrary client-supplied forwarding header as identity. This configuration overwrites the forwarded address with the directly connected peer.

For a hypothetical workload, four active streams lasting 20 seconds imply approximately 4 / 20 = 0.2 completions per second at steady state if all four slots stay occupied. A 10-request-per-second arrival setting does not create capacity for that workload. Measure the service's quality and latency under the intended request distribution before widening access.

8. Troubleshoot one boundary at a time

The local access log contains status and timing fields. It intentionally omits request bodies, query strings and authorization values. Preserve that restraint when adding diagnostics; a model prompt can contain sensitive data.

9. Check your understanding and clean up

The tunnel is healthy and a request receives HTTP 401. Should you open another firewall port? No. You reached an application that rejected the credential. Trace authentication next.

A stream sends data every four seconds and lasts a minute. Must the proxy's five-second read timeout stop it? No. That timeout measures inactivity between upstream reads. A separate total-duration policy is needed if a minute is too long.

Keep the configuration, version outputs and test results. Stop the foreground Nginx and fixture processes with Ctrl+C, and close the SSH tunnel separately. Remove the disposable certificate directory only after stopping the process that uses it. Do not use a system-wide killall nginx or reload an unrelated site's configuration.

Before serving multiple external clients, replace the lab certificate with a correctly issued and renewed certificate for your hostname, define credential lifecycle and caller identity, apply network restrictions, and test workload limits and monitoring. The self-hosting field guide covers the broader operational decisions; the hands-on project here establishes the concrete local path and its failure checks.

Complete project files

nginx-config.txt

worker_processes 1;
pid nginx.pid;
error_log error.log warn;
events { worker_connections 256; }
http {
    log_format lab '$status $request_time $upstream_status';
    access_log access.log lab;
    client_body_temp_path body;
    proxy_temp_path proxy;
    fastcgi_temp_path fastcgi;
    uwsgi_temp_path uwsgi;
    scgi_temp_path scgi;
    limit_req_zone $binary_remote_addr zone=lab_rate:1m rate=10r/s;
    limit_conn_zone $binary_remote_addr zone=lab_connections:1m;
    server {
        listen 127.0.0.1:18443 ssl;
        server_name localhost;
        ssl_certificate cert.pem;
        ssl_certificate_key key.pem;
        ssl_protocols TLSv1.2 TLSv1.3;
        client_max_body_size 16k;
        client_body_timeout 10s;
        send_timeout 10s;
        keepalive_timeout 10s;
        gzip off;
        limit_req zone=lab_rate burst=5 nodelay;
        limit_req_status 429;
        limit_conn lab_connections 4;
        limit_conn_status 429;
        proxy_connect_timeout 2s;
        proxy_read_timeout 5s;
        proxy_send_timeout 5s;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Authorization $http_authorization;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto https;
        proxy_buffering off;
        proxy_cache off;
        proxy_next_upstream off;
        location = /v1/chat/completions {
            if ($request_method != POST) { return 405; }
            proxy_pass http://127.0.0.1:18080;
        }
        location = /v1/models {
            if ($request_method != GET) { return 405; }
            proxy_pass http://127.0.0.1:18080;
        }
        location / { return 404; }
    }
}

prepare_proxy.py

"""Prepare an isolated localhost TLS lab. Does not edit system Nginx or firewalls."""
import argparse
import os
from pathlib import Path
import shutil
import subprocess


def prepare(target):
    target = Path(target).resolve()
    target.mkdir(mode=0o700, parents=True, exist_ok=False)
    for name in ('body', 'proxy', 'fastcgi', 'uwsgi', 'scgi'):
        (target / name).mkdir(mode=0o700)
    shutil.copyfile(Path(__file__).with_name('nginx-config.txt'), target / 'nginx.conf')
    old = os.umask(0o077)
    try:
        subprocess.run(['openssl', 'req', '-x509', '-newkey', 'rsa:2048', '-nodes',
                        '-days', '1', '-subj', '/CN=localhost',
                        '-addext', 'subjectAltName=DNS:localhost,IP:127.0.0.1',
                        '-keyout', str(target/'key.pem'), '-out', str(target/'cert.pem')],
                       check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    finally:
        os.umask(old)
    return target


if __name__ == '__main__':
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument('--directory', default='proxy-runtime')
    a = p.parse_args()
    print('Prepared local TLS files in ' + str(prepare(a.directory)))

upstream_fixture.py

"""Authenticated synthetic SSE upstream, not an inference engine."""
import argparse
import hmac
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import os
import threading
import time


class Handler(BaseHTTPRequestHandler):
    protocol_version = 'HTTP/1.1'

    def log_message(self, *_):
        pass

    def answer(self, status, body):
        body = json.dumps(body).encode()
        self.send_response(status)
        self.send_header('Content-Type', 'application/json')
        self.send_header('Content-Length', str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def authorized(self):
        if not hmac.compare_digest(self.headers.get('Authorization', ''), 'Bearer ' + self.server.token):
            self.answer(401, {'error': 'Unauthorized'})
            return False
        return True

    def do_GET(self):
        if not self.authorized():
            return
        self.answer(200 if self.path == '/v1/models' else 404,
                    {'data': [{'id': 'synthetic-fixture'}]} if self.path == '/v1/models' else {})

    def do_POST(self):
        size = int(self.headers.get('Content-Length', 0))
        if not 0 < size <= 16384:
            self.close_connection = True
            self.answer(413, {'error': 'Body limit'})
            return
        request = json.loads(self.rfile.read(size))
        if not self.authorized():
            return
        if self.path != '/v1/chat/completions' or request.get('stream') is not True:
            self.answer(400, {'error': 'Use the streaming chat route'})
            return
        first = b'data: {"choices":[{"index":0,"delta":{"content":"READY"},"finish_reason":null}]}\n\n'
        end = b'data: {"choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}\n\ndata: [DONE]\n\n'
        self.send_response(200)
        self.send_header('Content-Type', 'text/event-stream')
        self.send_header('Content-Length', str(len(first)+len(end)))
        self.end_headers()
        try:
            self.wfile.write(first)
            self.wfile.flush()
            if self.server.release is None:
                time.sleep(0.4)
            else:
                self.server.release.wait(3)
            self.server.finished.set()
            self.wfile.write(end)
            self.wfile.flush()
        except (BrokenPipeError, ConnectionResetError):
            pass


def server(token, port=18080, release=None):
    if not token:
        raise ValueError('Set a nonempty MODEL_API_KEY')
    instance = ThreadingHTTPServer(('127.0.0.1', port), Handler)
    instance.token, instance.release = token, release
    instance.finished = threading.Event()
    return instance


if __name__ == '__main__':
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument('--port', type=int, default=18080)
    a = p.parse_args()
    with server(os.environ['MODEL_API_KEY'], a.port) as instance:
        print('Synthetic authenticated upstream on 127.0.0.1:' + str(instance.server_port), flush=True)
        try:
            instance.serve_forever()
        except KeyboardInterrupt:
            pass

test_proxy.py

"""Run actual Nginx/TLS against an authenticated synthetic upstream."""
import argparse
import http.client
import json
from pathlib import Path
import secrets
import socket
import ssl
import subprocess
import tempfile
import threading
import time
from prepare_proxy import prepare
from upstream_fixture import server


def run(nginx):
    checks = []
    token = secrets.token_hex(24)
    release = threading.Event()
    # Fixed lab ports deliberately fail if in use; never stop another listener.
    for port in (18080, 18443):
        with socket.socket() as sock:
            sock.bind(('127.0.0.1', port))
    with tempfile.TemporaryDirectory() as tmp:
        runtime = prepare(Path(tmp)/'runtime')
        prefix = str(runtime) + '/'
        subprocess.run([nginx, '-t', '-p', prefix, '-c', 'nginx.conf'], check=True,
                       stdout=subprocess.DEVNULL)
        checks.append('nginx configuration accepted')
        proxy = subprocess.Popen([nginx, '-p', prefix, '-c', 'nginx.conf', '-g', 'daemon off;'],
                                 stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
        try:
            deadline = time.monotonic() + 5
            while True:
                try:
                    with socket.create_connection(('127.0.0.1',18443),timeout=0.2):
                        break
                except OSError:
                    if time.monotonic() > deadline or proxy.poll() is not None:
                        raise RuntimeError('Proxy did not start')
                    time.sleep(0.02)
            context = ssl.create_default_context(cafile=str(runtime/'cert.pem'))
            def request(path, key=None, body=None):
                conn = http.client.HTTPSConnection('localhost',18443,context=context,timeout=5)
                headers = {'Content-Type':'application/json'}
                if key:
                    headers['Authorization']='Bearer '+key
                conn.request('POST' if body is not None else 'GET',path,body=body,headers=headers)
                response=conn.getresponse()
                return conn,response
            try:
                conn=http.client.HTTPSConnection('localhost',18443,timeout=2)
                try:
                    conn.request('GET','/v1/models')
                    raise AssertionError('Untrusted certificate was accepted')
                finally:
                    conn.close()
            except ssl.SSLCertVerificationError:
                checks.append('untrusted TLS certificate rejected')
            with server(token,release=release) as upstream:
                worker=threading.Thread(target=upstream.serve_forever,daemon=True);worker.start()
                try:
                    for path,key,status in [('/v1/models',None,401),('/v1/models',token,200),('/metrics',token,404)]:
                        conn,response=request(path,key)
                        assert response.status==status,(path,response.status)
                        response.read();conn.close()
                        checks.append(f'{path} returned {status}')
                    body=json.dumps({'model':'synthetic-fixture','stream':True,'messages':[{'role':'user','content':'READY'}]})
                    conn,response=request('/v1/chat/completions',token,body)
                    assert response.status==200
                    first=response.readline()
                    assert first.startswith(b'data: ')
                    assert not upstream.finished.is_set(),'Proxy buffered until completion'
                    release.set()
                    assert b'[DONE]' in response.read()
                    conn.close();checks.append('first event delivered before upstream completion')
                    conn,response=request('/v1/chat/completions',token,'x'*17000)
                    assert response.status==413,response.status
                    response.read();conn.close();checks.append('oversized body rejected')
                finally:
                    release.set();upstream.shutdown();worker.join()
            conn,response=request('/v1/models',token)
            assert response.status==502,response.status
            response.read();conn.close();checks.append('unavailable upstream returns failure')
        finally:
            proxy.terminate()
            try:
                proxy.wait(timeout=5)
            except subprocess.TimeoutExpired:
                proxy.kill();proxy.wait()
        return checks


if __name__ == '__main__':
    p=argparse.ArgumentParser(description=__doc__)
    p.add_argument('--nginx',default='nginx')
    a=p.parse_args()
    print(json.dumps({'scope':'Real local TLS and Nginx; synthetic upstream, no model',
                      'checks':run(a.nginx)},indent=2))