vLLM

Place the API behind controlled ingress. vLLM should not become an unauthenticated shared compute endpoint because a port is reachable. Add authentication at the appropriate boundary, enumerate caller applications, use network rules, and set limits that protect shared accelerator capacity. Log request metadata according to policy without recording sensitive prompts by default.

Map the full request route: user or service, application gateway, identity or token check, vLLM endpoint, model artifacts, and accelerator host. Each step has a failure mode. A 500 response may be a caller time-out, an ingress limit, a model-server queue, an out-of-memory condition, or a failed dependency. The runbook should identify the signals that separate them.

Model artifacts need controlled storage and introduction. Verify their source and hashes where your process requires it, set file permissions, and prevent unreviewed model changes on a shared host. The model is executable operational input: it affects memory requirements, license terms, output behaviour, and potential attack surface.

Benchmark target models on production-class hardware. Measure time to first token, complete response time, throughput, queue delay, failure rate, accelerator memory, and behaviour under parallel load. Run at expected context and output lengths. Keep a little headroom for operating-system work, monitoring, fragmentation, and a burst that arrives when a dependent service retries.

Set request controls near the public application boundary and, where suitable, at the serving boundary. Limit concurrent work, prompt size, generation length, and per-caller rate according to the product need. A single unbounded request can monopolise accelerator memory or capacity. Return a clear retryable or non-retryable error so clients do not magnify an overload.

Quality acceptance deserves a separate test from throughput. Use a safe, representative prompt set, record expected behaviours and unacceptable failures, and compare candidate model or runtime changes before rollout. Inference software can serve a request correctly at the protocol layer while producing output that no longer meets the application need.

Monitor endpoint availability, caller error patterns, queue depth, latency, accelerator health, memory, disk for artifacts, and host logs. Connect each alert to a response: reduce traffic, fail over where a reviewed alternative exists, drain a node, or involve the host owner. Capacity graphs without ownership turn into passive observation.

Plan releases across the model, vLLM version, CUDA or driver stack, image, and serving parameters. Change one studied boundary at a time when possible. Pin what passed validation, hold a previous known-good release, and test both API compatibility and representative application behaviour before accepting a change.

Recovery should describe how to rebuild the endpoint with approved images, artifacts, configuration, secret references, ingress controls, and monitoring. Generated model output may not be part of recovery, but the service configuration is. Test the rebuild away from production and state the capacity or data limitations that remain.