How to Serve LLMs Locally with vLLM and uv

uv Scientific Python verified with uv 0.13

vLLM serves large language models on Linux with NVIDIA CUDA. Separate builds support AMD ROCm, Intel XPU, and CPU execution; Apple Silicon GPU execution uses the vLLM-Metal package. Windows users need WSL 2 with GPU access. This guide targets Linux NVIDIA GPUs with compute capability 7.5 or newer, such as a T4, using Python 3.12. vLLM’s compiled extensions and PyTorch must use compatible CUDA and PyTorch builds.

Install vLLM for quick experimentation

Create a virtual environment and install vLLM in one pass:

uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

vLLM does not publish Windows wheels. Install WSL 2 and run the Linux instructions inside it:

uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

The --torch-backend flag selects an index for PyTorch packages. auto queries the visible driver and accelerators; without a detected accelerator it selects CPU wheels. It does not select a different vLLM wheel. Use an explicit deployment backend for GPU-less image builds, then test the image on a GPU.

The default CUDA build changes between releases. Check the GPU installation instructions and release assets for the chosen vLLM version. Keep vLLM’s declared PyTorch dependencies; replacing them with an arbitrary newer PyTorch or another CUDA build can break compiled extensions even when installation succeeds.

Important

For a CUDA 12.9 deployment on Linux x86-64 with glibc 2.34 or newer, this release-matched example selects both vLLM and PyTorch builds:

uv pip install \
  "https://github.com/vllm-project/vllm/releases/download/v0.21.0/vllm-0.21.0+cu129-cp38-abi3-manylinux_2_34_x86_64.whl" \
  --torch-backend=cu129

The command installs a pinned vLLM release rather than tracking the latest release. Confirm driver and GPU support for that CUDA family; the CUDA field in nvidia-smi is a driver capability, not the installed toolkit version. A +cpu wheel is available on the same release page for CPU deployments.

For that pinned CUDA 12.9 install on a host without a CUDA toolkit, install the matching FlashInfer kernel cache before serving a model:

uv pip install "flashinfer-jit-cache==0.6.8.post1" --index https://flashinfer.ai/whl/cu129

This installs the cache matching its FlashInfer dependency and CUDA build. Without cached kernels, FlashInfer’s attention or sampler can invoke nvcc during startup even though vLLM installed from a wheel. Other releases need their own matching cache; kernels absent from the cache need a compatible CUDA toolkit and C++ compiler. See the FlashInfer installation requirements.

Run without a persistent environment

On Linux x86-64 with glibc 2.34 or newer, use the same pinned CUDA 12.9 configuration in a temporary venv. Install both vLLM and its matching FlashInfer cache into this new environment before serving:

uv venv --python 3.12 --seed --managed-python /tmp/vllm-env
source /tmp/vllm-env/bin/activate
uv pip install \
  "https://github.com/vllm-project/vllm/releases/download/v0.21.0/vllm-0.21.0+cu129-cp38-abi3-manylinux_2_34_x86_64.whl" \
  --torch-backend=cu129
uv pip install "flashinfer-jit-cache==0.6.8.post1" --index https://flashinfer.ai/whl/cu129
vllm serve Qwen/Qwen2.5-1.5B-Instruct

Note

--torch-backend only works with uv pip commands, not with uv run or uv sync. For one-off use, create a throwaway venv as shown here. Delete it when you’re done with rm -rf /tmp/vllm-env.

Configure a vLLM project with pyproject.toml

For reproducible deployments, configure CUDA index routing in your project file. This ensures every team member and CI runner installs the same GPU-enabled build:

pyproject.toml

[project]
name = "my-inference-service"
version = "0.1.0"
requires-python = ">=3.12,<3.15"
dependencies = [
    "vllm==0.21.0",
    "torch==2.11.0",
    "torchvision==0.26.0",
    "torchaudio==2.11.0",
]

[tool.uv]
environments = ["sys_platform == 'linux' and platform_machine == 'x86_64'"]

[[tool.uv.index]]
name = "pytorch-cu130"
url = "https://download.pytorch.org/whl/cu130"
explicit = true

[tool.uv.sources]
torch = [
  { index = "pytorch-cu130", marker = "sys_platform == 'linux'" },
]
torchvision = [
  { index = "pytorch-cu130", marker = "sys_platform == 'linux'" },
]
torchaudio = [
  { index = "pytorch-cu130", marker = "sys_platform == 'linux'" },
]

Then lock and sync:

uv lock
uv sync

The lockfile records the selected vLLM and PyTorch versions. The torch packages are direct dependencies so uv applies their source mappings. explicit = true keeps unrelated packages on PyPI. This example pairs a pinned vLLM release with its CUDA 13.0 PyTorch build on Linux; a platform marker does not make the vLLM project portable to macOS or Windows.

Without a matching CUDA 13.0 FlashInfer kernel cache, this configuration needs a CUDA 13.0 toolkit with nvcc and a supported C++ compiler for runtime kernel compilation. Installing PyTorch’s runtime libraries alone does not supply that toolchain. A CUDA 12.9 cache does not serve this CUDA 13.0 environment.

For CUDA 12.9, change the named PyTorch index and all three source references to pytorch-cu129, and set its URL to https://download.pytorch.org/whl/cu129. Add flashinfer-jit-cache==0.6.8.post1 to [project].dependencies, then add these entries to the existing [tool.uv.sources] table:

pyproject.toml

vllm = { url = "https://github.com/vllm-project/vllm/releases/download/v0.21.0/vllm-0.21.0+cu129-cp38-abi3-manylinux_2_34_x86_64.whl" }
flashinfer-jit-cache = { index = "flashinfer-cu129" }

Add its explicit cache index before locking:

pyproject.toml

[[tool.uv.index]]
name = "flashinfer-cu129"
url = "https://flashinfer.ai/whl/cu129"
explicit = true

The cache must be a direct dependency for uv to apply its source mapping. Changing only the PyTorch index leaves the default vLLM binary selected. Re-run uv lock and uv sync after changing these dependencies and sources. For other releases, choose matching vLLM and FlashInfer builds rather than reusing these pins.

Verify accelerator execution

uv run python -c "import torch, vllm; import vllm._C; assert torch.cuda.is_available(), 'CUDA unavailable'; x = torch.ones(2, device='cuda'); assert (x + x).cpu().tolist() == [2.0, 2.0]; print(vllm.__version__, torch.__version__, torch.version.cuda, torch.cuda.get_device_name(0))"

uv run executes the command inside the project’s virtual environment, ensuring all dependencies are installed first. The command must load vLLM’s compiled extension, complete a CUDA operation, and print versions plus the GPU name. This checks the environment; serving a model is the additional check that exercises vLLM’s inference kernels.

Serve a model with the OpenAI-compatible API

vLLM includes a server that exposes OpenAI-compatible endpoints. Start it with:

uv run vllm serve Qwen/Qwen2.5-1.5B-Instruct

Wait for the server’s application-startup message. Model download, compilation, and memory allocation must succeed before it accepts requests. Then query it with an OpenAI-compatible client:

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "prompt": "The capital of France is",
    "max_tokens": 20
  }'

The response contains generated text in choices. Successful installation or import vllm alone does not establish that this request runs on a GPU.

Run offline batch inference

For processing prompts without a running server, use vLLM’s Python API directly. Instruct-tuned models expect chat-formatted input, so use llm.chat() instead of llm.generate() (which passes raw text without applying the model’s chat template):

batch_inference.py

from vllm import LLM, SamplingParams


def main():
    llm = LLM(model="Qwen/Qwen2.5-1.5B-Instruct")
    params = SamplingParams(temperature=0.7, max_tokens=100)

    conversations = [
        [{"role": "user", "content": "Explain virtual environments in one sentence."}],
        [{"role": "user", "content": "What does uv pip install do?"}],
    ]

    outputs = llm.chat(conversations, params)
    for output in outputs:
        print(output.outputs[0].text)


if __name__ == "__main__":
    main()

The main guard prevents spawned workers from starting another engine when they import the script. Run it with:

uv run batch_inference.py

Diagnose common errors

“No CUDA GPUs are available.” Check the host driver, container GPU passthrough, and CUDA_VISIBLE_DEVICES. nvidia-smi and the CUDA operation check must both succeed inside the execution environment.
ImportError: libcudart.so. Reinstall a matching vLLM/PyTorch/CUDA combination in a fresh environment. Adding a runtime library alone does not fix a PyTorch ABI mismatch or an unsupported driver.
Could not find nvcc. FlashInfer is compiling a kernel that is absent from its installed cache. Install the matching FlashInfer cache or provide a compatible toolkit and C++ compiler; disabling the FlashInfer sampler alone does not disable its attention backend.
Out of memory on large models. Choose a smaller model or reduce --max-model-len to limit the context and KV cache. The model weights must fit alongside the runtime’s memory allocations.

Last updated on October 23, 2026

Please submit corrections and feedback...