How to Install llama-cpp-python

Install llama-cpp-python from an official wheel index to avoid compiling it. A bare pip install llama-cpp-python builds from source; GPU support depends on the selected backend, with Metal enabled by default on macOS.

Choose a backend

Prerequisites

Install uv using the uv installation guide, then create a virtual environment in an empty directory. --seed includes pip for the alternative commands.

uv venv --python 3.12 --seed

Expect Creating virtual environment with seed packages at: .venv. Activate the environment for your shell before continuing, then choose one backend.

Install from the CPU index

uv pip install llama-cpp-python --only-binary=llama-cpp-python --force-reinstall --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu

Expect an Installed summary including llama-cpp-python. For pip, replace uv pip install with pip install in the wheel and source commands; expect Successfully installed.

--only-binary=llama-cpp-python makes a missing wheel fail instead of silently compiling. --force-reinstall replaces an existing backend. If no wheel matches, build from source. On macOS, general wheels and default source builds include Metal; for damaged archives, use the Metal source fallback.

Install CUDA for an NVIDIA GPU

Choose an index from the official CUDA wheel matrix. Compute capability identifies the GPU architecture; look up the model in NVIDIA’s GPU tables.

Current Linux CUDA wheels require glibc 2.35 or newer, as recorded in the published wheel tags.

Install a compatible NVIDIA driver and matching CUDA runtime libraries, including cuBLAS; these wheels do not bundle them. Installing the matching CUDA Toolkit supplies those libraries. A toolkit is required for the source-build path, including nvcc and a supported C++ host compiler.

The CUDA Version in nvidia-smi describes driver support, not an installed toolkit. The documented compute-capability floor does not guarantee that an older toolkit supports a newer GPU; check the CUDA build requirements.

For a CUDA 12.5-compatible system:

uv pip install llama-cpp-python --only-binary=llama-cpp-python --force-reinstall --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu125

Expect installation from a wheel. Replace cu125 with the selected suffix. If no compatible wheel exists, install the compiler prerequisites and a toolkit that supports the GPU, ensure nvcc is on PATH, and rebuild:

uv pip install llama-cpp-python --no-binary=llama-cpp-python --force-reinstall --no-cache-dir -C cmake.args=-DGGML_CUDA=on

Expect a CUDA compilation followed by installation. In a GPU-less image build, set the target architecture explicitly instead of relying on native GPU detection. For a T4 (compute capability 7.5), use:

uv pip install llama-cpp-python --no-binary=llama-cpp-python --force-reinstall --no-cache-dir -C "cmake.args=-DGGML_CUDA=on;-DCMAKE_CUDA_ARCHITECTURES=75"

Expect the build to target architecture 75. Replace it with the deployment GPU’s architecture, and use a toolkit that supports that target. Building the image does not verify GPU execution; run the model check after deploying to the GPU host. A CUDA wheel may fail to import during a GPU-less build because the host driver library libcuda.so.1 is absent.

Install Metal on Apple Silicon

Use arm64-native Python on macOS 11 or later. Confirm the interpreter architecture:

python -c "import platform; print(platform.machine())"

Expect arm64. If it prints x86_64, recreate the environment using arm64 Python before installing:

uv pip install llama-cpp-python --only-binary=llama-cpp-python --force-reinstall --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/metal

A valid wheel installs without compilation. If uv reports ZIP file contains trailing contents or Bad uncompressed size, or pip reports Bad CRC-32, the published archive is damaged; use the source fallback with Xcode command-line tools installed:

uv pip install llama-cpp-python --no-binary=llama-cpp-python --force-reinstall --no-cache-dir -C cmake.args=-DGGML_METAL=on

Expect Built llama-cpp-python followed by installation.

Install other GPU backends

Use the CPU wheel command with the index suffix for the selected backend. Expect installation from a wheel.

Check AMD’s compatibility matrix before choosing ROCm. For source builds, follow the upstream backend instructions; Intel GPUs using SYCL need the oneAPI setup.

Save the backend in a uv project

For an existing uv project, add this configuration to pyproject.toml before adding the dependency. Replace cpu in the URL with the compatible backend suffix; the backend’s platform and runtime requirements still apply.

[[tool.uv.index]]
name = "llama-cpp-python"
url = "https://abetlen.github.io/llama-cpp-python/whl/cpu"
explicit = true

[tool.uv.sources]
llama-cpp-python = { index = "llama-cpp-python" }

explicit = true reserves this index for the named package. Add the dependency with source builds disabled:

uv add llama-cpp-python --no-build-package llama-cpp-python

Expect an installation summary and updates to pyproject.toml and uv.lock. Commit both files to preserve the backend choice and resolved version.

Verify the installation

python -c "from llama_cpp import llama_supports_gpu_offload; print('GPU offload supported:', llama_supports_gpu_offload())"

Expect GPU offload supported: False for Linux/Windows CPU builds and True for a working GPU backend. A GPU build can report False when no supported device is available. True checks offload availability; it does not prove that a model uses the GPU.

A missing libcuda.so.1 means the NVIDIA driver library is unavailable, even if the toolkit is installed. For containers, expose the host GPU and driver through the NVIDIA Container Toolkit. Missing libcudart.so or libcublas.so instead points to CUDA runtime libraries; missing HIP or Vulkan libraries require their matching runtime. For an incompatible architecture error, select a wheel matching the interpreter and OS.

For a GPU installation, supply a compatible local GGUF model that fits in GPU memory. Save this as verify_gpu.py:

import sys
from llama_cpp import Llama, llama_supports_gpu_offload

assert llama_supports_gpu_offload(), "Install a GPU backend first"
model = Llama(model_path=sys.argv[1], n_gpu_layers=-1, n_ctx=256, verbose=True)
result = model("Once upon a time", max_tokens=8, temperature=0)
assert result["usage"]["completion_tokens"] > 0
print(result["choices"][0]["text"])

With the environment still activated, replace model.gguf with the model’s path:

python verify_gpu.py model.gguf

Expect generated text and startup logs reporting a positive number of layers offloaded to the selected GPU, with CUDA, Metal, or HIP device buffers. Both successful generation and positive offload logs are required: generation alone can run on the CPU. n_gpu_layers defaults to 0, so installing a GPU wheel without setting it leaves model layers on the CPU.

Resolve installation failures

Source builds need GCC or Clang on Linux, Xcode command-line tools on macOS, or Visual Studio Build Tools with C++ on Windows. The isolated build installs CMake. If it requests missing clang, install Clang or select a supported compiler.

For the default source build:

uv pip install llama-cpp-python --no-binary=llama-cpp-python --force-reinstall --no-cache-dir

Expect Built llama-cpp-python followed by installation. Keep all three flags in source commands: --no-binary selects source, --force-reinstall replaces the installed package, and --no-cache-dir prevents cached builds from being reused.

For CUDA and Metal wheels, the README documents Python 3.10–3.12. Generic py3-none wheel tags do not enforce that range; check the index for the release and platform needed when using another Python version.