How to Install bitsandbytes

bitsandbytes provides 4-bit and 8-bit quantization for large language models. Its PyPI wheels include native libraries for several backends. Install the PyTorch build for the intended accelerator, then install bitsandbytes from PyPI.

Prerequisites

• Python 3.10 or later and PyTorch 2.4 or later, below 3.
• A platform and accelerator listed in the official hardware compatibility table.
• For NVIDIA: a driver compatible with PyTorch’s CUDA runtime. A local CUDA toolkit is unnecessary for a supported wheel.

For NVIDIA GPUs, Linear8bitLt (LLM.int8) needs compute capability 7.5 or higher. NF4/FP4 and 8-bit optimizers support older GPUs, but the selected CUDA binary must include their architecture: the CUDA 12.8 Linux x86-64 binary starts at compute capability 7.0, and the CUDA 13 binaries start at 7.5. Check the CUDA wheel target table before choosing a build for an older GPU.

Install the NVIDIA backend

In an existing virtual environment, install a CUDA-enabled PyTorch build with uv, then bitsandbytes:

uv pip install torch --index https://download.pytorch.org/whl/cu128
uv pip install bitsandbytes

The installer reports a torch build ending in +cu128 and a bitsandbytes wheel from PyPI. Choose another PyTorch index if the driver or GPU needs a different CUDA build; bitsandbytes itself needs no extra index.

For a uv project, configure the PyTorch source and explicit index in pyproject.toml first:

[tool.uv.sources]
torch = { index = "pytorch-cu128" }

[[tool.uv.index]]
name = "pytorch-cu128"
url = "https://download.pytorch.org/whl/cu128"
explicit = true

Then add both dependencies:

uv add torch bitsandbytes

This updates pyproject.toml, uv.lock, and the project environment. bitsandbytes and generic dependencies resolve from PyPI. Configure the index before the first resolve; changing explicit later does not rewrite sources already recorded in uv.lock.

bitsandbytes selects its CUDA library from torch.version.cuda, rather than from nvcc or the CUDA version displayed by nvidia-smi. Install order does not replace backend selection: a CPU-only PyTorch build cannot run the CUDA check.

Verify a CUDA operation

Save this script and run it on the GPU host:

import torch
import bitsandbytes as bnb

assert torch.cuda.is_available(), 'CUDA is unavailable to PyTorch'
assert torch.version.hip is None, 'This check targets NVIDIA CUDA'
assert torch.cuda.get_device_capability() >= (7, 5)
print(torch.__version__, torch.version.cuda, torch.cuda.get_device_name())
linear = bnb.nn.Linear8bitLt(256, 128, has_fp16_weights=False).to('cuda')
x = torch.randn(1, 256, dtype=torch.float16, device='cuda')
out = linear(x)
torch.cuda.synchronize()
assert out.is_cuda and tuple(out.shape) == (1, 128)
assert torch.isfinite(out).all()
print('bitsandbytes CUDA operation OK')

From the uv project, run:

uv run python check_bitsandbytes.py

uv run executes the command inside the project’s virtual environment, ensuring all dependencies are installed first. For an activated standalone virtual environment, use uv run --no-project --active python check_bitsandbytes.py instead.

Expect the GPU name followed by bitsandbytes CUDA operation OK. An import or a backend label alone does not establish that the native CUDA kernels work. A container image can install and import bitsandbytes without a GPU; execute this check after scheduling it onto GPU hardware.

Select another backend

CPU quantization is supported; CPU mode is not automatically an installation error. AMD ROCm, Intel XPU, Apple Silicon, and Gaudi have their own supported devices and feature limits in the upstream backend matrix. Install a matching PyTorch backend and use a device-specific operation to verify it. The NVIDIA check does not validate those combinations.

On Jetson, the server ARM64 wheel is not a substitute for a JetPack-compatible build. Follow upstream Jetson source-build guidance.

Fix a failed CUDA check

• CUDA is unavailable: check GPU visibility and the driver, then confirm PyTorch has a CUDA build using torch.version.cuda.
• A native library fails to load: inspect the full error and the bitsandbytes diagnostics. A missing runtime library also has causes beyond a CPU-only PyTorch installation, including a missing matching bitsandbytes binary or a library search-path conflict.
• The detected CUDA build is wrong: choose the correct PyTorch index. Use BNB_CUDA_VERSION only with the documented runtime-library override procedure; the variable does not install the alternate CUDA runtime or add GPU architecture support.