# How to Install bitsandbytes


bitsandbytes provides 4-bit and 8-bit quantization for large language models. Its [PyPI](https://pydevtools.com/handbook/explanation/what-is-pypi.md) [wheels](https://pydevtools.com/handbook/reference/wheel.md) include native libraries for several backends. Install the [PyTorch](https://pydevtools.com/handbook/how-to/how-to-install-pytorch-with-uv.md) build for the intended accelerator, then install bitsandbytes from PyPI.

## Prerequisites

- Python 3.10 or later and PyTorch 2.4 or later, below 3.
- A platform and accelerator listed in the [official hardware compatibility table](https://github.com/bitsandbytes-foundation/bitsandbytes#system-requirements).
- For NVIDIA: a driver compatible with PyTorch's CUDA runtime. A local CUDA toolkit is unnecessary for a supported wheel.

For NVIDIA GPUs, `Linear8bitLt` (LLM.int8) needs compute capability 7.5 or higher. NF4/FP4 and 8-bit optimizers support older GPUs, but the selected CUDA binary must include their architecture: the CUDA 12.8 Linux x86-64 binary starts at compute capability 7.0, and the CUDA 13 binaries start at 7.5. Check the [CUDA wheel target table](https://huggingface.co/docs/bitsandbytes/installation#nvidia-cuda) before choosing a build for an older GPU.

## Install the NVIDIA backend

In an existing virtual environment, install a CUDA-enabled PyTorch build with [uv](https://pydevtools.com/handbook/reference/uv.md), then bitsandbytes:

```bash
uv pip install torch --index https://download.pytorch.org/whl/cu128
uv pip install bitsandbytes
```

The installer reports a `torch` build ending in `+cu128` and a bitsandbytes wheel from PyPI. Choose another PyTorch index if the driver or GPU needs a different CUDA build; bitsandbytes itself needs no extra index.

For a uv project, configure the PyTorch source and explicit index in `pyproject.toml` first:

```toml
[tool.uv.sources]
torch = { index = "pytorch-cu128" }

[[tool.uv.index]]
name = "pytorch-cu128"
url = "https://download.pytorch.org/whl/cu128"
explicit = true
```

Then add both dependencies:

```bash
uv add torch bitsandbytes
```

This updates `pyproject.toml`, `uv.lock`, and the project environment. bitsandbytes and generic dependencies resolve from PyPI. Configure the index before the first resolve; changing `explicit` later does not rewrite sources already recorded in `uv.lock`.

bitsandbytes selects its CUDA library from `torch.version.cuda`, rather than from `nvcc` or the CUDA version displayed by `nvidia-smi`. Install order does not replace backend selection: a CPU-only PyTorch build cannot run the CUDA check.

## Verify a CUDA operation

Save this script and run it on the GPU host:

```python {filename="check_bitsandbytes.py"}
import torch
import bitsandbytes as bnb

assert torch.cuda.is_available(), 'CUDA is unavailable to PyTorch'
assert torch.version.hip is None, 'This check targets NVIDIA CUDA'
assert torch.cuda.get_device_capability() >= (7, 5)
print(torch.__version__, torch.version.cuda, torch.cuda.get_device_name())
linear = bnb.nn.Linear8bitLt(256, 128, has_fp16_weights=False).to('cuda')
x = torch.randn(1, 256, dtype=torch.float16, device='cuda')
out = linear(x)
torch.cuda.synchronize()
assert out.is_cuda and tuple(out.shape) == (1, 128)
assert torch.isfinite(out).all()
print('bitsandbytes CUDA operation OK')
```

From the uv project, run:

```bash
uv run python check_bitsandbytes.py
```

`uv run` executes the command inside the project's virtual environment, ensuring all dependencies are installed first. For an activated standalone virtual environment, use `uv run --no-project --active python check_bitsandbytes.py` instead.

Expect the GPU name followed by `bitsandbytes CUDA operation OK`. An import or a backend label alone does not establish that the native CUDA kernels work. A container image can install and import bitsandbytes without a GPU; execute this check after scheduling it onto GPU hardware.

## Select another backend

CPU quantization is supported; CPU mode is not automatically an installation error. AMD ROCm, Intel XPU, Apple Silicon, and Gaudi have their own supported devices and feature limits in the [upstream backend matrix](https://github.com/bitsandbytes-foundation/bitsandbytes#system-requirements). Install a matching PyTorch backend and use a device-specific operation to verify it. The NVIDIA check does not validate those combinations.

On Jetson, the server ARM64 wheel is not a substitute for a JetPack-compatible build. Follow [upstream Jetson source-build guidance](https://huggingface.co/docs/bitsandbytes/installation#nvidia-cuda).

## Fix a failed CUDA check

- **CUDA is unavailable:** check GPU visibility and the driver, then confirm PyTorch has a CUDA build using `torch.version.cuda`.
- **A native library fails to load:** inspect the full error and the [bitsandbytes diagnostics](https://huggingface.co/docs/bitsandbytes/errors). A missing runtime library also has causes beyond a CPU-only PyTorch installation, including a missing matching bitsandbytes binary or a library search-path conflict.
- **The detected CUDA build is wrong:** choose the correct PyTorch index. Use `BNB_CUDA_VERSION` only with the [documented runtime-library override procedure](https://huggingface.co/docs/bitsandbytes/errors); the variable does not install the alternate CUDA runtime or add GPU architecture support.
