# How to run tests in parallel with pytest-xdist


A large test suite that runs sequentially can take minutes. `pytest-xdist` distributes tests across multiple worker processes, which can lead to considerable speed ups depending on the test suite and hardware.

## Installation

Add pytest-xdist as a dev dependency using [uv](https://pydevtools.com/handbook/reference/uv.md):

```bash
uv add --dev pytest-xdist
```

## Running tests in parallel

Pass `-n auto` to let pytest-xdist detect the number of available CPU cores and spawn that many workers:

```bash
uv run pytest -n auto
```

To specify an exact number of workers:

```bash
uv run pytest -n 4
```

Each worker runs in its own process with its own Python interpreter. Tests are distributed to workers as they become available. The output is collected and displayed as a single report.

## Making it the default

Add `-n auto` to your [pyproject.toml](https://pydevtools.com/handbook/reference/pyproject.toml.md) so parallel execution happens without remembering the flag:

```toml
[tool.pytest.ini_options]
addopts = "-n auto"
```

To temporarily override this and run tests sequentially (useful for debugging):

```bash
uv run pytest -n 0
```

## When tests are not safe to parallelize

Parallel execution assumes tests are independent. Tests that share mutable state will produce intermittent failures when run concurrently. Common sources of trouble:

Shared files or databases. Two workers writing to the same SQLite file or temp directory will collide. Use pytest's `tmp_path` fixture to give each test its own directory, or use session-scoped fixtures with worker-aware setup.

Shared external resources. Because each worker is a separate process, in-memory state like module-level variables and singletons is already isolated. The real problems come from shared external resources: databases, files on disk, network ports, or external services that multiple workers access simultaneously.

Port binding. Tests that start a server on a fixed port will fail when two workers try to bind the same port. Use port 0 to let the OS assign a free port.

## Worker-scoped fixtures

pytest-xdist assigns each worker an ID (`gw0`, `gw1`, etc.). Fixtures that need to be unique per worker can use `worker_id`:

```python
@pytest.fixture(scope="session")
def db_name(worker_id):
    return f"test_db_{worker_id}"
```

For session-scoped fixtures that should run only once across *all* workers (not once per worker), use `tmp_path_factory` with a file lock. The first worker to acquire the lock performs setup; others wait and then read the result:

```python
import pytest
from filelock import FileLock


@pytest.fixture(scope="session")
def shared_data(tmp_path_factory, worker_id):
    if worker_id == "master":
        # not running with xdist, set up directly
        data_dir = tmp_path_factory.mktemp("data")
        populate_test_data(data_dir)
        return data_dir

    root_tmp = tmp_path_factory.getbasetemp().parent
    lock = root_tmp / "data.lock"
    data_dir = root_tmp / "shared_data"

    with FileLock(str(lock)):
        if not data_dir.exists():
            data_dir.mkdir()
            populate_test_data(data_dir)

    return data_dir
```

This requires the `filelock` package (`uv add --dev filelock`). The `populate_test_data` function is whatever setup your tests need (creating files, seeding a database, etc.).

## Controlling test distribution

By default, pytest-xdist distributes tests to available workers using load balancing (`--dist load`). It sends pending tests in chunks to whichever worker is free. The most useful alternative modes:

- `--dist loadscope` groups tests by module or class, keeping them on the same worker. This avoids redundant setup when tests share module- or class-scoped fixtures.
- `--dist loadfile` groups tests from the same file on one worker. Similar to `loadscope` but scoped to files only.
- `--dist worksteal` starts with an even split, then lets idle workers steal tests from busy ones. Good for suites with uneven test durations.

```bash
uv run pytest -n auto --dist loadscope
```

See the [pytest-xdist distribution docs](https://pytest-xdist.readthedocs.io/en/stable/distribution.html) for the full list of modes.

## Measuring the speedup

Compare sequential and parallel run times:

```bash
uv run pytest --durations=10          # sequential, shows slowest tests
uv run pytest -n auto --durations=10  # parallel
```

If the speedup is less than expected, check for a few slow tests that dominate total time. Parallelism helps most when many tests take roughly the same amount of time.

## See also

- [Setting up testing with pytest and uv](https://pydevtools.com/handbook/tutorial/setting-up-testing-with-pytest-and-uv.md) for project setup
- [How to run tests using uv](https://pydevtools.com/handbook/how-to/how-to-run-tests-using-uv.md) for basic test running, filtering, and output control
- [How to test against multiple Python versions using uv](https://pydevtools.com/handbook/how-to/how-to-test-against-multiple-python-versions-using-uv.md)
- [Do you still need tox or nox if you use uv?](https://pydevtools.com/handbook/explanation/do-you-still-need-tox-or-nox-if-you-use-uv.md) discusses when to add a test orchestrator
- [pytest reference](https://pydevtools.com/handbook/reference/pytest.md)
- [pytest-xdist documentation](https://pytest-xdist.readthedocs.io/)