How to run tests in parallel with pytest-xdist

by Tim Hopper

Testing

A large test suite that runs sequentially can take minutes. pytest-xdist distributes tests across multiple worker processes, which can lead to considerable speed ups depending on the test suite and hardware.

Installation

Add pytest-xdist as a dev dependency using uv:

uv add --dev pytest-xdist

Running tests in parallel

Pass -n auto to let pytest-xdist detect the number of available CPU cores and spawn that many workers:

uv run pytest -n auto

To specify an exact number of workers:

uv run pytest -n 4

Each worker runs in its own process with its own Python interpreter. Tests are distributed to workers as they become available. The output is collected and displayed as a single report.

Making it the default

Add -n auto to your pyproject.toml so parallel execution happens without remembering the flag:

[tool.pytest.ini_options]
addopts = "-n auto"

To temporarily override this and run tests sequentially (useful for debugging):

uv run pytest -n 0

When tests are not safe to parallelize

Parallel execution assumes tests are independent. Tests that share mutable state will produce intermittent failures when run concurrently. Common sources of trouble:

Shared files or databases. Two workers writing to the same SQLite file or temp directory will collide. Use pytest’s tmp_path fixture to give each test its own directory, or use session-scoped fixtures with worker-aware setup.

Shared external resources. Because each worker is a separate process, in-memory state like module-level variables and singletons is already isolated. The real problems come from shared external resources: databases, files on disk, network ports, or external services that multiple workers access simultaneously.

Port binding. Tests that start a server on a fixed port will fail when two workers try to bind the same port. Use port 0 to let the OS assign a free port.

Worker-scoped fixtures

pytest-xdist assigns each worker an ID (gw0, gw1, etc.). Fixtures that need to be unique per worker can use worker_id:

@pytest.fixture(scope="session")
def db_name(worker_id):
    return f"test_db_{worker_id}"

For session-scoped fixtures that should run only once across all workers (not once per worker), use tmp_path_factory with a file lock. The first worker to acquire the lock performs setup; others wait and then read the result:

import pytest
from filelock import FileLock


@pytest.fixture(scope="session")
def shared_data(tmp_path_factory, worker_id):
    if worker_id == "master":
        # not running with xdist, set up directly
        data_dir = tmp_path_factory.mktemp("data")
        populate_test_data(data_dir)
        return data_dir

    root_tmp = tmp_path_factory.getbasetemp().parent
    lock = root_tmp / "data.lock"
    data_dir = root_tmp / "shared_data"

    with FileLock(str(lock)):
        if not data_dir.exists():
            data_dir.mkdir()
            populate_test_data(data_dir)

    return data_dir

This requires the filelock package (uv add --dev filelock). The populate_test_data function is whatever setup your tests need (creating files, seeding a database, etc.).

Controlling test distribution

By default, pytest-xdist distributes tests to available workers using load balancing (--dist load). It sends pending tests in chunks to whichever worker is free. The most useful alternative modes:

--dist loadscope groups tests by module or class, keeping them on the same worker. This avoids redundant setup when tests share module- or class-scoped fixtures.
--dist loadfile groups tests from the same file on one worker. Similar to loadscope but scoped to files only.
--dist worksteal starts with an even split, then lets idle workers steal tests from busy ones. Good for suites with uneven test durations.

uv run pytest -n auto --dist loadscope

See the pytest-xdist distribution docs for the full list of modes.

Measuring the speedup

Compare sequential and parallel run times:

uv run pytest --durations=10          # sequential, shows slowest tests
uv run pytest -n auto --durations=10  # parallel

If the speedup is less than expected, check for a few slow tests that dominate total time. Parallelism helps most when many tests take roughly the same amount of time.