What is the Python typing conformance suite?
A checker can have different conformance percentages depending on the metric. One may count only test files that pass completely, while the official dashboard gives half credit to partial results. Without the metric name, neither percentage tells you what was counted.
The Python typing conformance suite checks how static type checkers implement the Python typing specification. It gives checker authors a shared set of examples for rules that otherwise invite different interpretations.
See what the suite tests
The suite organizes Python files around specification chapters such as generics, protocols, typed dictionaries, overloads, and type narrowing. Each file can exercise several related cases. Comments mark where a checker must report an error, may report one, or must choose one of several valid locations.
The Python typing community maintains the suite in the python/typing repository. Its lockfile pins the checker versions used for a run. The result for each file is Pass, Partial, Unsupported, or Unknown, and maintainers inspect output when it changes because different tools report errors with different wording and source ranges. The suite README says its automated comparison is unreliable for some cases.
This means the dashboard is not counting individual assertions. Its denominator is the number of test files, and each file contributes one result.
Read the weighted score
The dashboard generator assigns these values:
| Result | Credit |
|---|---|
Pass |
1 |
Partial |
0.5 |
Unsupported or Unknown |
0 |
Suppose a checker passes one file, partially passes one, and does not support one. The dashboard reports 1.5 out of 3, or 50%. A count of fully passing files reports 1 out of 3, or 33.3%, from the same results.
Call the first number the weighted conformance score and the second the full-pass count. Do not compare a weighted percentage for one checker with a full-pass percentage for another.
Use the score as one signal
A higher weighted score means that a checker received more weighted credit across the tested files. This helps compare coverage of the typing specification, but it does not establish that the checker catches more bugs in a particular codebase.
The suite does not measure speed, memory use, editor features, diagnostic quality, framework support, or behavior outside the specification. The dashboard warns against using conformance as the primary basis for choosing a checker.
Checker versions also differ across dashboard runs. Results describe the versions named at the top of the table, not every release of each tool. The suite does not provide a separate result for Basedpyright, so Pyright’s score cannot be reported as Basedpyright’s score.
Check the current results
Use the live conformance dashboard when a current number matters. Record the suite revision, checker version, date, numerator, denominator, and metric name with any copied score. When running the suite yourself, also preserve the commands and environment so the result can be reproduced.
For choosing a tool, read the score alongside the practical trade-offs in How do Python type checkers compare?.