# \[RFC\] Type checking to python code

**URL:** <https://discourse.llvm.org/t/rfc-type-checking-to-python-code/86605>\
**Category:** LLDB\
**Created:** [May 29, 2025, 4:20pm UTC](https://discourse.llvm.org/t/rfc-type-checking-to-python-code/86605 "2025-05-29T16:20:12Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![ashgti](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/ashgti/32/18088_2.png) [@ashgti](https://discourse.llvm.org/u/ashgti)\
**Post date:** [May 29, 2025, 4:20pm UTC](https://discourse.llvm.org/t/rfc-type-checking-to-python-code/86605/1 "2025-05-29T16:20:12Z")

</div>

## Background

In lldb, we have a minimum python version of 3.8, if python support is enabled.

Since python 3.5 it has been possible to add type annotations to code and the new ‘typing’ module was introduced.

I have been working on improving the unit tests for lldb-dap and while working on that, I found its helpful to have additional type information. This can help in the IDE by helping me ensure the value or structure or parameters of a function are correct. Here is an [example PR](https://github.com/llvm/llvm-project/pull/141689) of the kinds of type annotations I have been adding.

However by default the type annotations are not going to do much (if anything).

In python, you need to run a static type checker to validate the types. ~~There is some support for runtime type assertions using [typing.assert\_type](https://docs.python.org/3/library/typing.html#typing.assert_type),~~ (EDIT: assert\_type has no runtime side effects, its only a hint to the type checker) otherwise you need to invoke a tool like [mypy](https://github.com/python/mypy) or [pyright](https://github.com/microsoft/pyright). Other type checkers exist, but I believe those are the two most popular choices.

## Proposal

At the moment, any python code annotated with types is not validated as part of the CI. This means that the types could regress over time.

We could address this in a few ways:

- Add a pre-commit hook to run the type checker, like the linter checks
- Add a pre-merge hook to run the type checker, like the linter checks
- Run the type checker before we run the python based tests

Additionally, if we wanted to run a type checker, we would also need to pick which type checker we’d like to use. Like I mentioned previously, `mypy` and `pyright` are two popular choices but others exist.

I think `mypy` is a good option because its supported by the python core team and is a fairly common choice.

I also think it would be worth adding the `type_extensions` library to the `lldb/test/requirements.txt` since it includes back ports of new types for older versions of python.

### Running mypy

At the moment, we don’t have a config setup specifically for running a type checker, but you can run the files directly like:

```bash
$ time mypy --python-version 3.8 \
  lldb/packages/Python/lldbsuite/**/*.py \
  lldb/test/API/tools/lldb-dap/**/*.py
... a couple of warnings / errors from existing code
mypy --python-version 3.8 lldb/packages/Python/lldbsuite/**/*.py 0.66s user 0.18s system 66% cpu 1.270 total

```

On my machine, an M1 Max MBP, it takes ~1.5s to run the type checker on the lldb unit tests.

* * *

Issue tracker link: [[lldb] Add a CI step to run a python type checker · Issue #141877 · llvm/llvm-project · GitHub](https://github.com/llvm/llvm-project/issues/141877)

---

<div class="post-metadata">

**Author:** ![JDevlieghere](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/jdevlieghere/32/2048_2.png) [@JDevlieghere](https://discourse.llvm.org/u/JDevlieghere)\
**Post date:** [May 29, 2025, 5:18pm UTC](https://discourse.llvm.org/t/rfc-type-checking-to-python-code/86605/2 "2025-05-29T17:18:43Z")

</div>

I like the idea of supporting typing where it benefits us. The PR you linked is a good example of where this adds value and help catch mistakes. I can also imagine a bunch of places where it doesn’t make sense and dynamic typing is totally fine (and static types would just add an additional burden). If we’re all on the same page about that, let’s add a snippet to the [testing page](https://lldb.llvm.org/resources/test.html).

As for the proposal to enforce this, my vote would be to do the same as the formatters and make this a pre-commit PR check. We can add a convenience CMake target (something like `check-lldb-types`) that runs the types checker on our test suite at desk. I imagine in CI we don’t want to go through the CMake configuration stage.

I don’t have an opinion on `mypy` vs `pyright`. I use the latter as my Python LSP server, but that’s pretty much orthogonal.

---

<div class="post-metadata">

**Author:** ![Endill](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/endill/32/13614_2.png) [@Endill](https://discourse.llvm.org/u/Endill)\
**Post date:** [May 30, 2025, 9:41am UTC](https://discourse.llvm.org/t/rfc-type-checking-to-python-code/86605/3 "2025-05-30T09:41:38Z")

</div>

Over on the Clang side, we’re slowly adding type annotations to libclang’s Python bindings. This compelled us to look closely at this layer of our API and find and fix several bugs.

@JannickKremer Any thoughts here?

---

<div class="post-metadata">

**Author:** ![JannickKremer](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/jannickkremer/32/18576_2.png) [@JannickKremer](https://discourse.llvm.org/u/JannickKremer)\
**Post date:** [May 30, 2025, 10:22am UTC](https://discourse.llvm.org/t/rfc-type-checking-to-python-code/86605/4 "2025-05-30T10:22:36Z")

</div>

100% in favor of type-checking in the CI!

That was also among my plans for the libclang Python bindings in the future. A few concerns and open questions that I’d like to raise:

1. the main issue I see is how to deal with incomplete type hints and what the stated goal should be. As Endill mentioned, we are currently in the process of adding annotations for the libclang Python bindings, but haven’t added CI checks here yet. That’s because it is difficult to define a meaningful check for incomplete type hints. Is any PR that doesn’t _increase_ the current number of typing errors a pass? Should the project have 0 type errors? Should it pass a strict type check?
2. The answers to the above question also depend heavily on the needs of the subproject, as JDevlieghere said. The benefits of type annotations are both internal (they serve as additional documentation and help find bugs) and external (better IDE support for downstream users). The goal with libclang’s Python bindings, for example, is to eventually pass a strict type check, because we want users of the bindings to have better support in their IDE by annotating all the interfaces. In comparison, your project of annotating tests sounds like it only needs the internal benefits, and thus wouldn’t gain much from a strict pass.
3. As a result, rather than an LLVM-wide type-checking job, it might be preferable to have a job that can be added on a per-project basis, or with different parameters per folder or similar, so every project can decide if/when they are ready and to what extent they want to integrate type-checking in their CI.

Aside from that, I’d vote for mypy but that’s mostly because that’s what I’ve used and contributed to before.

---

<div class="post-metadata">

**Author:** ![ashgti](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/ashgti/32/18088_2.png) [@ashgti](https://discourse.llvm.org/u/ashgti)\
**Post date:** [May 30, 2025, 6:18pm UTC](https://discourse.llvm.org/t/rfc-type-checking-to-python-code/86605/5 "2025-05-30T18:18:09Z")

</div>

> [@JannickKremer](#):
>
> That’s because it is difficult to define a meaningful check for incomplete type hints. Is any PR that doesn’t _increase_ the current number of typing errors a pass? Should the project have 0 type errors? Should it pass a strict type check?

I think it would be good to define a config that we can use opt in as we work on areas of the code base. In `mypy` you need to use the `--check-untyped-defs` to check all declarations, otherwise it will only check functions that have type annotations.

Here is a prototype init file for lldb with a few error codes disabled:

```ini
$ cat lldb/.mypy.ini
[mypy]
python_version = 3.8
files = lldb/packages/Python/lldbsuite/ **/*.py,lldb/test/API/tools/lldb-dap/** /*.py
warn_no_return = True
warn_redundant_casts = True
warn_return_any = True
warn_unreachable = True
warn_unused_ignores = True

[mypy-lldb]
disable_error_code = import-not-found,no-redef

[mypy-dap_server]
allow_redefinition_new = True
local_partial_types = True

```

To run this, use:

```auto
$ MYPYPATH=`<build>/bin/lldb -P` mypy --config-file=lldb/.mypy.ini

```

Right now, we have the following errors:

```auto
lldb/packages/Python/lldbsuite/support/gmodules.py:6: error: Need type annotation for "GMODULES_SUPPORT_MAP" (hint: "GMODULES_SUPPORT_MAP: Dict[<type>, <type>] = ...") [var-annotated]
lldb/packages/Python/lldbsuite/test/lldbtest_config.py:13: error: Need type annotation for "channels" (hint: "channels: List[<type>] = ...") [var-annotated]
lldb/packages/Python/lldbsuite/test/configuration.py:29: error: Need type annotation for "skip_categories" (hint: "skip_categories: List[<type>] = ...") [var-annotated]
lldb/packages/Python/lldbsuite/test/configuration.py:31: error: Need type annotation for "xfail_categories" (hint: "xfail_categories: List[<type>] = ...") [var-annotated]
lldb/packages/Python/lldbsuite/test/configuration.py:33: error: Need type annotation for "failures_per_category" (hint: "failures_per_category: Dict[<type>, <type>] = ...") [var-annotated]
lldb/packages/Python/lldbsuite/test/configuration.py:56: error: Need type annotation for "settings" (hint: "settings: List[<type>] = ...") [var-annotated]
lldb/packages/Python/lldbsuite/test/configuration.py:70: error: Need type annotation for "filters" (hint: "filters: List[<type>] = ...") [var-annotated]
lldb/packages/Python/lldbsuite/test/configuration.py:124: error: Need type annotation for "all_tests" (hint: "all_tests: Set[<type>] = ...") [var-annotated]
lldb/packages/Python/lldbsuite/test/configuration.py:138: error: Need type annotation for "enabled_plugins" (hint: "enabled_plugins: List[<type>] = ...") [var-annotated]
lldb/packages/Python/lldbsuite/test/decorators.py:1107: error: "list" is not subscriptable, use "typing.List" instead [misc]
lldb/packages/Python/lldbsuite/test/decorators.py:1114: error: "None" has no attribute "lower" [attr-defined]
lldb/packages/Python/lldbsuite/test/tools/lldb-server/lldbgdbserverutils.py:986: error: Module has no attribute "WinDLL" [attr-defined]
lldb/packages/Python/lldbsuite/test/tools/lldb-server/lldbgdbserverutils.py:1105: error: Name "Pipe" already defined on line 1046 [no-redef]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:200: error: Unsupported target for indexed assignment ("object") [index]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:216: error: Need type annotation for "varref_dict" (hint: "varref_dict: Dict[<type>, <type>] = ...") [var-annotated]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:352: error: Value of type "object" is not indexable [index]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:353: error: Value of type "object" is not indexable [index]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:383: error: Value of type "object" is not indexable [index]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:384: error: Value of type "object" is not indexable [index]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:385: error: Value of type "object" is not indexable [index]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:391: error: Value of type "object" is not indexable [index]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:565: error: Need type annotation for "expr_varref_dict" (hint: "expr_varref_dict: Dict[<type>, <type>] = ...") [var-annotated]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:566: error: "Collection[str]" has no attribute "items" [attr-defined]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:672: error: Incompatible types in assignment (expression has type "Tuple[Dict[str, Dict[str, List[str]]]]", target has type "Dict[str, Dict[str, str]]") [assignment]
Found 24 errors in 6 files (checked 98 source files)

```

Which I think is a reasonable set of errors we could address by either disabling specific warnings in specific files or fixing the files.

> [@JannickKremer](#):
>
> The answers to the above question also depend heavily on the needs of the subproject, as JDevlieghere said. The benefits of type annotations are both internal (they serve as additional documentation and help find bugs) and external (better IDE support for downstream users). The goal with libclang’s Python bindings, for example, is to eventually pass a strict type check, because we want users of the bindings to have better support in their IDE by annotating all the interfaces. In comparison, your project of annotating tests sounds like it only needs the internal benefits, and thus wouldn’t gain much from a strict pass.

Yea, the lldb use case is mostly for test development. At the moment, it appears that swig does not support type annotations on the generated output. There is an [open issue](https://github.com/swig/swig/issues/735) for adding support to swig though.

---

<div class="post-metadata">

**Author:** ![JannickKremer](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/jannickkremer/32/18576_2.png) [@JannickKremer](https://discourse.llvm.org/u/JannickKremer)\
**Post date:** [May 31, 2025, 1:59am UTC](https://discourse.llvm.org/t/rfc-type-checking-to-python-code/86605/6 "2025-05-31T01:59:59Z")

</div>

> [@ashgti](#):
>
> I think it would be good to define a config that we can use opt in as we work on areas of the code base. In `mypy` you need to use the `--check-untyped-defs` to check all declarations, otherwise it will only check functions that have type annotations.

Sounds good to me!  
Note that when I say “strict type check”, I refer to mypy’s [`--strict`](https://mypy.readthedocs.io/en/stable/config_file.html#confval-strict) flag, which implies all optional checks.  
Definitely agree that a flexible, detailed per-project config would be desirable (though not necessarily in the very first step)

---

<div class="post-metadata">

**Author:** ![ashgti](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/ashgti/32/18088_2.png) [@ashgti](https://discourse.llvm.org/u/ashgti)\
**Post date:** [June 5, 2025, 8:31pm UTC](https://discourse.llvm.org/t/rfc-type-checking-to-python-code/86605/7 "2025-06-05T20:31:05Z")

</div>

I got a basic prototype of this integration working by adding this to the lldb `dotest.py` script.

> <https://github.com/ashgti/llvm-project/commit/585ce5d9f6684a346790e7cf12a9f1ecea91b57d>
>
> This is a prototype of integrating mypy into lldb's test suite to validated the …files as part of running tests.
> 
> To enable this, add \`-DLLDB\_PYTHON\_TESTS\_CHECK\_TYPES:BOOL=true\` to your cmake configuration.
> 
> When this flag is enabled, prior to executing a test the dotest.py helper will run mypy, which will report any type checking issues to stdout.
> 
> This will not block the test, many existing scripts do not fully pass mypy in its current configuration.

With this patch, if you use `-DLLDB_PYTHON_TESTS_CHECK_TYPES:BOOL=true` when you run cmake then the `dotest.py` script will run mypy on the test prior to running the test.

If you run an individual test with `dotest.py` with this configuration you’ll see the mypy report in stdout like:

```bash
$ python3 llvm-project/lldb/test/API/dotest.py \
  -u CXXFLAGS \
  -u CFLAGS \
  --out-of-tree-debugserver \
  --env LLVM_LIBS_DIR=lldb-build/./lib \
  --env LLVM_INCLUDE_DIR=lldb-build/include \
  --env LLVM_TOOLS_DIR=lldb-build/./bin \
  --libcxx-include-dir lldb-build/include/c++/v1 \
  --libcxx-library-dir lldb-build/lib \
  --arch arm64 \
  --build-dir lldb-build/lldb-test-build.noindex \
  --lldb-module-cache-dir lldb-build/lldb-test-build.noindex/module-cache-lldb/lldb-api \
  --clang-module-cache-dir lldb-build/lldb-test-build.noindex/module-cache-clang/lldb-api \
  --executable lldb-build/./bin/lldb \
  --compiler lldb-build/./bin/clang \
  --dsymutil lldb-build/./bin/dsymutil \
  --make /usr/bin/make \
  --llvm-tools-dir lldb-build/./bin \
  --lldb-obj-root lldb-build/tools/lldb \
  --lldb-libs-dir lldb-build/./lib \
  --framework lldb-build/bin/LLDB.framework \
  --cmake-build-type RelWithDebInfo \
  --check-types \
  llvm-project/lldb/test/API/tools/lldb-dap/io -p TestDAP_io.py
lldb version 21.0.0git (git@github.com:ashgti/llvm-project.git revision c93fc07792ca4d2e2a082fe3e69aaa27ddd0c83e)
  clang revision c93fc07792ca4d2e2a082fe3e69aaa27ddd0c83e
  llvm revision c93fc07792ca4d2e2a082fe3e69aaa27ddd0c83e
Skipping the following test categories: ['libstdcxx', 'dwo', 'llgs', 'fork']
Success: no issues found in 1 source file

Ran 5 tests in 1.961s

OK

```

Adjusting the file to include an error you will see:

```auto
lldb/test/API/tools/lldb-dap/io/TestDAP_io.py:59: error: Incompatible types in assignment (expression has type "int", variable has type "str") [assignment]
Found 1 error in 1 file (checked 1 source file)

Ran 5 tests in 1.824s

OK

```

I’m not failing the test if the file does not pass the type check at the moment. We could consider adding that in the future though.

I’m open to suggestions if anyone has any other ideas on how we can run this integration.

---

<div class="post-metadata">

**Author:** ![jingham](https://avatars.discourse-cdn.com/v4/letter/j/a698b9/32.png) [@jingham](https://discourse.llvm.org/u/jingham)\
**Post date:** [June 6, 2025, 5:31pm UTC](https://discourse.llvm.org/t/rfc-type-checking-to-python-code/86605/8 "2025-06-06T17:31:53Z")

</div>

I think we should at least have a mode where lldb-dotest does not fail if the type checking fails. When you are initially writing a test, lldb-dotest is the most convenient way to iteratively develop it. This is particularly useful with the `-d` flag when you’re using the test you’ve written to get the feature to work. But in that process you are often adding little utility functions that may not even survive in the final test, more of the “How the heck did we get here” detectors…

Having to be pedantic about type checking during the development of the test would be annoying.

---

<div class="post-metadata">

**Author:** ![ashgti](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/ashgti/32/18088_2.png) [@ashgti](https://discourse.llvm.org/u/ashgti)\
**Post date:** [June 9, 2025, 5:09pm UTC](https://discourse.llvm.org/t/rfc-type-checking-to-python-code/86605/9 "2025-06-09T17:09:26Z")

</div>

> [@jingham](#):
>
> I think we should at least have a mode where lldb-dotest does not fail if the type checking fails. When you are initially writing a test, lldb-dotest is the most convenient way to iteratively develop it.

I can update the `--check-types` flag to take a parameter to control the strictness. Maybe something like `--check-types=error` (causes an error if type checking fails) or `--check-types=warn` (only reports any errors without failing the test).

EDIT:

Updated my branch with this approach: [[lldb] Prototyping type checking python files when running tests. · ashgti/llvm-project@85739f3 · GitHub](https://github.com/ashgti/llvm-project/commit/85739f380c87ce327c307dc828669a03962b2bc2)

When building with cmake, you can use `-DLLDB_PYTHON_TESTS_CHECK_TYPES:STRING=strict|warn`. It defaults to disabled.

When running with `dotest.py` you can use the flag `--check-types strict|warn` or not include the flag at all and it will skip running mypy.

---

<div class="post-metadata">

**Author:** ![ashgti](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/ashgti/32/18088_2.png) [@ashgti](https://discourse.llvm.org/u/ashgti)\
**Post date:** [June 12, 2025, 4:09pm UTC](https://discourse.llvm.org/t/rfc-type-checking-to-python-code/86605/10 "2025-06-12T16:09:20Z")

</div>

In my first iteration at implementing this, I added the flag to lldb’s `dotest.py` script, but maybe I should be looking into llvm-lit instead.

I’ll take a little time to see if I can find a reasonable way to integrate into lit instead and update my branch.

---

<div class="post-metadata">

**Author:** ![ashgti](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/ashgti/32/18088_2.png) [@ashgti](https://discourse.llvm.org/u/ashgti)\
**Post date:** [June 12, 2025, 9:43pm UTC](https://discourse.llvm.org/t/rfc-type-checking-to-python-code/86605/11 "2025-06-12T21:43:47Z")

</div>

Reading into `lit` a bit more, I don’t think this makes sense to have in `lit`. `lit` isn’t aware of the implementation details of the tests. So, for lldb’s tests right now at least, I think it makes the most sense for this to remain as part of dotest.py for now.