[RFC] Upgrading LLVM's minimum required Python version

jpienaar · January 9, 2023, 5:28pm

Proposal to update minimum Python version required to 3.8.

What version do you suggest to increase to?

3.8 (3.7 is EOL in ~5 months)

Why this upgrade?

Existing minimum has been EOL since Dec 2021.

Where is the pain in keeping the current min version?

LLVM devs are unable to test with the current minimum version (e.g., no longer carried by some distros or homebrew) and code that passes on newer version fails on the older due to (among others) changes in type specification.

What (common) dists / OSes would be affected (not have access to this version by default)

Finding these have been a bit difficult, pkgs.org only shows openSUSE 15.04, ALT Linux P9, Amazon Linux 2, Amazon Linux 2, Rocky Linux 8 as ones where 3.6 is available but not 3.8, but even for some of these spot check 3.8 was as easy to get installed as 3.6.

Centos 7 (Centos 8 seems to support)
Ubuntu 18.04 default Python install
RHEL 8 seems default is Python 2.7 and recommendation is to use SCL for Python 3 (which has 3.8)
openSUSE 15.04 (only in experimental, official in Tumbleweed)

What’s the process to if you don’t have the right version. For users and developers.

Installing a new version of Python is relatively easy and doesn’t require root access. Projects like pyenv, conda/miniconda/mamba make it easy to install additional Python versions. For many distros it is a case of just installing (e.g., for Ubuntu 18.04/mlir-nvidia buildbot one needs only apt-get pyton3.8 instead of python, CentOS 7 requires using SCL). Or just building from source.

We could start with updating the build bots and then set the minimum version in CMake.

hansw2000 · January 10, 2023, 8:59am

and code that passes on newer version fails on the older due to (among others) changes in type specification.

Do you have an example of where this has happened?

What (common) dists / OSes would be affected

Debian buster (oldstable) is on 3.7.

Do we even currently codify the minimum version in cmake or some other place? I didn’t know we have a minimum version requirement.

mehdi_amini · January 10, 2023, 9:14am

It would also be worth codifying which components require python. What is there anything beyond python bindings? Ideally you don’t pay for what you don’t need.
I can think of LLDB which has python support (I don’t know if this is enabled by a flag?).

tschuett · January 10, 2023, 9:23am

In the Pre-RFC, you can see that it is documented, but it is not enforced by the build system:

mstorsjo · January 10, 2023, 9:36am

Lit uses (quite a bit) of python. Additionally, each individual project may have a nontrivial amount of project specific code - in particular, libcxx has quite a lot of test framework code of its own, running on top of lit. Libcxx also has some scripts uses for wrapping execution (libcxx/utils, run.py and ssh.py), where the main one, run.py gets used in most cases, while ssh.py only is used in remote testing setups (so a regular run through a normal test setup wouldn’t exercise that code).

Additionally, a bunch of scripts in llvm/utils (update_*_test_checks etc) also are written in Python.

tschuett · January 10, 2023, 12:13pm

LLDB is a heavy user of Python: integrated into the debugger and test suite ( -DLLDB_ENABLE_PYTHON=On).

cmake will move to 3.20. Since 3.19 you can enforce a Python version with find_package:
https://cmake.org/cmake/help/latest/module/FindPython3.html

serge-sans-paille · January 10, 2023, 5:43pm

The build system enforces a minimum Python requirement, see

github.com

llvm/llvm-project/blob/main/llvm/CMakeLists.txt#L832


      
            set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default OFF)
          endif()
          set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR ${LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default} CACHE BOOL
            "Enable per-target runtimes directory")
          
          
set(LLVM_PROFDATA_FILE "" CACHE FILEPATH
            "Profiling data file to use when compiling in order to improve runtime performance.")
          
          
if(LLVM_INCLUDE_TESTS)
            # Lit test suite requires at least python 3.6
            set(LLVM_MINIMUM_PYTHON_VERSION 3.6)
          else()
            # FIXME: it is unknown if this is the actual minimum bound
            set(LLVM_MINIMUM_PYTHON_VERSION 3.0)
          endif()
          
          
# Find python before including config-ix, since it needs to be able to search
          # for python modules.
          find_package(Python3 ${LLVM_MINIMUM_PYTHON_VERSION} REQUIRED
              COMPONENTS Interpreter)

I’m personnaly fine with moving to Python 3.8. In addition to 3.6 being EOL, python 3.7 + 3.8 bring (see What’s New In Python 3.8 — Python 3.11.1 documentation and What’s New In Python 3.7 — Python 3.11.1 documentation)

Data classes (What’s New In Python 3.7 — Python 3.11.1 documentation)
easier debugging (What’s New In Python 3.7 — Python 3.11.1 documentation)
core support for typing (What’s New In Python 3.7 — Python 3.11.1 documentation What’s New In Python 3.7 — Python 3.11.1 documentation)
walrus operator (PEP 572 – Assignment Expressions | peps.python.org)
positional only parameters (PEP 570 – Python Positional-Only Parameters | peps.python.org)

jpienaar · January 10, 2023, 6:03pm

Yes, Buildbot and I commented on a commit where it seemed like it would break too (but to test they or I have to fire up docker). The reason I mentioned the change in typing support is that has changed how many folks write new Python code.

@mstorsjo and @tschuett mentioned the others. lit is the one that I was thinking of being least easy to ignore (well unless you aren’t testing which I would consider very rare but have not data to back that). So we can require conditionally same as today only if you are testing/using lit, building LLDB or generating MLIR Python bindings (And I’d prefer that to be 1 version). We could expand the minimum requirement there to specify which components need it or potentially instead add “asserts” in the CMake file of those components to make the tracking more active.

tschuett · January 10, 2023, 6:16pm

I misread the CMake documentation.

hansw2000 · January 11, 2023, 12:58pm

In general, I think we should try to keep the requirements to build LLVM as low as possible. It’s already pretty complex to get started with as it is.

LLVM has always required fairly specific versions of C++ compiler, since that’s a core technology that the project uses to the max, but for auxiliary tools like Python I don’t think it makes sense to be as aggressive.

Perhaps sub-projects that are Python heavy (LLDB, MLIR?) could have stricter requirements, but for just llvm+clang+lit tests it would be nice if pretty much any Python 3 version was sufficient.