Python build dependency in LLVM and/or clang?

Hello llvm-devs,

I'm currently starting to look at implementing compiler intrinsics for the Arm MVE vector instruction set.

In a similar sort of style to the existing NEON intrinsics, this is going to involve describing the set of functions needed in Tablegen, and then processing that data into a header file and some bits and pieces to compile into clang (the list of builtins, data tables needed by Sema or CodeGen, etc). But the differences between MVE and NEON are large enough - particularly the predication system - that I'm going to need new Tablegen backends, rather than reusing the NEON ones.

I would like, if possible, to write these Tablegen backends in Python, by consuming the output of the 'llvm-tblgen -dump-json' feature I added to Tablegen last year. I think they'll be simpler and more concise that way than in C++, not to mention quicker to develop.

But before I do that, I wanted to check whether there would be any objection on grounds of build dependencies. Is it acceptable to require Python as part of the normal build process for LLVM? Or is Python something that can be used in places like the test suite and developer utilities, but has to be kept out of the main build to make bootstrapping as easy as possible? I looked for a written policy on this, but didn't find one.

Cheers,
Simon

At the moment, Python is not truely necessary for building LLVM. The
cmake build system depends on it for some parts, but that's all. I would
be very sad for that to change.

Joerg

From: llvm-dev [mailto:llvm-dev-bounces@lists.llvm.org] On Behalf Of Joerg
Sonnenberger via llvm-dev
Sent: Friday, July 05, 2019 8:01 AM
To: llvm-dev@lists.llvm.org
Subject: Re: [llvm-dev] Python build dependency in LLVM and/or clang?

> But before I do that, I wanted to check whether there would be any
> objection on grounds of build dependencies. Is it acceptable to require
Python
> as part of the normal build process for LLVM?

At the moment, Python is not truely necessary for building LLVM. The
cmake build system depends on it for some parts, but that's all. I would
be very sad for that to change.

Technically true, but Python is required to run the 'lit' tests which is
a pretty fundamental operation within LLVM. I can't imagine a situation
where I would want to run a build without running check-whatever after.
So really, you do have to have Python already.
--paulr

More than the bootstrapping question or adding a dependency to the build, I’m wary of using Python pervasively with respect to maintenance and readability of the code. While convenient for small scripts, the lack of static types is making it hard to understand and track the code beyond this simple use-case. We don’t have a style guide for python, the tools (formatting, indexing, IDEs, …) used for C++ development in LLVM aren’t providing the same experience with Python, etc.

llvm-build (LLVMBuild.txt) requires python. The LLVM build will abort with

   message(FATAL_ERROR
    "Unable to find Python interpreter, required for builds and testing.

    Please install Python or specify the PYTHON_EXECUTABLE CMake variable.")

without it.

Michael

That's what I said. The cmake based build system needs it. LIT is
optional. It is entirely feasible to have a custom build system and no
Python at all. It would be quite painful to break that.

Joerg

I'd like to cautiously push back on this too.
While python may or may not already be required to the certain extents
in certain configs,
requiring it for build (for one target, true) 'even more' by writing a
whole upstream TableGen
backed in it is just bad practice IMO.

> >
> > > But before I do that, I wanted to check whether there would be any
> > > objection on grounds of build dependencies. Is it acceptable to require Python
> > > as part of the normal build process for LLVM?
> >
> > At the moment, Python is not truely necessary for building LLVM. The
> > cmake build system depends on it for some parts, but that's all. I would
> > be very sad for that to change.
>
> llvm-build (LLVMBuild.txt) requires python. The LLVM build will abort with
>
> message(FATAL_ERROR
> "Unable to find Python interpreter, required for builds and testing.
>
> Please install Python or specify the PYTHON_EXECUTABLE CMake variable.")
>
> without it.

That's what I said. The cmake based build system needs it. LIT is
optional. It is entirely feasible to have a custom build system and no
Python at all. It would be quite painful to break that.

Joerg

Roman.

To be explicit (and correct me if I misstate facts), NetBSD wants to
build Clang with minimal dependencies because it's the system
compiler. And so it completely replaces our build system with
something not far removed from a hard-coded set of build steps to
eliminate CMake/make/ninja (maybe you allow make?). Tests are not run
(at that stage?) since they do always require Python.

To be frank, I do not think we should compromise LLVM to support this
system. It's just not a realistic thing to impose on an external
project: NetBSD's footprint constraints are its choice, and it should
bear the burden of supporting them.

Personally, I support adding Python as a build dependency. Though I'd
be interested in seeing responses to Mehdi's comments.

Cheers.

Tim.

LLVM’s dependency on Python to build is historical, and could actually be removed. We use it for LLVMBuild, which was important back when we had autoconf as a method of maintaining library dependency graphs between autoconf and CMake. We could actually remove LLVMBuild and with it the python dependency. There was even a few patches a few years back to do it.

See:

https://reviews.llvm.org/D28086
https://reviews.llvm.org/D28856
https://reviews.llvm.org/D28855

-Chris

This would be lovely.

                   -David

Chris Bieneman via llvm-dev <llvm-dev@lists.llvm.org> writes:

Hi Simon,

Just a reminder: TableGen is a huge build bottleneck on some machines. I worry that dumping JSON and then pushing that through Python would make this bottleneck much worse. Consider this: TableGen performance is so critical that there is an entire build option just to force it to be built with optimizations when creating debug builds: LLVM_OPTIMIZED_TABLEGEN

Dave