GN build roundtable summary; adding GN build files to the repo

Project Infrastructure LLVM Dev List Archives
1

Hi,

first things first: If you’re happy with cmake, you can stop reading now. Nobody is proposing that LLVM moves off cmake, and nobody is proposing anything that’s causing people using cmake more work.

At the LLVM conference, I gave a lightning talk [1] about using GN [2] to build LLVM and clang. cmake is great for many use cases, but in my opinion local day-to-day development isn’t one of them. So I wrote GN build files for LLVM and clang, enough to make ninja check-llvm check-clang check-lld build everything needed for these three test suites and that all tests pass. This works on Linux, Mac, Win hosts targeting X86, ARM, AArch64. You can see them at [3].

I had been worried that it would be a lot of work to keep the build files up to date, but I’ve been using this for all my LLVM/clang/lld development the last 8 months, and it turned out to not be a big problem – LLVM’s build files don’t change very often, and GN build files are a pleasure to work with in my opinion.

I gave the lightning talk just to talk about my personal workflow, but there was a lot of interest. We had a roundtable on the next day, and about 20 people said they’d be interested in using this for their development too. The main request was that the .gn files are checked in upstream, so that we can collaborate on keeping them working. Two to three orgs even said they’d be interested in moving their buildbots to GN.

As mentioned at the top, the intention here is not to replace cmake, only to offer an opt-in alternative for people who are interested in it. Keeping the GN build going would be the responsibility of people using it, not of the general LLVM community. If this fails to find use and bitrots, we can easily remove it again.

Are there any concerns with checking in GN files? I’ve put some initial docs for the GN build at https://reviews.llvm.org/D53944 ; it describes what the GN build is and is not, what its advantages are (speed and easier configurability), and some points about the philosophy behind the LLVM GN build.

If folks are generally ok with GN files living in-tree, I’ll start sending patches for gradually adding gn files through the regular review process.

If having a BUILD.gn file in every directory being confusing is a concern, GN has the concept of a “secondary tree” so that all GN files could be below llvm/gn/tree/{llvm,clang,lld,…}.

Cheers,
Nico

1: https://llvm.org/devmtg/2018-10/talk-abstracts.html#lt2
2: https://gn.googlesource.com/gn , https://is.gd/gn_intro
3: https://github.com/llvm-project/llvm-project-20170507/compare/master…nico:gn , click “Files Changed” to see the GN files.

2

I’m in favor of including the gn build files.

Aside: build of gn itself failed for me on Ubuntu 14, link errors related to unsupported relocations, AFAICT this is just an incompatibility between the Debian filesystem downloaded and Ubuntu 14. Could you provide a release tarball of gn for a couple of key platforms?

-Brian

3

We provide prebuilts for GN now, see https://gn.googlesource.com/gn/#getting-started

4

Hi,

first things first: If you’re happy with cmake, you can stop reading now. Nobody is proposing that LLVM moves off cmake, and nobody is proposing anything that’s causing people using cmake more work.

At the LLVM conference, I gave a lightning talk [1] about using GN [2] to build LLVM and clang. cmake is great for many use cases, but in my opinion local day-to-day development isn’t one of them. So I wrote GN build files for LLVM and clang, enough to make ninja check-llvm check-clang check-lld build everything needed for these three test suites and that all tests pass. This works on Linux, Mac, Win hosts targeting X86, ARM, AArch64. You can see them at [3].

I’m in favor of including the gn build files.

Aside: build of gn itself failed for me on Ubuntu 14, link errors related to unsupported relocations, AFAICT this is just an incompatibility between the Debian filesystem downloaded and Ubuntu 14. Could you provide a release tarball of gn for a couple of key platforms?

We provide prebuilts for GN now, see https://gn.googlesource.com/gn/#getting-started

Oh, I see, that works well. Thanks!

-Brian

5

If these files exist in the repo, I could imagine that reviewers will reject patches that might/likely break the gn files.

I could also imagine that contributors will be pressured to fix breakage of gn files that they introduce, or commits being reverted etc.

So, I'm not sure that a policy of 'it's ok to break this stuff' will actually work.

Do you think there will be some point in time that these files will be out of 'experimental' status?

Thanks,

Stephen.

6

From: cfe-dev [mailto:cfe-dev-bounces@lists.llvm.org] On Behalf Of Stephen
Kelly via cfe-dev
Sent: Wednesday, October 31, 2018 4:20 PM
To: cfe-dev@lists.llvm.org
Cc: llvm-dev@lists.llvm.org
Subject: Re: [cfe-dev] GN build roundtable summary; adding GN build files
to the repo

> Keeping the GN build going would be the responsibility of people using
> it, not of the general LLVM community.

If these files exist in the repo, I could imagine that reviewers will
reject patches that might/likely break the gn files.

I could also imagine that contributors will be pressured to fix breakage
of gn files that they introduce, or commits being reverted etc.

So, I'm not sure that a policy of 'it's ok to break this stuff' will
actually work.

If there are public bots using GN, which I think Nico said some people
wanted to do, that really won't work. (Private bots that don't complain
to the general user community are a different story.)
--paulr

7

first things first: If you're happy with cmake, you can stop reading now. Nobody is proposing that LLVM moves off cmake, and nobody is proposing anything that's causing people using cmake more work.

At the LLVM conference, I gave a lightning talk [1] about using GN [2] to build LLVM and clang. cmake is great for many use cases, but in my opinion local day-to-day development isn't one of them. So I wrote GN build files for LLVM and clang, enough to make `ninja check-llvm check-clang check-lld` build everything needed for these three test suites and that all tests pass. This works on Linux, Mac, Win hosts targeting X86, ARM, AArch64. You can see them at [3].

I had been worried that it would be a lot of work to keep the build files up to date, but I've been using this for all my LLVM/clang/lld development the last 8 months, and it turned out to not be a big problem -- LLVM's build files don't change very often, and GN build files are a pleasure to work with in my opinion.

I gave the lightning talk just to talk about my personal workflow, but there was a lot of interest. We had a roundtable on the next day, and about 20 people said they'd be interested in using this for their development too. The main request was that the .gn files are checked in upstream, so that we can collaborate on keeping them working. Two to three orgs even said they'd be interested in moving their buildbots to GN.

As mentioned at the top, the intention here is not to replace cmake, only to offer an opt-in alternative for people who are interested in it. Keeping the GN build going would be the responsibility of people using it, not of the general LLVM community. If this fails to find use and bitrots, we can easily remove it again.

Are there any concerns with checking in GN files? I've put some initial docs for the GN build at ⚙ D53944 Initial documentation for the GN build. ; it describes what the GN build is and is not, what its advantages are (speed and easier configurability), and some points about the philosophy behind the LLVM GN build.

If folks are generally ok with GN files living in-tree, I'll start sending patches for gradually adding gn files through the regular review process.

If having a BUILD.gn file in every directory being confusing is a concern, GN has the concept of a "secondary tree" so that all GN files could be below llvm/gn/tree/{llvm,clang,lld,...}.

So maintain it in a separate repository.

Seriously, I am also working on another project (Mesa) which has a proliferation of build systems for historical and cultural (Android...) reasons, and it sucks.

Besides, how on earth are you using LLVM that cmake times are an issue? Given that cmake times seem to be the only argument you have in favor of GN...

I just measured the typical case (for LLVM only, debug build, all non-experimental targets enabled) by touching all files. cmake takes 5 seconds to run, while the build as a whole takes about 5 minutes. (This is on Ubuntu 18.10, in case that matters.)

Cheers,
Nicolai

8

I am using CMake with LLVM with all targets OFF except for x86. Only clang, lldb, and lld enabled.

$ timecmd cmake …......\llvm-mono\llvm

command took 0:0:34.05 (34.05s total)

9

And by the way, that is with my cache already populated. Here it is on a fresh build directory where it has to generate everything. (this time release build)

$ timecmd cmake -G Ninja -DLLVM_ENABLE_PROJECTS=clang;lld;lldb -DLLVM_TARGETS_TO_BUILD=X86 -DCMAKE_BUILD_TYPE=Release …......\llvm-mono\llvm

command took 0:0:57.42 (57.42s total)

This is on a 48-core machine btw, with an SSD.

10

FWIW, this is consistent with my experience as well.

I also wrote a presubmit script that I run locally to build/test the cross-product of:

COMPILER={gcc+gold, clang+lld}
MODE={debug, release-with-assertions}
TARGETS={llvm+clang+lld+compiler-rt, compiler-rt(standalone)}

While the build/test times are helped by using ninja, the CMake config times are less than optimal.

It’s bad enough that I think I’ll give gn a whirl if the files were in the project(s) too.

11

It is possible that we have features/complexity in our CMakeLists.txt which is not needed. The CMake files are not using modern cmake features. Maybe it's time to change that. It might simplify the buildsystem a bit.

Thanks,

Stephen.

12

I'll volunteer to maintain the GN files if the community is in favor
of having those in the tree. I mentioned that I'd consider using GN
build on bots (and Nico said the same for Chromium) during the
roundtable discussion, but what I meant was for our downstream bots.
If there are upstream bots using GN, they should be FYI only bots (I
don't know if there are any such bots in LLVM already) that don't send
out any notifications on breakages.

There are other advantages to GN but those are not immediately
obvious. GN has a first-class concept of a toolchain and it allows
"applying" these toolchains on dependencies, so you can express the
fact that a particular dependency should be built with a different
toolchain, which could be a completely different compiler or just a
different set of flags. This is an extremely powerful concept that
would be useful for things like runtimes build. Right now, those
require re-running CMake multiple times for each target you want to
build runtimes for, so take the number from above responses and
multiply that by the number of targets you're supporting in your
toolchain and suddenly it may become significant (it definitely is in
our case).

Regarding the experimental status, I think we need to get the GN build
to the point where it provides parity with the CMake build and handles
all common use cases and workflows before we can have that discussion.

13

That is certainly possible, however if we’re going to re-write the entire build system anyway…

One thing to think about is that generated IDE projects from CMake are less than ideal. In fact I actually made a post about this a few weeks ago. In MSVC I consider them barely usable. I can edit files and get code completion with them, but that’s about it. Navigation is painful at best (can take several up to 10 seconds just to open a file), and everything is extremely slow because MSVC is churning away trying to process its build dependency graph. Occasionally things just stop working for 10-20 seconds at a time.

I’ve heard that Xcode is also barely usable but can’t speak for this myself.

Is this any better with gn-generated IDE projects? I actually don’t know! Because I’ve never used them. But it’s at least something to think about.

14

I'll volunteer to maintain the GN files if the community is in favor
of having those in the tree. I mentioned that I'd consider using GN
build on bots (and Nico said the same for Chromium) during the
roundtable discussion, but what I meant was for our downstream bots.
If there are upstream bots using GN, they should be FYI only bots (I
don't know if there are any such bots in LLVM already) that don't send
out any notifications on breakages.

There are other advantages to GN but those are not immediately
obvious. GN has a first-class concept of a toolchain and it allows
"applying" these toolchains on dependencies, so you can express the
fact that a particular dependency should be built with a different
toolchain, which could be a completely different compiler or just a
different set of flags. This is an extremely powerful concept that
would be useful for things like runtimes build. Right now, those
require re-running CMake multiple times for each target you want to
build runtimes for, so take the number from above responses and
multiply that by the number of targets you're supporting in your
toolchain and suddenly it may become significant (it definitely is in
our case).

This.

Although I don’t use GN myself, it would be great if we can somehow port the compiler-rt unit-test generation process to use the “just-recently-built toolchain”, it would break up the non-parallelism of those actions.

Regarding the experimental status, I think we need to get the GN build
to the point where it provides parity with the CMake build and handles
all common use cases and workflows before we can have that discussion.

+1 from me.

-- Dean

15

That is certainly possible, however if we’re going to re-write the entire build system anyway…

One thing to think about is that generated IDE projects from CMake are less than ideal. In fact I actually made a post about this a few weeks ago. In MSVC I consider them barely usable. I can edit files and get code completion with them, but that’s about it. Navigation is painful at best (can take several up to 10 seconds just to open a file), and everything is extremely slow because MSVC is churning away trying to process its build dependency graph. Occasionally things just stop working for 10-20 seconds at a time.

One of the advantages of clang being in a separate repo which can be built independent of llvm is that both are then in smaller solutions. The clang build would treat llvm as an external dependency and not have it’s cpp files in the solution.

I know we’re moving to a monorepo and VS solution size is not part of that consideration, so that size reduction won’t be possible anymore.

I don’t know if there are other ways to reduce solution size. I also haven’t hit performance problems related to VS, though I only build llvm/clang/cte on windows, I don’t do development on it.

I can see if I can reproduce the kinds of problems you see tomorrow. Can you give me a got URL and cake command line to repro exactly your experience?

Thanks,

Stephen.

16

I’ll volunteer to maintain the GN files if the community is in favor
of having those in the tree.

…

GN has a first-class concept of a toolchain and it allows
“applying” these toolchains on dependencies, so you can express the
fact that a particular dependency should be built with a different
toolchain, which could be a completely different compiler or just a
different set of flags. This is an extremely powerful concept that
would be useful for things like runtimes build.

I think your use case and Nico’s are a bit different. In the fullness of time, I expect that you will show that gn’s toolchain support works really well, and more people will want to use it for similar reasons. Eventually, consensus will grow that it should be an officially supported build system. I don’t doubt that you are volunteering to maintain it yourself for now, but I want to acknowledge that there is a very real possibility of scope creep. If runtimes are in scope for this, then that really means the gn files are going to be a full second build system with support for making a real toolchain package. At some point, we may wind up with a second system just as complicated as cmake, although it’ll be much faster.

After writing the paragraph above, I realized that one of the big costs of cmake is iteration time. If you want to fix a build system problem today, you have to run cmake as part of your inner development loop. This incentivizes people to avoid touching the build system if at all possible, and it fills up with little quick fixes that nobody wants to refactor. If we were using gn, I would be much happier iterating on changes to the build system, since I would be able to test small incremental changes quickly.

I guess my opinion is, LLVM is big enough to have a second build system in tree, with all the costs that that implies. gn is really the only new build system that I’m actually excited about, so if we’re going to have a second one, it should be this one.

17

I think this might be easier when we finish the move to the monorepo, and can better coordinate/simplify the build across projects. I certainly think it’s worth doing, even if we end up with GN and CMake files in the project.

I’m personally not aware of the “modern” CMake patterns. Do you have a reference online that interested individuals might be able to casually approach?

Cheers

-- Dean

18

FWIW, this is consistent with my experience as well.
I also wrote a presubmit script that I run locally to build/test the cross-product of:
COMPILER={gcc+gold, clang+lld}
MODE={debug, release-with-assertions}
TARGETS={llvm+clang+lld+compiler-rt, compiler-rt(standalone)}
While the build/test times are helped by using ninja, the CMake config times are less than optimal.

It is possible that we have features/complexity in our CMakeLists.txt which is not needed. The CMake files are not using modern cmake features. Maybe it’s time to change that. It might simplify the buildsystem a bit.

I think this might be easier when we finish the move to the monorepo, and can better coordinate/simplify the build across projects. I certainly think it’s worth doing, even if we end up with GN and CMake files in the project.

I’m personally not aware of the “modern” CMake patterns. Do you have a reference online that interested individuals might be able to casually approach?

Here is a talk with unfortunate audio quality:

https://steveire.wordpress.com/2017/11/05/embracing-modern-cmake

And documentation:

https://cmake.org/cmake/help/latest/manual/cmake-buildsystem.7.html

The big topic that llvm does not take advantage of is usage requirements on targets as documented there.

Thanks,

Stephen.

19

>
>> FWIW, this is consistent with my experience as well.
>> I also wrote a presubmit script that I run locally to build/test the cross-product of:
>> COMPILER={gcc+gold, clang+lld}
>> MODE={debug, release-with-assertions}
>> TARGETS={llvm+clang+lld+compiler-rt, compiler-rt(standalone)}
>> While the build/test times are helped by using ninja, the CMake config times are less than optimal.
>
>
> It is possible that we have features/complexity in our CMakeLists.txt which is not needed. The CMake files are not using modern cmake features. Maybe it's time to change that. It might simplify the buildsystem a bit.
>

I think this might be easier when we finish the move to the monorepo, and can better coordinate/simplify the build across projects. I certainly think it’s worth doing, even if we end up with GN and CMake files in the project.

I’m personally not aware of the “modern” CMake patterns. Do you have a reference online that interested individuals might be able to casually approach?

Here is a talk with unfortunate audio quality:

Embracing Modern CMake | Steveire's Blog

Thanks — I’ll try and listen/watch through this in the background.

And documentation:

cmake-buildsystem(7) — CMake 3.28.0-rc5 Documentation

The big topic that llvm does not take advantage of is usage requirements on targets as documented there.

Interesting.

I don’t see anything here that’s popping out to indicate that there’s something new/modern compared to the CMake files I’ve read/written before.

I may need to read more about this.

Thanks for the links!

Cheers

-- Dean

20

Hello!

At the LLVM conference, I gave a lightning talk [1] about using GN [2] to build LLVM and clang. cmake is great for many use cases, but in my opinion local day-to-day development isn't one of them. So I wrote GN build files for LLVM and clang, enough to make `ninja check-llvm check-clang check-lld` build everything needed for these three test suites and that all tests pass. This works on Linux, Mac, Win hosts targeting X86, ARM, AArch64. You can see them at [3].
(...)
Are there any concerns with checking in GN files? I've put some initial docs for the GN build at ⚙ D53944 Initial documentation for the GN build. ; it describes what the GN build is and is not, what its advantages are (speed and easier configurability), and some points about the philosophy behind the LLVM GN build.

My personal experience with GN - last time I tested it - was that it was impossible to
build from source as the build procedure was anything but straight-forward and pretty
much architecture-locked in.

And I just tried it again on a POWER machine:

glaubitz@redpanda:/srv/glaubitz/gn$ python build/gen.py
Installing Debian root image from https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/1015a998c2adf188813cca60b558b0ea1a0b6ced/debian_sid_amd64_sysroot.tar.xz
Downloading https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/1015a998c2adf188813cca60b558b0ea1a0b6ced/debian_sid_amd64_sysroot.tar.xz
glaubitz@redpanda:/srv/glaubitz/gn$

Jepp, hasn't changed. If building your build system involves downloading a Debian
chroot with a hard-coded architecture, you're doing something wrong. And from my personal
experience with Google upstream, you can outright forget that they would accept patches
for any target that they are not using themselves.

I really wouldn't recommend using GN. It might be fast, but the whole design seems
to be tailored to Google's own needs.

Adrian