wasteful cmake defaults

Hi all

I wanted to do a quick build of a clean branch yesterday and noticed
something surprising in the configure log:

-- No build type selected, default to Debug

It appears that llvm's configuration forces Debug builds if the user
does not specify the build type.

    https://github.com/llvm/llvm-project/blob/9218ff50f93085d0a16a974db28ca8f14bc66f64/llvm/CMakeLists.txt#L57-L60

I've just done a build of llvm and clang 10 in debug mode for X86 and
ARM targets and it weighs in at a whopping 75GiB. I feel that forcing
Debug builds in the absence of an option to be a wasteful default. It is
a valid and useful thing to call cmake without specifying a build type;
absence of a command line switch does not always imply absence of a
choice.

I took some measurements. The machine under test is an otherwise
unloaded Ryzen 2 server with 6 cores / 12 threads and fast NVME SSDs
running Debian sid with gcc-10.2 as the host toolchain.

    $ cmake ../llvm -DCMAKE_BUILD_TYPE=Debug -GNinja \
        -DLLVM_TARGETS_TO_BUILD=X86 -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
    ...
    $ time ninja llc
    ...
    [1199/1199] Linking CXX executable bin/llc
    ninja llc 6858.95s user 218.53s system 1095% cpu 10:45.78 total

and then:

    $ git apply <<'EOF'
    diff --git a/llvm/CMakeLists.txt b/llvm/CMakeLists.txt
    index 21563e15118..3f5821351a1 100644
    --- a/llvm/CMakeLists.txt
    +++ b/llvm/CMakeLists.txt
    @@ -55,8 +55,8 @@ set(CMAKE_CXX_STANDARD_REQUIRED YES)
     set(CMAKE_CXX_EXTENSIONS NO)

      if (NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES)
      - message(STATUS "No build type selected, default to Debug")
      - set(CMAKE_BUILD_TYPE "Debug" CACHE STRING "Build type (default Debug)" FORCE)
      + # message(STATUS "No build type selected, default to Debug")
      + # set(CMAKE_BUILD_TYPE "Debug" CACHE STRING "Build type (default Debug)" FORCE)
       endif()

        # Side-by-side subprojects layout: automatically set the
    EOF

    $ cmake ../llvm -DCMAKE_BUILD_TYPE= -GNinja -DLLVM_TARGETS_TO_BUILD=X86 \
        -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
    ...
    $ time ninja llc
    [1199/1199] Linking CXX executable bin/llc
    ninja llc 5714.52s user 159.31s system 1125% cpu 8:41.99 total

There's two minutes difference just for an X86 capable `llc` target on
this machine. I imagine there are plenty of configurations in which the
ratio is *much* larger e.g. machines with slow disks and lots of cores.

I think this speedup will be especially useful in CI systems where
getting a compiler build quickly is more important than getting a quick
compiler. I don't imagine we ever use the debuginfo generated by the
host toolchain in the buildbots, either.

Is there anything I'm missing, or can this override be safely deleted?
Would a patch to Zorg be needed / appropriate for the fast buildbots?

All the Best

Luke

Just trying to understand: Are you suggesting a way to have a “-O0” build without all the debug information?

-Min

Exactly.

Hi all

I wanted to do a quick build of a clean branch yesterday and noticed
something surprising in the configure log:

> -- No build type selected, default to Debug

It appears that llvm's configuration forces Debug builds if the user
does not specify the build type.

    https://github.com/llvm/llvm-project/blob/9218ff50f93085d0a16a974db28ca8f14bc66f64/llvm/CMakeLists.txt#L57-L60

I've just done a build of llvm and clang 10 in debug mode for X86 and
ARM targets and it weighs in at a whopping 75GiB. I feel that forcing
Debug builds in the absence of an option to be a wasteful default. It is
a valid and useful thing to call cmake without specifying a build type;
absence of a command line switch does not always imply absence of a
choice.

I don't think the current state of things suggests that it's not a
valid use case to call cmake without a build type - the question is
what users are we targeting for that use case? Is it new developers to
the project? New CI configurations? etc.

I believe historically it's been setup for new developers - debug
info, assertions enabled (I think?), etc.

CI configurations are likely more deliberately chosen for long-term
deployment and balancing a variety of tradeoffs, so I don't think
tailoring the default to that audience is high value compared to other
audiences/users.

What is such a build useful for the average user? If changing the
default, I think that CMAKE_BUILD_TYPE=Release is what most
non-developer users want.

You mention CI systems, but I'd expect CI maintainers to know they can
set CMAKE_CXX_FLAGS_RELEASE. Does -O0 even build faster than -O1?

Michael

I’m curious how much of this is due to not using LLD? Last I checked, using ld.bfd instead of LLD (or gold) was one of my major build time issues, and somehow our default configuration would use ld.bfd.

– Sean Silva

Yeah, that's one I'd be in favor of fixing, if it's still the case.

Hi,

The CMake default is an empty build type, which yield a non-optimized build without debug information and without assertions, it isn’t clear that it is really good for most (developers likely want debug info and assertions in non-optimized builds, non developers may want a release build).
See also the revision that introduced the current behavior: http://reviews.llvm.org/D7360

We could make Release the default I guess, the main drawback I see right now is that it could affect bots that are relying on the default and wouldn’t get assertions anymore?
That said our upstream buildbots infra seems to have explicit defaults: https://github.com/llvm/llvm-zorg/blob/master/zorg/buildbot/builders/UnifiedTreeBuilder.py#L125-L127

For LLD, I don’t think we have auto-detection, but I’d be on-board with auto-detecting it and have -DLLVM_ENABLE_LLD default to the auto-detection when empty. But that’s another topic…

I think you are conflating two things somewhat here:

  - Is it useful to do an -O0 build without assertions or debug info?
  - What is the most useful default build configuration?

These are somewhat separable. As others have said, the first can be achieved by specifying a release build with -O0 in the CXXFLAGS. This is honestly not something I've ever considered doing, because the extra time to run a handful of tests at -O0 is likely to offset any build-time improvements. It's useful if you want to get an LLVM binary quickly, but not so useful if you want to actually run it. If you run it and it doesn't work, then you'll still need to do a debug build to get useful debug information. As such, I'm not sure that I understand the use case.

In terms of the most useful build configuration to be the default, I think there are a bunch of users that we need to consider:

  - Developers of LLVM
  - Developers of downstream projects that use LLVM
  - Package builders
  - CI admins.

Some of these are easy. CI admins are likely to want to configure their builds explicitly and so won't want to use the defaults. Package builders are going to want a Release build, but generally put that in their configuration for the package file and so don't care too much what the default is.

Developers of LLVM and downstream projects are similar. Downstream projects come in two flavours:

  - Ones that depend on an LLVM release version existing
  - Ones that ship their own LLVM as a submodule.

The second is less of a problem because it will be driving the LLVM build from its own build infrastructure and so has a one-time cost for configuration and anyone building that project inherits the benefits.

The first is more of a problem because build configurations of LLVM change its ABI, so you need to match the LLVM build to the downstream project's build configuration. This is generally fairly easy to do by just pointing at the llvm-config built by the specific LLVM output, so you can easily have Debug / Release / Whatever builds of your project using different llvm-config versions. Supporting this means that you are already propagating config from LLVM and people building your project in anything other than a relase configuration need to do a non-default build of LLVM.

This leaves LLVM developers. I'd assume that these are the people building LLVM manually the most often and so it makes sense to provide them with a default that makes sense. Unfortunately, there isn't a single one that makes sense for developers.

For the last few years, I've been following Chandler's suggestion and building a Release + Asserts build and a Debug + ASan + expensive checks build whenever I work on LLVM. Release with asserts is reasonably fast running the tests and fast to build, so a compile-test cycle is not too bad. When that fails, I can switch to the other build and get something that's easy to debug. I also generally try to run the entire test suite in this configuration once before merging to master because it picks up memory management bugs that may not actually break anything in the (simple) test cases.

The default isn't for either of those. I'd be quite happy for the default to be either Debug + ASan + expensive tests or Release + Asserts, but given that I have to manually configure one of them, there's limited value in needing less manual configuration for the other.

Perhaps the right solution is for the build to fail and tell people to pick one if they don't specify CMAKE_BUILD_TYPE.

David

Using lld or gold *does* improve build times in my experience. Combining
a -O0 build with a fast linker would likely shave off enough few
minutes too. This doesn't preclude the issue I'm talking about. It's a
separate albeit valid speed improvement.

All the Best

Luke

>
>
> > Just trying to understand: Are you suggesting a way to have a “-O0”
> > build without all the debug information?
> >
> Exactly.

What is such a build useful for the average user? If changing the
default, I think that CMAKE_BUILD_TYPE=Release is what most
non-developer users want.

You're probably right, in that if I just want to build e.g. clang from
source because my distro package manager contains an old version and I'm
not interested in hacking on it myself, then a release build makes
sense. However, I think this is a violation of expectations in that this
is overriding a *cmake* variable. It takes control away from the user.
cmake probably has something to answer for here, because there's not an
explicit "empty" buildtype like e.g. meson's `--buildtype plain`, but
only the absense of a value.

Perhaps we could find a middle ground. e.g.
LLVM_DONT_TWEAK_CMAKES_BUILD_TYPE=ON which would allow fast builds, but also
enable us to mess with the cmake default to please new users?

You mention CI systems, but I'd expect CI maintainers to know they can
set CMAKE_CXX_FLAGS_RELEASE. Does -O0 even build faster than -O1?

On my configuration, yes: -O1 is slower than -O0

    $ cmake ../llvm -DCMAKE_BUILD_TYPE=Release
    -DCMAKE_CXX_FLAGS_RELEASE=-O1 -GNinja -DLLVM_TARGETS_TO_BUILD=X86
    -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
    ...

    $ time ninja llc
    [1199/1199] Linking CXX executable bin/llc
    ninja llc 6210.72s user 149.40s system 1145% cpu 9:15.22 total

    It was 8:41.99 total for the -O0 build

Michael

All the Best

Luke

Good points. I guess without data on who builds llvm for what reason,
we can't really make informed choices. I personally think that if you're
setting yourself up for development on llvm itself, you're in for the
long haul anyway, so a rebuild because you forgot
-DCMAKE_BUILD_TYPE=Debug is no sweat. However, if you just want to
install a compiler from source, the total cost of installing and running
a debug build ends up being significantly higher when you miss that
flag. If we *are* going to override cmake's behaviour and set a default,
I think a Release build will be better for the
broader-than-llvm-developer user base and we should upgrade this message
to a warning while we're at it.

All the Best

Luke

From: llvm-dev <llvm-dev-bounces@lists.llvm.org> On Behalf Of David
Chisnall via llvm-dev
Sent: Wednesday, November 18, 2020 6:32 AM
To: llvm-dev@lists.llvm.org
Subject: Re: [llvm-dev] wasteful cmake defaults

>> -- No build type selected, default to Debug
> It appears that llvm's configuration forces Debug builds if the user
> does not specify the build type.
>
> https://urldefense.com/v3/__https://github.com/llvm/llvm-
project/blob/9218ff50f93085d0a16a974db28ca8f14bc66f64/llvm/CMakeLists.txt*
L57-
L60__;Iw!!JmoZiZGBv3RvKRSx!pbrEqQxC9HKXzCXnbVPOezybndBSk6GZShkECVJPUB7Pjax
HX2noFrIRX8qXPuJ15Q$
>
> I've just done a build of llvm and clang 10 in debug mode for X86 and
> ARM targets and it weighs in at a whopping 75GiB. I feel that forcing
> Debug builds in the absence of an option to be a wasteful default. It is
> a valid and useful thing to call cmake without specifying a build type;
> absence of a command line switch does not always imply absence of a
> choice.

I think you are conflating two things somewhat here:

  - Is it useful to do an -O0 build without assertions or debug info?
  - What is the most useful default build configuration?

These are somewhat separable. As others have said, the first can be
achieved by specifying a release build with -O0 in the CXXFLAGS. This
is honestly not something I've ever considered doing, because the extra
time to run a handful of tests at -O0 is likely to offset any build-time
improvements. It's useful if you want to get an LLVM binary quickly,
but not so useful if you want to actually run it. If you run it and it
doesn't work, then you'll still need to do a debug build to get useful
debug information. As such, I'm not sure that I understand the use case.

In terms of the most useful build configuration to be the default, I
think there are a bunch of users that we need to consider:

  - Developers of LLVM
  - Developers of downstream projects that use LLVM
  - Package builders
  - CI admins.

Some of these are easy. CI admins are likely to want to configure their
builds explicitly and so won't want to use the defaults. Package
builders are going to want a Release build, but generally put that in
their configuration for the package file and so don't care too much what
the default is.

Developers of LLVM and downstream projects are similar. Downstream
projects come in two flavours:

  - Ones that depend on an LLVM release version existing
  - Ones that ship their own LLVM as a submodule.

The second is less of a problem because it will be driving the LLVM
build from its own build infrastructure and so has a one-time cost for
configuration and anyone building that project inherits the benefits.

The first is more of a problem because build configurations of LLVM
change its ABI, so you need to match the LLVM build to the downstream
project's build configuration. This is generally fairly easy to do by
just pointing at the llvm-config built by the specific LLVM output, so
you can easily have Debug / Release / Whatever builds of your project
using different llvm-config versions. Supporting this means that you
are already propagating config from LLVM and people building your
project in anything other than a relase configuration need to do a
non-default build of LLVM.

This leaves LLVM developers. I'd assume that these are the people
building LLVM manually the most often and so it makes sense to provide
them with a default that makes sense. Unfortunately, there isn't a
single one that makes sense for developers.

For the last few years, I've been following Chandler's suggestion and
building a Release + Asserts build and a Debug + ASan + expensive checks
build whenever I work on LLVM. Release with asserts is reasonably fast
running the tests and fast to build, so a compile-test cycle is not too
bad. When that fails, I can switch to the other build and get something
that's easy to debug. I also generally try to run the entire test suite
in this configuration once before merging to master because it picks up
memory management bugs that may not actually break anything in the
(simple) test cases.

My own personal default is Release + Asserts + -g1 (line tables only).
This gives tracebacks with source locations on a crash/assert, and is
noticeably faster to build than RelWithDebInfo (the main benefit to the
latter is symbolized tracebacks). As a rule I find logging/printf to
work more effectively than a debugger so I don't usually miss having
debug info; I can always run a debug build if I have to.
--paulr

This is missing the probably largest group: Users of clang who compile
clang themselves (e.g. because their OS does not come with a package
for clang, or is too old)

Michael

Unfortunately CMAKE_* variables is commonplace. Doesn't mean it is
good and should avoid it.
In this case, however, it is not overriding a user setting, but
provide a default one when the user did not specify one explicitly. In
this specific case, IMHO cmake's default-empty CMAKE_BUILD_TYPE is a
common beginner's trap, serves no use case, and is even recommended by
KitWare itself:
https://blog.kitware.com/cmake-and-the-default-build-type/

Michael

I think our target for this group should be that it is size 0. Windows, macOS, and *BSD all have up-to-date packages of the latest version, as do many Linux distros. We provide APT repos for Debian and Ubuntu. What are the large platforms that do not have packages of the latest release and how do we fix that?

David

First, it is not rare that you are working on an LTS release, but need
a newer compiler. I myself am still working on a remote machine using
Red Hat EL7 with gcc 4.8 and no clang pre-installed. Since I am not
the admin of that machine, I have to compile clang (and gcc) with a
prefix in my home directory.

Second, for OpenMP target offloading, we recommend users to compile
the latest trunk because it is under heavy development. In practice,
if you are using target offloading, you have to compile it yourself
because it needs to match the installed version of CUDA (or ROCm) and
I have yet to see a distribution where OpenMP target offloading works
out of the box.

Michael

First, it is not rare that you are working on an LTS release, but need
a newer compiler. I myself am still working on a remote machine using
Red Hat EL7 with gcc 4.8 and no clang pre-installed.

We build nightly packages for Ubuntu LTS releases, what do we need to do to add the same support for Red Hat?

Since I am not
the admin of that machine, I have to compile clang (and gcc) with a
prefix in my home directory.

As I recall, nothing in the LLVM install hard codes the install location. Deb packages are just ar archives, RPMs are just cpio archives, so users without admin access could just grab the latest nightly package for their distro and extract it into their home directory. We could provide a script for that (or, to be more friendly, build a .txz from the same directory tree).

Second, for OpenMP target offloading, we recommend users to compile
the latest trunk because it is under heavy development. In practice,
if you are using target offloading, you have to compile it yourself
because it needs to match the installed version of CUDA (or ROCm) and
I have yet to see a distribution where OpenMP target offloading works
out of the box.

That sounds unfortunate. I presume that there is a long-term plan for fixing this?

David

> Since I am not
> the admin of that machine, I have to compile clang (and gcc) with a
> prefix in my home directory.

As I recall, nothing in the LLVM install hard codes the install
location. Deb packages are just ar archives, RPMs are just cpio
archives, so users without admin access could just grab the latest
nightly package for their distro and extract it into their home
directory. We could provide a script for that (or, to be more friendly,
build a .txz from the same directory tree).

Why is this preferable over compiling from source? We provide
documentation to compile from source, but not how to extract files
from distribution packages and where clang expects them.
Even if one would consider that, there are reasons to compile from
source such as optimizing the binary for the concrete CPU
(-match=native) and not having to trust a 3rd party that the compiled
binary corresponds to the sources (not that I would argue like that,
but I know people who compile from source, including their kernel, on
principle).

> Second, for OpenMP target offloading, we recommend users to compile
> the latest trunk because it is under heavy development. In practice,
> if you are using target offloading, you have to compile it yourself
> because it needs to match the installed version of CUDA (or ROCm) and
> I have yet to see a distribution where OpenMP target offloading works
> out of the box.

That sounds unfortunate. I presume that there is a long-term plan for
fixing this?

I hope. It is somewhat difficult to provide offloading plugins for all
vendors at the same time (Nvidia+AMD+NEC+...) that each also have
different versions.

Michael