RFC: Dropping support for building sanitizers with autotools

Hi everyone,

TL;DR I plan to do subj unless anyone objects.

Autotools support for building sanitizers was never complete, is a proper subset of CMake capabilities, and the gap between the features of CMake build and autotools is quickly increasing:

  1. OS: autotools support Linux, Mac OS X and Android; CMake also supports FreeBSD and Windows.
  2. Architectures: autotools support i386 and x86_64, CMake also supports MIPS, ARM/AArch64, PowerPC).
  3. Library variants: autotools don’t support building shared ASan runtime on Linux.
    4 [!!!]) Autotools build doesn’t have any support for building and running tests.
  4. The sets of compiler flags we use to build runtimes are different in two builds, and are hard to kept in sync.

Because of (4), sanitizer runtimes built with autotools are severely undertested, and maintaining two different build systems is a burden I would like to get rid of. Now seems to be a good time for that: we’ve cut off 3.6 release branch, and Hans assures me that 3.7 release process will use CMake.

Are there existing users of configure/make that want sanitizers to be available in their build? Are there other arguments for keeping the autotools build “working” I’m missing?

+1, ship it.

From: "Chandler Carruth" <chandlerc@google.com>
To: "Alexey Samsonov" <vonosmas@gmail.com>
Cc: "Nico Weber" <nicolasweber@gmx.de>, "Clang Developers List" <cfe-dev@cs.uiuc.edu>, "LLVM Dev"
<llvmdev@cs.uiuc.edu>
Sent: Thursday, February 12, 2015 8:41:18 PM
Subject: Re: [LLVMdev] RFC: Dropping support for building sanitizers with autotools

Hi everyone,

TL;DR I plan to do subj unless anyone objects.

Autotools support for building sanitizers was never complete, is a
proper subset of CMake capabilities, and the gap between the
features of CMake build and autotools is quickly increasing:
1) OS: autotools support Linux, Mac OS X and Android; CMake also
supports FreeBSD and Windows.
2) Architectures: autotools support i386 and x86_64, CMake also
supports MIPS, ARM/AArch64, PowerPC).
3) Library variants: autotools don't support building shared ASan
runtime on Linux.
4 [!!!]) Autotools build doesn't have *any* support for building and
running tests.
5) The sets of compiler flags we use to build runtimes are different
in two builds, and are hard to kept in sync.

Because of (4), sanitizer runtimes built with autotools are severely
undertested, and maintaining two different build systems is a burden
I would like to get rid of. Now seems to be a good time for that:
we've cut off 3.6 release branch, and Hans assures me that 3.7
release process will use CMake.
+1, ship it.

I agree. I've found the autotools building support fairly useless precisely because there's no way to run the tests.

-Hal

I’ve used the autotools support to ensure that it still builds after llvm changes, but we have enough testers that I don’t really think it’s an issue.

-eric

Sounds fine to me, but be aware that some of the libraries in compiler-rt other than the sanitizers currently are only built correctly in the autotools build (for example, I think libprofile_rt for iOS isn’t built correctly in the cmake build).

Sounds fine to me, but be aware that some of the libraries in compiler-rt
other than the sanitizers currently are only built correctly in the
autotools build (for example, I think libprofile_rt for iOS isn't built
correctly in the cmake build).

Sure, I don't plan to remove support for builtins and libprofile.

+1 (in case you still need mine)

Unfortunately, we do rely on the Autotools to build for darwin.

The lack of proper testing is very unfortunate. In the future, we will either drop the dependency on Makefile/autoconf build or add the support for sanitizer testing.

Anna.

Do you have an ETA on this effort?

-eric

Unfortunately, we do rely on the Autotools to build for darwin.

The lack of proper testing is very unfortunate. In the future, we will
either drop the dependency on Makefile/autoconf build or add the support
for sanitizer testing.

Sorry, for some reason I had the impression that you're also more
interested in CMake buiild :-/. Only now I noticed that recent Kuba's
patches about -isysroot fix both makefile and CMake
build system (and he's probably not very happy about it).

I believe the effort required to port sanitizer unit tests to autotools is
not worth it, and it's better to spend that time migrating the release
tools to CMake. Keeping two build systems "working"
was and is a manageable, but unpleasant burden. Providing a good test
coverage for two runtime versions is hard, and keeping two runtime versions
"equivalent" (built with the same compiler flags)
is close to impossible.

I understand that removing autotools support right now breaks your
integration. What if we just delete autotools support from all Linux builds?

Speaking of autotools support, I should lay out the reason why I raised the
question of removing it last week. We plan to make intrusive changes to
ASan and UBSan library layout:
1) embed UBSan runtime into ASan to solve various initialization and
integration problems of using ASan+UBSan (fixing
http://llvm.org/bugs/show_bug.cgi?id=21112)
2) make UBSan runtime standalone, and forbid to use it with another
sanitizers except for ASan.
Unfortunately, it means changing a bunch of build rules in *both* CMake and
makefile build.

Ideally, I would be happy to transfer ownership of current autotools build
on Darwin to you, and ask for help in implementing/testing necessary
changes on Mac OS X side :slight_smile:
At the moment we don't even have a live buildbot running autotools-built
ASan on Mac, so you're in much better shape to catch and notice regressions.

Unfortunately, we do rely on the Autotools to build for darwin.

The lack of proper testing is very unfortunate.

This isn't just "proper testing". This is any testing at all. I have to
guess that you're not using it for *development*, just using it as part of
some release process build system?

If that's the case, I think it would be better for you to maintain whatever
you need to support that in the interim out-of-tree. If the tests don't
run, I don't think it's really reasonable to keep a build system in tree
and expect open source contributors to update it and keep it working. How
will they even know if it is working? We don't even have build bots that
cover it as far as I know? (If we do, that somewhat lessens this pain...)

In the future, we will either drop the dependency on Makefile/autoconf

build or add the support for sanitizer testing.

Given the push to move away from autotools altogether, it would seem really
weird to add support for testing to it rather than migrating the dependency
away. You can even use the cmake build *from* an autoconf/Makefile setup by
running cmake directly. It's a bit annoying, but it can be done.

We do have buildbots, e.g.:
http://lab.llvm.org:8080/green/view/Clang/job/clang-stage1-configure-RA_build/

There are still a lot of issues blocking full CMake equivalence with autoconf/Makefile (PR15732). Removing the ASan makefile support is really premature.

But there is no support? How can you build something without testing it?

I totally get why we can't remove autoconf/Makefile support for everything
else. It's the sanitizer support I don't understand.

Unfortunately, we do rely on the Autotools to build for darwin.

The lack of proper testing is very unfortunate. In the future, we will either drop the dependency on Makefile/autoconf build or add the support for sanitizer testing.

Sorry, for some reason I had the impression that you’re also more interested in CMake buiild :-/. Only now I noticed that recent Kuba’s patches about -isysroot fix both makefile and CMake
build system (and he’s probably not very happy about it).

I believe the effort required to port sanitizer unit tests to autotools is not worth it, and it’s better to spend that time migrating the release tools to CMake. Keeping two build systems “working”

was and is a manageable, but unpleasant burden. Providing a good test coverage for two runtime versions is hard, and keeping two runtime versions “equivalent” (built with the same compiler flags)
is close to impossible.

I agree. The solution where we only rely on cmake build would be the best option.

I understand that removing autotools support right now breaks your integration. What if we just delete autotools support from all Linux builds?

Yes. That is fine by me.

Speaking of autotools support, I should lay out the reason why I raised the question of removing it last week. We plan to make intrusive changes to ASan and UBSan library layout:

  1. embed UBSan runtime into ASan to solve various initialization and integration problems of using ASan+UBSan (fixing http://llvm.org/bugs/show_bug.cgi?id=21112)
  2. make UBSan runtime standalone, and forbid to use it with another sanitizers except for ASan.
    Unfortunately, it means changing a bunch of build rules in both CMake and makefile build.

Ideally, I would be happy to transfer ownership of current autotools build on Darwin to you, and ask for help in implementing/testing necessary changes on Mac OS X side :slight_smile:
At the moment we don’t even have a live buildbot running autotools-built ASan on Mac, so you’re in much better shape to catch and notice regressions.

We do have public buildbots set up for OS X:

Here is one that is testing a makefile build. There are probably others in different configurations.

http://lab.llvm.org:8080/green/view/Clang/job/clang-stage1-configure-RA_build/

We also have a buildbot running cmake tests on OS X:
http://lab.llvm.org:8080/green/view/All/job/clang-stage1-cmake-RA_check/

I may be missing something obvious (the jenkins UI is very unfamiliar to
me) but I don't see this build bot failing at any time because of this
change... Is it actually building the ASan runtimes?

The change has been reverted shortly after the initial commit to allow for more discussions. This is why the bot is green. You can see that the asan library is being built if you choose “Console Output” menu and click “full log”, search for “libclang_rt.asan_osx_dynamic.dylib”.

I don’t think there is a specific reason why we need the sanitizers to be built with autotools. I agree that the current setup is broken in several ways and needs attention. Thank you for the suggestion to run cmake from an autoconf/Makefile setting; we’ll investigate if that could be an acceptable solution.

Anna.

Hi Alexey,

I’d like to be pinged when you start doing that refactoring, just in case I miss it.

Thanks

Unfortunately, we do rely on the Autotools to build for darwin.

The lack of proper testing is very unfortunate. In the future, we will
either drop the dependency on Makefile/autoconf build or add the support
for sanitizer testing.

Sorry, for some reason I had the impression that you're also more
interested in CMake buiild :-/. Only now I noticed that recent Kuba's
patches about -isysroot fix both makefile and CMake
build system (and he's probably not very happy about it).

I believe the effort required to port sanitizer unit tests to autotools is
not worth it, and it's better to spend that time migrating the release
tools to CMake. Keeping two build systems "working"
was and is a manageable, but unpleasant burden. Providing a good test
coverage for two runtime versions is hard, and keeping two runtime versions
"equivalent" (built with the same compiler flags)
is close to impossible.

I agree. The solution where we only rely on cmake build would be the best
option.

I understand that removing autotools support right now breaks your
integration. What if we just delete autotools support from all Linux builds?

Yes. That is fine by me.

OK, I've removed the support for sanitizers in makefile/autoconf builds on
Linux in r229754-r229756.

It is possible to build LLVM & Clang with autotools and compiler-rt with cmake?

Currently, dropping the autotools support is breaking the build of compiler-rt for llvm.org/apt/ and
the Debian & Ubuntu packaging.

I agree that the autotools support in compiler-rt was limited but it was doing the job correctly...

Could you consider reverting this changes?

Thanks,
Sylvestre
PS: I don't mind switching to cmake but here are still a few blocking bugs:
http://llvm.org/bugs/show_bug.cgi?id=15732

Can you point at which open bugs in that list actually affect packaging for
Debian? I assume http://llvm.org/pr15493, which is about building a
monolithic libLLVM-N.M.so, is the main one and nothing else matters.

When packaging, do you build all the LLVM subprojects together or
separately one after another? If you build separately, you should be able
to build compiler-rt with cmake and keep building LLVM with autoconf.