Slowness of clang build

I am periodically retrieving the latest sources from the llvm/clang repository, as specified in the Getting Started page.

When I do a build of clang, through the 'make' command from the build directory, the build runs very slow. Evidently the default is a Debug+Assert build. Compiling a single source file averages 5-10 seconds and overall the build takes hours considering how many modules must be built. It is all successful in the end and when I am finished I have the latest clang, which is great.

I have plenty of memory, plenty of disk space, a fairly fast CPU, and am doing the build on Fedora 17 with hardly any other applications running.

Why is compiling clang so slow ? Is there any way to speed up the build ? If I do a release build ( how ? )is it significantly quicker ?

I am periodically retrieving the latest sources from the llvm/clang
repository, as specified in the Getting Started page.

When I do a build of clang, through the 'make' command from the build
directory, the build runs very slow. Evidently the default is a Debug+Assert
build. Compiling a single source file averages 5-10 seconds and overall the
build takes hours considering how many modules must be built. It is all
successful in the end and when I am finished I have the latest clang, which
is great.

I have plenty of memory, plenty of disk space, a fairly fast CPU, and am
doing the build on Fedora 17 with hardly any other applications running.

Could you be more specific about the CPU, cores, and available memory?
(have you watched the utilization while building Clang? Is the build
topping out on RAM/using swap? using all your CPU cores?)

The simplest thing to do is probably to pass "-j N" where N is the
number of cores you have available (by default the make build isn't
parallelized).

Other ideas:
* what compiler are you using to perform the build? If it's a previous
Debug+Asserts build of Clang, then that's really going to slow things
down - don't use that, use Release+Asserts or Release-Asserts.
* you could try the cmake+ninja build (ninja will run maximally
parallel by default & exploits more parallelism than the make build)
* to do a release build of clang (which might speed things up a little
since you won't be dealing with debug info, etc) I think it's
"--disable-optimized" when configuring the make based build

I am periodically retrieving the latest sources from the llvm/clang repository, as specified in the Getting Started page.

When I do a build of clang, through the 'make' command from the build directory, the build runs very slow.

Are you doing a parallel build? On an i7 laptop, I can do a clean build in 5-10 minutes, with -j4, on a faster desktop it's about 3 minutes. I usually use ninja (just add -G Ninja to your cmake command), rather than make, although there isn't much difference in performance between the two when doing a clean build. For incremental builds, ninja is so much faster that make isn't even in the same league.

Evidently the default is a Debug+Assert build. Compiling a single source file averages 5-10 seconds and overall the build takes hours considering how many modules must be built. It is all successful in the end and when I am finished I have the latest clang, which is great.

The Debug+Asserts build is usually faster to compile, because it turns off optimisations, but slower to link, because the debug info is very large. While it can take 5-10 seconds to compile each file, that only matters if you're doing one file at a time.

I have plenty of memory, plenty of disk space, a fairly fast CPU, and am doing the build on Fedora 17 with hardly any other applications running.

ninja -j8, if you've got enough RAM. If you're not on an SSD, then more processes than you have cores is generally a good idea, because it lets some run while the other are waiting for the disk, but be careful not to have to many more than you have GBs of RAM. A few of the files take about 1GB to compile, and for a debug build with ld-bdf you'll go over 2GB for the final linking step (enabling shared libraries speeds this up, but results in about a 5% slower executable at the end on x86-64).

Why is compiling clang so slow ? Is there any way to speed up the build ? If I do a release build ( how ? )is it significantly quicker ?

I've not found the compilation time to be much different for release and debug builds (I've not measured it, but I do both fairly regularly and neither feels much slower than the other). The big difference is the time spent linking. If the linker runs out of physical memory and starts to swap then linking time can easily hit 5+ minutes. I've seen it go over 2GB with a statically-linked debug+asserts build of clang.

If you're doing a bootstrap build, make sure that you do a release build first, because the builds with asserts run a lot slower (factor of 10 or so), and trying to build LLVM/Clang with a build with asserts enabled is quite painful.

David

I am periodically retrieving the latest sources from the llvm/clang
repository, as specified in the Getting Started page.

When I do a build of clang, through the 'make' command from the build
directory, the build runs very slow. Evidently the default is a Debug+Assert
build. Compiling a single source file averages 5-10 seconds and overall the
build takes hours considering how many modules must be built. It is all
successful in the end and when I am finished I have the latest clang, which
is great.

I have plenty of memory, plenty of disk space, a fairly fast CPU, and am
doing the build on Fedora 17 with hardly any other applications running.

Could you be more specific about the CPU, cores, and available memory?

I have 8 GB of memory and 4 CPUs.

(have you watched the utilization while building Clang? Is the build
topping out on RAM/using swap? using all your CPU cores?)

The simplest thing to do is probably to pass "-j N" where N is the
number of cores you have available (by default the make build isn't
parallelized).

Use "make -j 4" ?

Other ideas:
* what compiler are you using to perform the build?

I just execute 'make' from my buid directory. I have no idea what compiler clang uses. Is there a way to find out ? I would expect it uses 'clang' itself.

If it's a previous
Debug+Asserts build of Clang, then that's really going to slow things
down - don't use that, use Release+Asserts or Release-Asserts.

How do I specify Release+Asserts in the 'make' command line ?

* you could try the cmake+ninja build (ninja will run maximally
parallel by default & exploits more parallelism than the make build)

Again I am just following clang's own instructions in their getting started page. I have not tried cmake.

* to do a release build of clang (which might speed things up a little
since you won't be dealing with debug info, etc) I think it's
"--disable-optimized" when configuring the make based build

Is this a configure option ? Do I go back and rerun ./configure passing it '--disable-optimized' ?

I am periodically retrieving the latest sources from the llvm/clang
repository, as specified in the Getting Started page.

When I do a build of clang, through the 'make' command from the build
directory, the build runs very slow. Evidently the default is a
Debug+Assert
build. Compiling a single source file averages 5-10 seconds and overall
the
build takes hours considering how many modules must be built. It is all
successful in the end and when I am finished I have the latest clang,
which
is great.

I have plenty of memory, plenty of disk space, a fairly fast CPU, and am
doing the build on Fedora 17 with hardly any other applications running.

Could you be more specific about the CPU, cores, and available memory?

I have 8 GB of memory and 4 CPUs.

I'd expect a full/clean build to take maybe 10 minutes across 4 cores
(but if it's 4 cpus each with multiple cores, then less time/more
parallelism).

(have you watched the utilization while building Clang? Is the build
topping out on RAM/using swap? using all your CPU cores?)

The simplest thing to do is probably to pass "-j N" where N is the
number of cores you have available (by default the make build isn't
parallelized).

Use "make -j 4" ?

Correct.

Other ideas:
* what compiler are you using to perform the build?

I just execute 'make' from my buid directory. I have no idea what compiler
clang uses. Is there a way to find out ? I would expect it uses 'clang'
itself.

That'd only be possible if you have an existing clang installation?
(otherwise there's the bootstrap problem: how could it build clang
with clang if it hasn't build clang yet?) Assuming you didn't do
anything special, it should be using your system compiler (which means
that's probably not the performance problem - since you'd only get
really heinous performance from a debug/asserts build of Clang being
used to build Clang, and you wouldn't have a debug+asserts built
compiler as your system compiler)

If it's a previous
Debug+Asserts build of Clang, then that's really going to slow things
down - don't use that, use Release+Asserts or Release-Asserts.

How do I specify Release+Asserts in the 'make' command line ?

* you could try the cmake+ninja build (ninja will run maximally
parallel by default & exploits more parallelism than the make build)

Again I am just following clang's own instructions in their getting started
page. I have not tried cmake.

The getting started docs do briefly mention cmake.

* to do a release build of clang (which might speed things up a little
since you won't be dealing with debug info, etc) I think it's
"--disable-optimized" when configuring the make based build

Is this a configure option ? Do I go back and rerun ./configure passing it
'--disable-optimized' ?

That's the rough idea, yes. I'm not sure I've remembered the exact
spelling of the option off-hand, though.

I am periodically retrieving the latest sources from the llvm/clang repository, as specified in the Getting Started page.

When I do a build of clang, through the 'make' command from the build directory, the build runs very slow.

Are you doing a parallel build?

I am just executing 'make' as in the clang Getting Started instructions.

On an i7 laptop, I can do a clean build in 5-10 minutes, with -j4, on a faster desktop it's about 3 minutes. I usually use ninja (just add -G Ninja to your cmake command), rather than make, although there isn't much difference in performance between the two when doing a clean build. For incremental builds, ninja is so much faster that make isn't even in the same league.

That is all nice, but a full build using 'make' after updating to the latest llvm and clang literally takes hours to complete. Im fact I can start it at noon, go away for most of the afternoon, and hope i completes by the time I get back.

So obviously something is ver different between your 5-10 minutes and my hours and hours. Which is why I originally posted.

Evidently the default is a Debug+Assert build. Compiling a single source file averages 5-10 seconds and overall the build takes hours considering how many modules must be built. It is all successful in the end and when I am finished I have the latest clang, which is great.

The Debug+Asserts build is usually faster to compile, because it turns off optimisations, but slower to link, because the debug info is very large. While it can take 5-10 seconds to compile each file, that only matters if you're doing one file at a time.

It is compiling one file at a time. Now it appears my originally estimate was wrong and it really takes on average 10-20 seconds to compile a single file.

I have plenty of memory, plenty of disk space, a fairly fast CPU, and am doing the build on Fedora 17 with hardly any other applications running.

ninja -j8, if you've got enough RAM. If you're not on an SSD, then more processes than you have cores is generally a good idea, because it lets some run while the other are waiting for the disk, but be careful not to have to many more than you have GBs of RAM. A few of the files take about 1GB to compile, and for a debug build with ld-bdf you'll go over 2GB for the final linking step (enabling shared libraries speeds this up, but results in about a 5% slower executable at the end on x86-64).

What is 'ninja -j8' ? Is that a command line option to 'make' ?

Why is compiling clang so slow ? Is there any way to speed up the build ? If I do a release build ( how ? )is it significantly quicker ?

I've not found the compilation time to be much different for release and debug builds (I've not measured it, but I do both fairly regularly and neither feels much slower than the other). The big difference is the time spent linking. If the linker runs out of physical memory and starts to swap then linking time can easily hit 5+ minutes. I've seen it go over 2GB with a statically-linked debug+asserts build of clang.

If you're doing a bootstrap build, make sure that you do a release build first, because the builds with asserts run a lot slower (factor of 10 or so), and trying to build LLVM/Clang with a build with asserts enabled is quite painful.

I am not sure what you mean by 'bootstrap build'. I did an original "configure" when I first got llvm and clang, then a 'make' Afterward, every time I get the latest I simpy do 'make'.

I am following the instructions on the clang 'Getting Started' page at Clang - Getting Started.

I am periodically retrieving the latest sources from the llvm/clang
repository, as specified in the Getting Started page.

When I do a build of clang, through the 'make' command from the build
directory, the build runs very slow. Evidently the default is a
Debug+Assert
build. Compiling a single source file averages 5-10 seconds and overall
the
build takes hours considering how many modules must be built. It is all
successful in the end and when I am finished I have the latest clang,
which
is great.

I have plenty of memory, plenty of disk space, a fairly fast CPU, and am
doing the build on Fedora 17 with hardly any other applications running.

Could you be more specific about the CPU, cores, and available memory?

I have 8 GB of memory and 4 CPUs.

I'd expect a full/clean build to take maybe 10 minutes across 4 cores
(but if it's 4 cpus each with multiple cores, then less time/more
parallelism).

(have you watched the utilization while building Clang? Is the build
topping out on RAM/using swap? using all your CPU cores?)

The simplest thing to do is probably to pass "-j N" where N is the
number of cores you have available (by default the make build isn't
parallelized).

Use "make -j 4" ?

Correct.

I have tried the above with some improvement. Only a few seconds for each compilation now. Thanks !

Other ideas:
* what compiler are you using to perform the build?

I just execute 'make' from my buid directory. I have no idea what compiler
clang uses. Is there a way to find out ? I would expect it uses 'clang'
itself.

That'd only be possible if you have an existing clang installation?
(otherwise there's the bootstrap problem: how could it build clang
with clang if it hasn't build clang yet?) Assuming you didn't do
anything special, it should be using your system compiler (which means
that's probably not the performance problem - since you'd only get
really heinous performance from a debug/asserts build of Clang being
used to build Clang, and you wouldn't have a debug+asserts built
compiler as your system compiler)

I did install a 3.0-14 earlier binary version of clang just so I can run certain tests against both the earler version and the latest I am building. Whether it is using the earlier clang or gcc 4.7.2-2, which I also have installed, to build the latest version of clang I do not know.

If it's a previous
Debug+Asserts build of Clang, then that's really going to slow things
down - don't use that, use Release+Asserts or Release-Asserts.

How do I specify Release+Asserts in the 'make' command line ?

* you could try the cmake+ninja build (ninja will run maximally
parallel by default & exploits more parallelism than the make build)

Again I am just following clang's own instructions in their getting started
page. I have not tried cmake.

The getting started docs do briefly mention cmake.

Yes, I see it there. But aside from mentioning CMake it does not tell how to actually use it AFAICS.

* to do a release build of clang (which might speed things up a little
since you won't be dealing with debug info, etc) I think it's
"--disable-optimized" when configuring the make based build

Is this a configure option ? Do I go back and rerun ./configure passing it
'--disable-optimized' ?

That's the rough idea, yes. I'm not sure I've remembered the exact
spelling of the option off-hand, though.

OK, thanks. I will check that out also to try to speed up my builds. I don't really need the debug version since I am just using clang rather than being a clang developer.

I am periodically retrieving the latest sources from the llvm/clang
repository, as specified in the Getting Started page.

When I do a build of clang, through the 'make' command from the build
directory, the build runs very slow. Evidently the default is a
Debug+Assert
build. Compiling a single source file averages 5-10 seconds and overall
the
build takes hours considering how many modules must be built. It is all
successful in the end and when I am finished I have the latest clang,
which
is great.

I have plenty of memory, plenty of disk space, a fairly fast CPU, and
am
doing the build on Fedora 17 with hardly any other applications
running.

Could you be more specific about the CPU, cores, and available memory?

I have 8 GB of memory and 4 CPUs.

I'd expect a full/clean build to take maybe 10 minutes across 4 cores
(but if it's 4 cpus each with multiple cores, then less time/more
parallelism).

(have you watched the utilization while building Clang? Is the build
topping out on RAM/using swap? using all your CPU cores?)

The simplest thing to do is probably to pass "-j N" where N is the
number of cores you have available (by default the make build isn't
parallelized).

Use "make -j 4" ?

Correct.

I have tried the above with some improvement. Only a few seconds for each
compilation now. Thanks !

Other ideas:
* what compiler are you using to perform the build?

I just execute 'make' from my buid directory. I have no idea what
compiler
clang uses. Is there a way to find out ? I would expect it uses 'clang'
itself.

That'd only be possible if you have an existing clang installation?
(otherwise there's the bootstrap problem: how could it build clang
with clang if it hasn't build clang yet?) Assuming you didn't do
anything special, it should be using your system compiler (which means
that's probably not the performance problem - since you'd only get
really heinous performance from a debug/asserts build of Clang being
used to build Clang, and you wouldn't have a debug+asserts built
compiler as your system compiler)

I did install a 3.0-14 earlier binary version of clang just so I can run
certain tests against both the earler version and the latest I am building.
Whether it is using the earlier clang or gcc 4.7.2-2, which I also have
installed, to build the latest version of clang I do not know.

If it's a previous
Debug+Asserts build of Clang, then that's really going to slow things
down - don't use that, use Release+Asserts or Release-Asserts.

How do I specify Release+Asserts in the 'make' command line ?

* you could try the cmake+ninja build (ninja will run maximally
parallel by default & exploits more parallelism than the make build)

Again I am just following clang's own instructions in their getting
started
page. I have not tried cmake.

The getting started docs do briefly mention cmake.

Yes, I see it there. But aside from mentioning CMake it does not tell how to
actually use it AFAICS.

Yeah, doesn't go into much detail.

If you have ninja & cmake with ninja support, you'd run:

cmake -G Ninja ../src

and then simply run "ninja"

* to do a release build of clang (which might speed things up a little
since you won't be dealing with debug info, etc) I think it's
"--disable-optimized" when configuring the make based build

Is this a configure option ? Do I go back and rerun ./configure passing
it
'--disable-optimized' ?

That's the rough idea, yes. I'm not sure I've remembered the exact
spelling of the option off-hand, though.

OK, thanks. I will check that out also to try to speed up my builds. I don't
really need the debug version since I am just using clang rather than being
a clang developer.

& of course I had this quite backwards, --disable-optimized is how you
get a Debug build not how you avoid it (derr - I blame the weekend).

As David pointed out, the release V debug tradeoff isn't strictly
obvious in terms of compile-time, but disabling assertions might get
you a little bit of a compile-time improvement. (--disable-asserts or
some similar spelling)

Edward, I just finished a build from the start(i.e. everything had to be
compiled). It was done with my system's gcc compiler, AFAICT:

~/download/llvm/pre-releases/3.2/rc3/download/build_debug $ g++ -v
Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu
4.4.3-4ubuntu5.1'
--with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs
--enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr
--enable-shared --enable-multiarch --enable-linker-build-id
--with-system-zlib --libexecdir=/usr/lib --without-included-gettext
--enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4
--program-suffix=-4.4 --enable-nls --enable-clocale=gnu
--enable-libstdcxx-debug --enable-plugin --enable-objc-gc
--disable-werror --with-arch-32=i486 --with-tune=generic
--enable-checking=release --build=x86_64-linux-gnu
--host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5.1)

And the complete build, with configuration set with:

#!/bin/bash
this_dir=`pwd`
$this_dir/../llvm.src/configure \
  --prefix=$this_dir/../install_debug \
  --enable-optimized=NO \
  --enable-assertions=YES \
  --enable-debug-runtime=YES \
  --enable-debug-symbols=YES \

[snip]

And the complete build, with configuration set with:

#!/bin/bash
this_dir=`pwd`
$this_dir/../llvm.src/configure \
  --prefix=$this_dir/../install_debug \
  --enable-optimized=NO \
  --enable-assertions=YES \
  --enable-debug-runtime=YES \
  --enable-debug-symbols=YES \
  #

completed in about 1 hr. I've got 2Gig memory and 2 cpu's
and plenty of disk space.

Another difference w.r.t. your build is that mine didn't use
the svn code. Mine was downloaded from:

http://llvm.org/pre-releases/3.2/rc3/clang-3.2rc3.src.tar.bz2
http://llvm.org/pre-releases/3.2/rc3/compiler-rt-3.2rc3.src.tar.bz2
http://llvm.org/pre-releases/3.2/rc3/llvm-3.2rc3.src.tar.bz2
http://llvm.org/pre-releases/3.2/rc3/test-suite-3.2rc3.src.tar.bz2

You can look in config.log in the build directory and see what compiler the configure script picked up. IIRC it tries "clang" first, unless you explicitly specify a compiler via CC and CXX.

-Krzysztof

I am periodically retrieving the latest sources from the llvm/clang
repository, as specified in the Getting Started page.

When I do a build of clang, through the 'make' command from the build
directory, the build runs very slow. Evidently the default is a
Debug+Assert build. Compiling a single source file averages 5-10 seconds
and overall the build takes hours considering how many modules must be
built. It is all successful in the end and when I am finished I have the
latest clang, which is great.

I have plenty of memory, plenty of disk space, a fairly fast CPU, and am
doing the build on Fedora 17 with hardly any other applications running.

Why is compiling clang so slow ? Is there any way to speed up the build
? If I do a release build ( how ? )is it significantly quicker ?

Edward, I just finished a build from the start(i.e. everything had to be
compiled). It was done with my system's gcc compiler, AFAICT:

~/download/llvm/pre-releases/3.2/rc3/download/build_debug $ g++ -v
Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu
4.4.3-4ubuntu5.1'
--with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs
--enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr
--enable-shared --enable-multiarch --enable-linker-build-id
--with-system-zlib --libexecdir=/usr/lib --without-included-gettext
--enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4
--program-suffix=-4.4 --enable-nls --enable-clocale=gnu
--enable-libstdcxx-debug --enable-plugin --enable-objc-gc
--disable-werror --with-arch-32=i486 --with-tune=generic
--enable-checking=release --build=x86_64-linux-gnu
--host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5.1)

And the complete build, with configuration set with:

#!/bin/bash
this_dir=`pwd`
$this_dir/../llvm.src/configure \
   --prefix=$this_dir/../install_debug \
   --enable-optimized=NO \
   --enable-assertions=YES \
   --enable-debug-runtime=YES \
   --enable-debug-symbols=YES \
   #

In my ./configure command I gave no options. Perhaps one of the options you set makes the building of clang run faster as against whatever default options it uses when you do not specify any.

completed in about 1 hr. I've got 2Gig memory and 2 cpu's
and plenty of disk space.

When I told 'make' to use all of my processor via 'make -j 4' things went appreciably quicker. So it may have finished in an hour if I had used that to begin with.