System hangs during last stages of LLVM build | Tips on speeding it up ?

Hello,

My system hangs every time during last stages of building LLVM ( starting at 95% in a CMake build ) using CMake or Ninja, sometime close to the linking of llvm-dysmutil. Could you please suggest tips that could speed up the compilation ?

Thank You,
Sanjay

Try to compile with "-j1". You likely do not have enough RAM and the
system might be swapping.

Best,
Tobias

Hi Sanjay,

My system hangs every time during last stages of building LLVM ( starting at
95% in a CMake build ) using CMake or Ninja, sometime close to the linking
of llvm-dysmutil. Could you please suggest tips that could speed up the
compilation ?

This is probably because multiple linker processes are using all your
memory and bogging down in swap. There are various things you can do
to improve performance:

  * There's an LLVM_PARALLEL_LINK_JOBS option which you can use to
reduce the number of ld instances running at once (you'll need to be
using Ninja rather than Make for this to work, but that's faster
anyway). This is a slightly more fine-grained approach than Tobias's
"-j 1"; systems can usually support more compile jobs than link.
* If on Linux, you can switch to ld.gold (or lld if you're feeling
brave!), which is much faster than the default ld.bfd
* Also on ELF platforms, there's LLVM_USE_SPLIT_DWARF which separates
the debug info into a different file. This really speeds things up,
but you need a recent(ish) gdb to be able to debug the result.

Those are the obvious ways to reduce memory. You might also consider
only building the targets you need, which will speed things up but not
get around swapping usually.

Cheers.

Tim.

And if you can afford it, passing -DBUILD_SHARED_LIBS=ON to cmake makes
the last step blazing fast.

I did try make -j 1. But, the build was slower and yes, there was a lot of swapping.

Hello Tim,

Thanks for the tips !

Hi Sanjay,

My system hangs every time during last stages of building LLVM ( starting at
95% in a CMake build ) using CMake or Ninja, sometime close to the linking
of llvm-dysmutil. Could you please suggest tips that could speed up the
compilation ?

This is probably because multiple linker processes are using all your
memory and bogging down in swap. There are various things you can do
to improve performance:

  • There’s an LLVM_PARALLEL_LINK_JOBS option which you can use to
    reduce the number of ld instances running at once (you’ll need to be
    using Ninja rather than Make for this to work, but that’s faster
    anyway). This is a slightly more fine-grained approach than Tobias’s
    “-j 1”; systems can usually support more compile jobs than link.
  • If on Linux, you can switch to ld.gold (or lld if you’re feeling
    brave!), which is much faster than the default ld.bfd

Would complications arise because of a cyclic dependency, using lld to build (a newer) lld, and possibly changing the lld binary while it’s being used to link other files ?

  • Also on ELF platforms, there’s LLVM_USE_SPLIT_DWARF which separates
    the debug info into a different file. This really speeds things up,
    but you need a recent(ish) gdb to be able to debug the result.

Those are the obvious ways to reduce memory. You might also consider
only building the targets you need, which will speed things up but not
get around swapping usually.

This is my experience after targeting only X86 and NVPTX !! :laughing:

Cheers.

Tim.

Thank You,
Sanjay Srivallabh

Yes. You can't use a tool in its own build, so you need something like
a 2-stage compilation. You'd have two build directories; one uses the
system linker to build lld (probably in release mode for speed); the
other uses that lld to build the rest of LLVM.

Or use an lld provided by your OS, which would be much easier. They're
thin on the ground these days, but they should be getting more common;
I think Debian at least might get it for the 4.0 release.

Cheers.

Tim.