Current state-of-the-art in whole program optimization

As I understand it, clang currently has the -flto flag which activates some degree of whole program optimization. However, this requires clang to be presented with all the source files at the same time. This is easy enough to arrange in a greenfield project, but may be trickier with an existing program.

On Unix, C and C++ programs are usually compiled with makefiles (whether handwritten or auto generated), and these typically call the compiler on one source file at a time and then call the system linker only after compilation has proceeded to machine code object files. Thus, even with clang as a drop-in replacement for GCC, there is the challenge of obtaining exact compilation instructions from the makefile and then modifying these to allow clang to see all the files at once, no later than bitcode stage.

On Windows, it’s common to have a Visual Studio project file that msbuild can use. This sometimes does call the compiler on multiple files at the same time; then there’s the question of how to get clang-cl invoked with suitable options.

At least that’s my thinking on this aspect of the problem so far; there are doubtless things I’m missing. What’s currently being done in this area?

As I understand it, clang currently has the -flto flag which activates
some degree of whole program optimization. However, this requires clang to
be presented with all the source files at the same time. This is easy
enough to arrange in a greenfield project, but may be trickier with an
existing program.

On Unix, C and C++ programs are usually compiled with makefiles (whether
handwritten or auto generated), and these typically call the compiler on
one source file at a time and then call the system linker only after
compilation has proceeded to machine code object files. Thus, even with
clang as a drop-in replacement for GCC, there is the challenge of obtaining
exact compilation instructions from the makefile and then modifying these
to allow clang to see all the files at once, no later than bitcode stage.

This is generally already addressed by just making 'object' files that are
actually LLVM bitcode. I forget which flags are needed, exactly, but you'll
also need a linker that's aware of this trick (there's an LLVM plugin for
the gold linker, for example).

I forget which flags you have to pass to clang to make it do this (maybe it
is just -flto) but once you pass that flag and have an LLVM-aware linker,
it should 'just work' at least as far as the build system issues are
concerned.

- David

As I understand it, clang currently has the -flto flag which activates
some degree of whole program optimization. However, this requires clang to
be presented with all the source files at the same time.

Not at all. The -flto flag causes clang to emit a half-compiled .o file
that is not at all a real object file. It contains no object code and is
not elf, mach-o nor pe/coff. It's an llvm bitcode file.

The second half of this story is that you're going to ask the linker to
compile a program with these non-object file .o files, and the linker needs
to handle it. On darwin "ld" is linked against libLTO [
http://llvm.org/docs/LinkTimeOptimization.html ]. On linux, you can use
"gold" (the newer ld in binutils) with a plugin [
http://llvm.org/docs/GoldPlugin.html ]. I'm told the plan is to make lld
support llvm lto when building windows/pecoff files, but this may not be
ready yet.

This is easy enough to arrange in a greenfield project, but may be trickier

with an existing program.

On Unix, C and C++ programs are usually compiled with makefiles (whether
handwritten or auto generated), and these typically call the compiler on
one source file at a time and then call the system linker only after
compilation has proceeded to machine code object files. Thus, even with
clang as a drop-in replacement for GCC, there is the challenge of obtaining
exact compilation instructions from the makefile and then modifying these
to allow clang to see all the files at once, no later than bitcode stage.

The gold plugin webpage covers exactly this case, including a section on
how to make llvm LTO work with autotools (configure+make) projects.

Nick

On Windows, it's common to have a Visual Studio project file that msbuild