Using thin archives when building llvm

I have just committed support to llvm-ar for creating thin archives.
The idea of thin archives is that they contain just the symbol table
and the path to find the original .o files.

By locally making thin archives the default I was able to build
llvm+lld+clang with them. The total size of the .a files goes from
181,658,164 to 7,116,900 bytes.

Is there any way to do that with cmake without having to have a hacked
llvm-ar? All that is needed is for cmake to run

.../llvm-ar cqT foo.a ....

instead of

.../llvm-ar cq foo.a ....

Thanks,
Rafael

I have just committed support to llvm-ar for creating thin archives.
The idea of thin archives is that they contain just the symbol table
and the path to find the original .o files.

By locally making thin archives the default I was able to build
llvm+lld+clang with them. The total size of the .a files goes from
181,658,164 to 7,116,900 bytes.

Nice!

Is there any way to do that with cmake without having to have a hacked
llvm-ar? All that is needed is for cmake to run

…/llvm-ar cqT foo.a …

instead of

…/llvm-ar cq foo.a ….

I found an answer, but its not much better than having a hacked up version of llvm-ar:
http://stackoverflow.com/questions/5659225/how-to-set-the-options-for-cmake-ar

Cheers,
Pete

FYI, I override CMAKE_CXX_CREATE_STATIC_LIBRARY.

See,
http://bb.pgr.jp/builders/clang-3stage-x86_64-linux/builds/8687/steps/CMake/logs/stdio

2015年7月16日(木) 12:37 Pete Cooper <peter_cooper@apple.com>:

Thanks, I didn't realized that I could set that in the command line.

I have updated my run-cmake script to use thin archives now:
https://github.com/espindola/llvm-scripts/commit/64f54889b5dda0281a5c30f2442e959dbe4e8f69

Now to figure out how to use them on windows.

Cheers,
Rafael

I have just committed support to llvm-ar for creating thin archives.
The idea of thin archives is that they contain just the symbol table
and the path to find the original .o files.

Neat! Is this expected to be used only for non-installed archives?

Is there any way to do that with cmake without having to have a hacked
llvm-ar? All that is needed is for cmake to run

.../llvm-ar cqT foo.a ....

instead of

.../llvm-ar cq foo.a ....

CMake has undocumented variables set by the Modules/Platform/*
files to specify the rules for creating archives. These are:

CMAKE_<LANG>_ARCHIVE_{CREATE,APPEND,FINISH}

for creating, appending to, and finishing an archive. For
tools/platforms that do not support separate steps we also
have:

CMAKE_<LANG>_CREATE_STATIC_LIBRARY

One or both of these may need to be updated to handle this.
However, if this should be done only for a subset of archives
then it needs to be optional on a per-target bases. For this
you may be able to just use the STATIC_LIBRARY_FLAGS property:

STATIC_LIBRARY_FLAGS — CMake 3.3.2 Documentation

I don't recall off the top of my head though where the flags
will be placed for "ar" and whether they will apply on all
three archiving steps. Hopefully the above is enough info
to get you started.

-Brad

I have just committed support to llvm-ar for creating thin archives.
The idea of thin archives is that they contain just the symbol table
and the path to find the original .o files.

Neat! Is this expected to be used only for non-installed archives?

Yes, the idea is to speed up a regular development build.

Is there any way to do that with cmake without having to have a hacked
llvm-ar? All that is needed is for cmake to run

.../llvm-ar cqT foo.a ....

instead of

.../llvm-ar cq foo.a ....

CMake has undocumented variables set by the Modules/Platform/*
files to specify the rules for creating archives. These are:

CMAKE_<LANG>_ARCHIVE_{CREATE,APPEND,FINISH}

I went with this one on my run-cmake script, thanks.

Cheers,
Rafael

I have just committed support to llvm-ar for creating thin archives.
The idea of thin archives is that they contain just the symbol table
and the path to find the original .o files.

Neat! Is this expected to be used only for non-installed archives?

Yes, the idea is to speed up a regular development build.

Is there any way to do that with cmake without having to have a hacked
llvm-ar? All that is needed is for cmake to run

.../llvm-ar cqT foo.a ....

instead of

.../llvm-ar cq foo.a ....

CMake has undocumented variables set by the Modules/Platform/*
files to specify the rules for creating archives. These are:

CMAKE_<LANG>_ARCHIVE_{CREATE,APPEND,FINISH}

I went with this one on my run-cmake script, thanks.

FWIW, i’d more than welcome a patch to get this in to cmake.

Chris added a flag (LLVM_OPTIMIZED_TBLGEN) to build an optimized tblgen and use that in the regular build. I’d be happy to see a similar flag used to have that step build an llvm-ar used by the rest of compilation to use thin archives. At least until we start shipping this.

I’m happy to propose a patch or help test one if you have one in future.

Cheers,
Pete

FWIW, i’d more than welcome a patch to get this in to cmake.

Chris added a flag (LLVM_OPTIMIZED_TBLGEN) to build an optimized tblgen and use that in the regular build. I’d be happy to see a similar flag used to have that step build an llvm-ar used by the rest of compilation to use thin archives. At least until we start shipping this.

I’m happy to propose a patch or help test one if you have one in future.

I normally locally install a recent trunk, so for me
CMAKE_CXX_CREATE_STATIC_LIBRARY does the job. Given that, I am
probably not the best one to write patch for building llvm-ar and then
use it.

Also, llvm-ar depends on target (to parse inline asm), so quite a few
.a files will still be built with the old ar.

I think the only linkers that support thin archives are
* bfd ld
* gold
* The new COFF lld

So far I have really only tested with gold (still doing windowns setup
to test the new lld).

Last but not least, the thin archive is in gnu format. If adding
support to OS X (lld or ld64), it might be interesting to experiment
with a thin BFD format (the symbol table there has offsets into the
symbol name table, making it a bit faster to use with unix .a
semantics).

Cheers,
Rafael

CMake has undocumented variables set by the Modules/Platform/*
files to specify the rules for creating archives. These are:

CMAKE_<LANG>_ARCHIVE_{CREATE,APPEND,FINISH}

for creating, appending to, and finishing an archive. For
tools/platforms that do not support separate steps we also
have:

CMAKE_<LANG>_CREATE_STATIC_LIBRARY

Setting CMAKE_CXX_CREATE_STATIC_LIBRARY works on OS X and linux, but
on windows I still see a call to "lld-link2 /lib..." when
CMAKE_CXX_CREATE_STATIC_LIBRARY is set to use llvm-lib.

Cheers,
Rafael

What CMake generator are you using on Windows? These command-line
rule variables are implementation details of the Makefile and Ninja
generators only. For the VS IDE project files msbuild constructs
the actual command lines.

-Brad

I was using ninja for windows too.

Cheers,
Rafael

Where does lld-link2 get chosen? CMake doesn't have code for that
name so something else in your project must be specifying it somehow.

Thanks,
-Brad

Yes, the simplest way to reproduce the problem is to run

mkdir build
cd build
cmake ..\llvm -G Ninja -DCMAKE_CXX_CREATE_STATIC_LIBRARY=foobar
-DCMAKE_LINKER=c:\users\espindola\llvm\inst\bin\lld-link2

and then

ninja -v lib\LLVMSupport.lib

I would expect it to fail trying to run foobar, but it works by
running lld-link2

Cheers,
Rafael

The Modules/Platform/Windows-MSVC.cmake module unconditionally
sets CMAKE_CXX_CREATE_STATIC_LIBRARY and uses CMAKE_LINKER.
As mentioned earlier the CMAKE_<LANG>_CREATE_STATIC_LIBRARY
and similar variables are internal implementation details
that are not meant to be set by users or project code. That
is why the above works only on certain platforms.

Will thin archive support be distributed as part of llvm-ar
proper for general use or is this only for LLVM's own build?
If it will be for general use then it may also be worthwhile
teaching CMake this option directly.

-Brad

The Modules/Platform/Windows-MSVC.cmake module unconditionally
sets CMAKE_CXX_CREATE_STATIC_LIBRARY and uses CMAKE_LINKER.
As mentioned earlier the CMAKE_<LANG>_CREATE_STATIC_LIBRARY
and similar variables are internal implementation details
that are not meant to be set by users or project code. That
is why the above works only on certain platforms.

Will thin archive support be distributed as part of llvm-ar
proper for general use or is this only for LLVM's own build?
If it will be for general use then it may also be worthwhile
teaching CMake this option directly.

It is generally useful. The gnu ar already supports it. The
implementation in llvm-ar is compatible with it.

Making it available in llvm-lib is an extension over lib.exe.

Cheers,
Rafael

It is generally useful. The gnu ar already supports it. The
implementation in llvm-ar is compatible with it.

Making it available in llvm-lib is an extension over lib.exe.

I was able to get this to work on windows by using

-DCMAKE_STATIC_LINKER_FLAGS=/llvmlibthin

Thanks,
Rafael

I was able to get this to work on windows by using

-DCMAKE_STATIC_LINKER_FLAGS=/llvmlibthin

And with r242926 it looks like the new lld just works with thin
archives on windows :slight_smile:

Cheers,
Rafael

Cool!

So the thin archive is a divergence from the standard ar file (although
it's compatible with GNU). Is there any room to push it further? Last time
I ran the linker with profiling enabled, it spends a good amount of time
just to find the terminating nul character in the archive file symbol
table. If we store string length for each symbol, the linker can read
archive files faster.

Cool!

So the thin archive is a divergence from the standard ar file (although it's
compatible with GNU). Is there any room to push it further? Last time I ran
the linker with profiling enabled, it spends a good amount of time just to
find the terminating nul character in the archive file symbol table. If we
store string length for each symbol, the linker can read archive files
faster.

We can probably do it, yes.

Take a look at the BSD format (used on OS X, I just implemented it in llvm).

It is a bit better in that the symbol table is organized as a series
of offset pairs. One to the member, one to the string table.

This already improves handling on the traditional unix linker model
where we scan each member to see if it should be included on the link.
Once we find out it is to be included, it is really fast to scan past
the member without looking for nulls as one has to do in the GNU
format.

That doesn't help with COFF were we do a single pass anyway, but there
is more that we could benefit from the BSD format. I think that in
practice the string table in in order, so that we can compute the
string size by looking at the next member. I will give that a try.

Another reason to come up with a thin BSD format variant :slight_smile:

Cheers,
Rafael

P.S.: While testing the thin archive format I noticed that the thin
.lib files were a lot bigger than what I was getting on linux. It
turns out it was because cl.exe was producing .obj files with a *lot*
more symbols than clang on linux. Trying clang on windows showed that
cl.exe was not dropping what we call linkonce_odr, but clang on
windows still produces more symbols than clang on linux.

> Cool!
>
> So the thin archive is a divergence from the standard ar file (although
it's
> compatible with GNU). Is there any room to push it further? Last time I
ran
> the linker with profiling enabled, it spends a good amount of time just
to
> find the terminating nul character in the archive file symbol table. If
we
> store string length for each symbol, the linker can read archive files
> faster.

We can probably do it, yes.

Take a look at the BSD format (used on OS X, I just implemented it in
llvm).

It is a bit better in that the symbol table is organized as a series
of offset pairs. One to the member, one to the string table.

This already improves handling on the traditional unix linker model
where we scan each member to see if it should be included on the link.
Once we find out it is to be included, it is really fast to scan past
the member without looking for nulls as one has to do in the GNU
format.

That doesn't help with COFF were we do a single pass anyway, but there
is more that we could benefit from the BSD format. I think that in
practice the string table in in order, so that we can compute the
string size by looking at the next member. I will give that a try.

Even if the string table is in in-order in practice, you have to search for
NUL characters byte-by-byte unless it's really guaranteed to be in-order,
no?

Another reason to come up with a thin BSD format variant :slight_smile: