LLD status update and performance chart

Project Infrastructure LLVM Dev List Archives
1

Hi,

Now that 2016 is almost over, I wanted to look back and summarize the progress we’ve made to LLD this year, as I guess most people who are not looking closely at LLD don’t know very well about the current status. I think I can say that this year was a fantastic year for LLD. Now I’m pretty sure that that is going to be a serious (and better, in my opinion) alternative to the existing GNU linkers thanks to all the improvements we’ve made this year.

LLD is now able to link most x86-64 userland programs. The FreeBSD project and we are trying to make LLD the system default linker of the operating system, and except a few tricky programs such as the kernel or a boot loader, the linker works mostly fine. We are still working on implementing long-tail features/bugs, but I’d say that’s just a matter of time. LLD supports x86, x86-64, x32, AArch64, AMDGPU, ARM, PPC64 and MIPS32/64, though completeness varies.

Looks like there are already a few systems that are using LLD as system linkers, such as CloudABI or Fuchsia. Chromium and Clang/LLVM itself has build options to use LLD to build them.

It is hard to argue about the complexity of a program quantitatively, and of course I’m biased, but I believe we succeeded to maintain LLD code base clean, easy to read, and easy to add new features. It is just 20k lines of modern C++ code which is much smaller than GNU linkers.

Even though LLD was fast from day one, LLD got faster this year, despite it got a lot of new features. Below is a chart of Clang link time for every commit made to the LLD repository this year. At the beginning of this year, LLD took about 16 seconds to produce a 1.5 GB clang (debug build) executable. Now, it takes about 14.5 seconds on single core and 8.5 seconds on 20 cores (*1). ld.gold takes about 25 seconds and 20 seconds, respectively, so we’ve widen the gap. You can see the benchmark results here (*2). If you have a problem of too long link time, I’d recommend to try LLD.

Last but not least, a lot of people joined to the LLD development this year to make LLD better. We are growing as a community, and I’m very happy about that!

Thanks,
Rui

(*1) My machine has Ivy Bridge Xeon 2.8 GHz 20 physical cores (40 hyper-threading cores). To measure a single-thread performance, I pinned a process to (physical and hyper-threading) core 0. To measure a multi-thread performance, I pinned to CPU socket 2, so that a process gets 10 physical cores (20 hyperthreading cores).

(*2) https://docs.google.com/spreadsheets/d/1VvOqiU5JvqlxU7aof8gsbh-yweeNchMgtkamyXrwzrA/edit?usp=sharing. Changes with more than 1% rise or drop compared to the average of previous 5 commits are colored in green or red, respectively.

2

Looks like the image wasn’t attached correctly. Here’s the chart.

Capture.GIF
Capture.GIF834×516 34 KB

3

Hmm, I think I made a mistake again. This is the correct chart.

Capture.GIF
Capture.GIF701×405 23.9 KB

Capture.GIF
Capture.GIF834×516 34 KB

4

Thank you for this exciting update about LLD.

I will start experimenting with using liblld in Zig (http://ziglang.org/) instead of starting a child process to invoke the system linker. Once liblld makes it into the various package managers out there, this will be a big step toward painless cross-platform compilation.

It should also reduce one of the compilation speed bottlenecks, since I could transition from writing objects to the file system to passing liblld LLVM modules directly. Is that correct?

5

Thank you for this exciting update about LLD.

I will start experimenting with using liblld in Zig (http://ziglang.org/)
instead of starting a child process to invoke the system linker. Once
liblld makes it into the various package managers out there, this will be a
big step toward painless cross-platform compilation.

It should also reduce one of the compilation speed bottlenecks, since I
could transition from writing objects to the file system to passing liblld
LLVM modules directly. Is that correct?

LLD's driver currently takes only a command line argument strings, so you
need to write ELF files to a filesystem and read them back at the moment.

Theoretically, we could accept a list of command line strings and
MemoryBuffer objects (instead of filenames) so that we can link in-memory
ELF object files. Adding that feature shouldn't be hard.

6

Thanks a lot for the summary!

As my last big task for the year I was asked to check the status of the
ports build in freebsd. That includes a tremendous amount of software,
and we are doing very well there.

With just a few hacks (including printing "not GNU" in --version) the
results are that we can build 19828 packages and 179 fail. There were
also 6175 skipped.

Some of the failures don't look lld specific (I noticed a missing
dependency on makeinfo for example). Others are because of our library
handling and yet others are because of broken object files.

I will try to upload all the failed logs and build dirs somewhere before
I go on vacation, but it is pretty impressive how much we can link and works.

Cheers,
Rafael

Rui Ueyama via llvm-dev <llvm-dev@lists.llvm.org> writes:

7

LLD is now able to link most x86-64 userland programs. The FreeBSD project
and we are trying to make LLD the system default linker of the operating
system, and except a few tricky programs such as the kernel or a boot
loader, the linker works mostly fine. We are still working on implementing
long-tail features/bugs, but I'd say that's just a matter of time. LLD
supports x86, x86-64, x32, AArch64, AMDGPU, ARM, PPC64 and MIPS32/64,
though completeness varies.

Thanks for the update!

From our side, AArch64 is stable and "production ready", though not

necessarily complete. We now have a buildbot that bootstraps Clang and runs
the test-suite with LLD:

http://lab.llvm.org:8011/builders/clang-cmake-aarch64-lld

We'll start looking at other large and complex programs next (though Clang
is quite complex and libclang is quite large already).

ARM support is coming a long way, but we're still not able to link Clang
yet, and there are some issues on the test-suite. We'll be continuing to
make progress throughout next year and hopefully have a similar buildbot on
ARM.

Rafael,

Great news about FreeBSD! Once LLD has become their default linker in
x86_64, I'd like to start the same process on AArch64.

cheers,
--renato

8

It would be nice to have a similar interface to the one in libclang, where the driver is given a map that substitutes in-memory objects for specific files, so that the same command line can be used whether the inputs are on disk or in memory.

David

9

David Chisnall via llvm-dev <llvm-dev@lists.llvm.org> writes:

LLD's driver currently takes only a command line argument strings, so you need to write ELF files to a filesystem and read them back at the moment.

Theoretically, we could accept a list of command line strings and MemoryBuffer objects (instead of filenames) so that we can link in-memory ELF object files. Adding that feature shouldn't be hard.

It would be nice to have a similar interface to the one in libclang, where the driver is given a map that substitutes in-memory objects for specific files, so that the same command line can be used whether the inputs are on disk or in memory.

I would strongly object to expanding our "library" interface at the
moment.

Even what we currently have I think is a bad idea and a distraction from
the current objective: writing an excellent stand alone linker. We
benefit massively from simple error and memory management.

Once the linker is "done", we can discuss other possible uses and their
costs. By "done" I mean at least having all the features we want and
having replaced bfd ld in at least two freebsd architectures.

Cheers,
Rafael

10

Renato Golin via llvm-dev <llvm-dev@lists.llvm.org> writes:

ARM support is coming a long way, but we're still not able to link Clang
yet, and there are some issues on the test-suite. We'll be continuing to
make progress throughout next year and hopefully have a similar buildbot on
ARM.

BTW, can you link it if you enable only the ARM target? Last time I
tried on TK1 the only remaining issue was the range extension thunks and
with only one target we got lucky and didn't need them.

Rafael,

Great news about FreeBSD! Once LLD has become their default linker in
x86_64, I'd like to start the same process on AArch64.

Awesome. I actually expect a much smaller tail on AArch64 than on X86_64.

Cheers,
Rafael

11

That's a good point! Peter, have you tried that?

Indeed, thunks were the main remaining issue for us on ARM.

cheers,
--renato

12

I can get the llvm* tools to link on ARM with shared libraries and
only ARM, AArch64 and x86 as targets. With libclang.so I wasn't quite
so lucky but I did have an extra couple of Targets over just ARM.

There are a few small niggles on ARM that I'm hoping to post fixes for
this week:
- Need to support thunks to undefined symbols (Python C extension
shared objects calling back to the Python interpreter).
- eglibc (default in Ubuntu 14.04) requires a .ARM.attributes section
for dlopen to work (it checks the attributes section for hard/soft
float).

After these it will be a good time to have a go at range extension thunks.

13

It seems reasonable to me to focus on this first, but do you agree that
once lld gets to this milestone, the library API is a worthy time
investment?

14

I'm fine as long as our API is the same as the command line. In that sense,
passing a map from filename to MemoryBuffer objects in addition to command
line arguments is fine to me.

15

It seems reasonable to me to focus on this first, but do you agree that
once lld gets to this milestone, the library API is a worthy time
investment?

It is not something I have an interest on, but I wouldn't object to it
being done if it doesn't regress performance or complicate the linker
too much.

Cheers,
Rafael

16

David Chisnall via llvm-dev <llvm-dev@lists.llvm.org> writes:

>>
>> LLD's driver currently takes only a command line argument strings, so
you need to write ELF files to a filesystem and read them back at the
moment.
>>
>> Theoretically, we could accept a list of command line strings and
MemoryBuffer objects (instead of filenames) so that we can link in-memory
ELF object files. Adding that feature shouldn't be hard.
>
> It would be nice to have a similar interface to the one in libclang,
where the driver is given a map that substitutes in-memory objects for
specific files, so that the same command line can be used whether the
inputs are on disk or in memory.

I would strongly object to expanding our "library" interface at the
moment.

Even what we currently have I think is a bad idea and a distraction from
the current objective: writing an excellent stand alone linker. We
benefit massively from simple error and memory management.

Once the linker is "done", we can discuss other possible uses and their
costs. By "done" I mean at least having all the features we want and
having replaced bfd ld in at least two freebsd architectures.

It seems reasonable to me to focus on this first, but do you agree that
once lld gets to this milestone, the library API is a worthy time
investment?

I'm fine as long as our API is the same as the command line. In that
sense, passing a map from filename to MemoryBuffer objects in addition to
command line arguments is fine to me.

This will also greatly facilitate certain measurements I'd like to do
w.r.t. different strategies for avoiding memory costs for input files (esp.
minor faults and dTLB costs). I've almost gotten to the point of
implementing this just to do those measurements.

-- Sean Silva

17

LLD/ELF is is only about a year and a half old, and it’s amazing how far we’ve come. I have LLD/ELF to thank for the fondest memories of my time at PlayStation; I can’t count how many times Michael and I were there past midnight getting things working, and after Davide joined us, seeing him leave every night limited only by how late the last Caltrain would run. It’s amazing to be a part of this.

One of the most exciting things in the last months has been this new phase LLD has entered where new developers are joining, fixing the bugs they find and implementing the features they need as they port LLD to new platforms – that is the ultimate validation that we’re doing something right.

– Sean Silva

18

Sean Silva via llvm-dev <llvm-dev@lists.llvm.org> writes:

This will also greatly facilitate certain measurements I'd like to do
w.r.t. different strategies for avoiding memory costs for input files (esp.
minor faults and dTLB costs). I've almost gotten to the point of
implementing this just to do those measurements.

If you do please keep it local. The bare minimum we have of library
support is already disproportionately painful and prevents easier sharing
with COFF. We should really not add more until the linker is done.

Cheers,
Rafael

19

This is so much in contrast with the LLVM development, I find it quite hard to see this as an acceptable position on llvm-dev.

20

From: "Mehdi Amini via llvm-dev" <llvm-dev@lists.llvm.org>
To: "Rafael Espíndola" <rafael.espindola@gmail.com>
Cc: "llvm-dev" <llvm-dev@lists.llvm.org>
Sent: Tuesday, December 13, 2016 11:28:18 AM
Subject: Re: [llvm-dev] LLD status update and performance chart

>
> Sean Silva via llvm-dev <llvm-dev@lists.llvm.org> writes:
>> This will also greatly facilitate certain measurements I'd like to
>> do
>> w.r.t. different strategies for avoiding memory costs for input
>> files (esp.
>> minor faults and dTLB costs). I've almost gotten to the point of
>> implementing this just to do those measurements.
>
> If you do please keep it local. The bare minimum we have of library
> support is already disproportionately painful and prevents easier
> sharing
> with COFF. We should really not add more until the linker is done.

This is so much in contrast with the LLVM development, I find it
quite hard to see this as an acceptable position on llvm-dev.

I agree. As a project, we should evaluate all potential contributions on their merrits, and it seems useful to withhold judgement on software-engineering tradeoffs until we can evaluate the code impact concretely.

-Hal