Please dogfood LLD

Hi all,

LLVM 4.0.0 is out, and I can say that LLD/ELF is now ready for production use at least for x86-64 (and probably for AArch64 and MIPS). I believe you’ve heard a few good news about the linker – it just works and is very fast, clean, compact and supported by the active community. I don’t think I need to reiterate why having a good linker is important for us as the LLVM community.

There’s one thing you can help us without writing even a line of code. Please use it!

Using LLD to link LLVM/clang/etc is easy. You need to check out the LLD repository just like you probably already did for clang, build it, and then install it. LLD will be installed as ld.lld (and it won’t be used by default.) After that, re-run cmake with -DLLVM_ENABLE_LLD=On along with your usual options so that LLD will be used to build LLVM.

For the details of the build process, please see http://lld.llvm.org/#build.

Thanks,
Rui

Is bugs.llvm.org still the best place to report LLD bugs? Is anyone
watching those bug reports?

For example I reported https://bugs.llvm.org/show_bug.cgi?id=32254
yesterday but I'm not sure if anyone saw it.

Regards,
Andrew

Yes that’s the right place to file a bug. However, as to LLD/Mach-O, that isn’t being developed actively.

Date: Tue, 14 Mar 2017 11:39:22 -0700
From: Rui Ueyama via llvm-dev <llvm-dev@lists.llvm.org>

Hi all,

LLVM 4.0.0 is out, and I can say that LLD/ELF is now ready for production
use at least for x86-64 (and probably for AArch64 and MIPS). I believe
you've heard a few good news about the linker -- it just works
<http://lld.llvm.org/#features&gt; and is very fast
<http://lld.llvm.org/#performance&gt;, clean, compact and supported by the
active community. I don't think I need to reiterate why having a good
linker is important for us as the LLVM community.

FWIW, we're using lld as the system linker (and clang as the system
compiler) on the upcoming OpenBSD/arm64. The system is self-hosting
and we've had to fix only a couple of small things in the way we use
the linker to get to this point. All security-related features we
rely on (-pie, -static -pie, W^X, .openbsd.randomdata) seem to work
just fine. So I'd say AArch64 is defenitely there.

A big thank you,

Mark

Out of curiosity, what's the story about architectures other than AArch64?

We’re now using it with all three architectures on FreeBSD (though only the n64 ABI for MIPS) and will file bugs. One QoI issue that’s bitten us a couple of times is that the versions of clang and lld must match for LTO to work. If they don’t, we don’t get an error message, we simply get programs that segfault on startup. I’m a bit surprised that reading the bitcode doesn’t return an error that LLD can report and refuse to link. We’ve seen this error mixing clang 4.0 and lld trunk.

That’s going to make enabling LTO for FreeBSD package builds problematic once we start shipping lld in the base system - anything built with the system compiler will work fine, but anything built with a compiler from ports will silently generate non-working binaries. Fortunately for LLVM, this includes building a non-working tablegen, so the build fails rather than appearing to succeed and giving non-working binaries.

For dogfooding purposes, it would be very helpful if the LLVM CMake files allowed you to specify the suffix for the system clang and LLD to use in a single place. Currently, telling it to use lld will pass -fuse-ld=lld, but we typically need to actually pass -fuse-ld=lld40 or -fuse-ld=lld-devel to pick the LLD version that matches the compiler version that we’re using.

David

From: Rui Ueyama <ruiu@google.com>
Date: Tue, 14 Mar 2017 14:04:34 -0700

> > Date: Tue, 14 Mar 2017 11:39:22 -0700
> > From: Rui Ueyama via llvm-dev <llvm-dev@lists.llvm.org>
> >
> > Hi all,
> >
> > LLVM 4.0.0 is out, and I can say that LLD/ELF is now ready for production
> > use at least for x86-64 (and probably for AArch64 and MIPS). I believe
> > you've heard a few good news about the linker -- it just works
> > <http://lld.llvm.org/#features&gt; and is very fast
> > <http://lld.llvm.org/#performance&gt;, clean, compact and supported by the
> > active community. I don't think I need to reiterate why having a good
> > linker is important for us as the LLVM community.
>
> FWIW, we're using lld as the system linker (and clang as the system
> compiler) on the upcoming OpenBSD/arm64. The system is self-hosting
> and we've had to fix only a couple of small things in the way we use
> the linker to get to this point. All security-related features we
> rely on (-pie, -static -pie, W^X, .openbsd.randomdata) seem to work
> just fine. So I'd say AArch64 is defenitely there.
>

Out of curiosity, what's the story about architectures other than AArch64?

We're planning to switch OpenBSD/i386 and OpenBSD/amd64 over to clang
in our next release cycle (for OpenBSD 6.2). Once we're happy with
that, we'll try to move them to lld. I'm confident that the base
system will be no problem (linker scripts will need some attention).
But we have to be careful that we don't inflict to much pain on our
ports people.

One issue that's cropped up is that GNU libtool doesn't recognize lld
as a GNU compatible linker. Ed Maste posted a message on the libtool
mailing list about this:

  libtool support for LLVM's LLD linker

But it seems there has been no response yet. Perhaps a coordinated
action is needed here. Or perhaps we can cheat and include "GNU"
somewhere in the lld -v output ;).

Let's please not do this. It would give the wrong signal.
If libtool won't acknowledge lld's existence, then there
are better ways like a rewrite and (temp) downstream patch.

Or perhaps we can cheat and include "GNU" somewhere in the
lld -v output ;).

This is the hack Rafael's been using for testing, adding "not GNU".
libtool also looks for "supported targets: elf" in --help output. You
can see the details of Rafael's hacks at
http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20161212/411904.html

Let's please not do this. It would give the wrong signal.
If libtool won't acknowledge lld's existence, then there
are better ways like a rewrite and (temp) downstream patch.

Note that even if/when we get changes incorporated into libtool it
will still take a long time for the change to appear in new releases
of software packages using libtool. A downstream patch to the libtool
package in various operating systems or distributions doesn't really
help that much for the same reason.

For the FreeBSD ports tree the most likely workaround would be some
common sed magic to fix libtool's tests, along with a setting in
(hundreds of) individual port Makefiles to apply it on a case-by-case
basis.

Note that even if/when we get changes incorporated into libtool it
will still take a long time for the change to appear in new releases
of software packages using libtool. A downstream patch to the libtool
package in various operating systems or distributions doesn't really
help that much for the same reason.

AFAIU, systems where lld is the default ld are limited right
now and can temporarily carry a minimally modified libtool.
That way systems which have bfd or gold as default ld, which is
most of the FOSS operating systems out there, won't notice any
disturbance before, during or after libtool gets to know lld.
I'm probably overlooking something but it looks like classical
downstream integration patching due to the fact that it may
take a while to get upstream or be rejected for non-technical
reasons.

What am I missing?

I've been using ldd for building various code bases with autotools
and libtool scripting, and must have been lucky to not run into
the problem yet. I'll keep an eye out for libtool errors.

For the FreeBSD ports tree the most likely workaround would be some
common sed magic to fix libtool's tests, along with a setting in
(hundreds of) individual port Makefiles to apply it on a case-by-case
basis.

If FreeBSD's libtool is patched to be aware of the base ld=lld,
would this still be true?

>
> LLVM 4.0.0 is out, and I can say that LLD/ELF is now ready for
production use at least for x86-64 (and probably for AArch64 and MIPS).

We’re now using it with all three architectures on FreeBSD (though only
the n64 ABI for MIPS) and will file bugs. One QoI issue that’s bitten us a
couple of times is that the versions of clang and lld must match for LTO to
work. If they don’t, we don’t get an error message, we simply get programs
that segfault on startup. I’m a bit surprised that reading the bitcode
doesn’t return an error that LLD can report and refuse to link. We’ve seen
this error mixing clang 4.0 and lld trunk.

If you can put a --reproduce archive up somewhere and file a bug that would
be really useful. The bitcode should be backward compatible. I would not be
surprised by older LLD and newer Clang causing issues, but the reverse
Should Work.

-- Sean Silva

I agree with Sean, that doesn't sound right. Not only I do LTO with
mixed versions of lld and LLVM several hundreds of times per week, but
when lld wasn't still so popular I was able to do LTO of the FreeBSD
base system almost entirely without seeing any SIGSEGV at startup in
the linked applications. In other words, this should just work, and if
it doesn't it's a legitimate bug (and I'll appreciate if you can
report it upstream).

It appears that I’d misdiagnosed the problem. Even with matching versions, I get a non-functional tablegen for a thin LTO build on FreeBSD, but when I set them to identical versions CMake had failed to pick up the LTO instruction and so was doing a normal non-LTO build.

David

Parts of libtool, including the detection of shared library support are
backed into nearly every configure script in the wild. Some care about
the result of those checks for further decisions, some don't. It is the
former which is problematic, especially since rebuilding configure isn't
always as easy as it sounds.

Joerg

Any idea of the quality/completeness of the gdb_index support? I’ve started using gdb_index recently (reduces GDB startup time for Clang from 50 seconds to 3 seconds - though I haven’t measure how much it increases link time*) & it’s really handy. Would love to have some assurance it’s expected to work for LLD.

Simple experiments seem to indicate that it’s OK/good. Though a quick llvm-dwarfdump of gdb_index between the two linkers shows about a 10% disparity in size (LLD’s is smaller) given the same inputs. (219MB v 196MB of text dumped - I didn’t compare the actual section sizes)

*though honestly I’d be willing to trade startup time for link time 1:1 even, because builds/links are already long enough that I go off and do something else - whereas getting GDB start down to 3 seconds, I can start writing my breakpoint command, etc, in those 3 seconds and be all but unaffected by it - at 50 seconds I basically have to go off and do something else/context switch/etc.

I personally haven't tried gdb_index, and I don't know the quality of the
produced index. Most of the code was written by George.

One thing I noticed about the feature (and filed as
http://bugs.llvm.org/show_bug.cgi?id=32228) is that our gdb_index feature
is much slower than the gold. Apparently there's room for improvement.

Yes, for the same reason Joerg points out in another reply in this
thread - some software packages are going to include in their
configure scripts parts of whatever libtool version existed on the
machine used by a developer to produce a release.

We'll soon have cases where there will be a LLD 4.0.0 /usr/bin/ld
provided by the base system, and Clang 4.1.0 /usr/local/bin/clang
installed from a package. It probably means we just need to have that
Clang use its own copy of LLD in the same package.

It'll be great if we can detect and error on this case rather than
producing broken output though.

Any chances you can report the llvm-tblgen failure? I built that all
the time (even with ThinLTO and I've never seen it, so I'm a little
surprised).

Thanks, never used libtool explicitly myself and wasn't aware
it's also one of those tools that has you bundle a generated
script with your build tooling. I've had my share of pain
when running autoconf configure.in so I guess it's the same
kind of problems that plague libtoolize.