Problems using Clang with LLD on embedded ARM

Project Infrastructure LLVM Dev List Archives
#1

Hi,

I stopped into IRC to ask about a problem I’ve been having using Clang in conjunction with LLD to compile and link for an embedded project on Cortex-M ARM processor.

First, I am able to separately compile with a call to clang and link with a call to lld, but I cannot use clang to link using lld using the -fuse-ld=lld flag. I have the output from clang -v -fuse-ld=lld -target arm-none-eabi main.c -o main here [1], where calling clang -target arm-none-eabi -c main.c -o main.o and ld.lld main.o -o main work fine. Doing the compiling and linking for my host computer works just fine using -fuse-ld=lld, and I verified that it is using LLD by checking the .comments section in the generated elf.

Second, combining LTO with specifying any specific arm architecture seems to break linking with LLD. That is, I compile the object files with LTO (either -flto or -flto=thin) and, in addition to specifying -target arm-none-eabi, I specify an architecture specific flag (such as -march=armv7e-m, -mcpu=cortex-m4, or -mthumb, or instead I use -target armv7em-none-eabi), then while calling lld to link I get an error like:
ld.lld: error: main.o: could not infer e_machine from bitcode target triple thumbv7em-none–eabi
LLD 4.0.0

I am working on Linux and these two problems seem to exist on both 4.0rc2 and the current head.

I am really happy with the progress LLVM has made, with the upcoming 4.0 release, I have managed to compile and link programs for my embedded projects without any of the gcc arm toolchain! There are still some shortcomings with the linker scripts compared to ld that make those a bit trickier to work with, even with the added MEMORY section parsing in the current head. Other than the above, the only other bother I’ve hit is having to re-link with --oformat binary to get raw binary output, since there I haven’t found an objcopy equivalent in llvm. I can add feature requests for these missing features, as well as the bug reports for the above two issues, on the bugzilla if that is appropriate.

Thanks for the help,

Sean

[1] https://gist.github.com/flacjacket/35d92ea619af96352d5ac06844f2f1b5

#2

Hi,

I stopped into IRC to ask about a problem I've been having using Clang in
conjunction with LLD to compile and link for an embedded project on
Cortex-M ARM processor.

First, I am able to separately compile with a call to clang and link with
a call to lld, but I cannot use clang to link using lld using the
-fuse-ld=lld flag. I have the output from `clang -v -fuse-ld=lld -target
arm-none-eabi main.c -o main` here [1], where calling `clang -target
arm-none-eabi -c main.c -o main.o` and `ld.lld main.o -o main` work fine.
Doing the compiling and linking for my host computer works just fine using
-fuse-ld=lld, and I verified that it is using LLD by checking the .comments
section in the generated elf.

For that triple, Clang seems to be calling into GCC driver
(/usr/bin/arm-none-eabi-gcc) for linking and GCC doesn't recognize
-fuse-ld=lld (supposedly -fuse-ld=gold selects ld.gold, -fuse-ld=bfd
selects ld.bfd and you would expect -fuse-lld=lld to select ld.lld, but it
doesn't work like that apparently).

@Renato: Do you know why Clang is forwarding to GCC like this for linking?
If we need to live with this arrangement for a long time we should
definitely invest in teaching GCC about -fuse-ld=lld

@Davide: is there any chance you could revive that GCC patch to add support
for LLD?

(though it does seem like we still fail H.J.'s test case in
28414 – lld can't be used with GCC 7 has not yet been fixed. I
further reduced it. Seems to be related to symbol versioning. We don't
recognize that `.symver bar, foo@@VERS` is a definition of `foo`.
)

Second, combining LTO with specifying any specific arm architecture seems
to break linking with LLD. That is, I compile the object files with LTO
(either -flto or -flto=thin) and, in addition to specifying `-target
arm-none-eabi`, I specify an architecture specific flag (such as
-march=armv7e-m, -mcpu=cortex-m4, or -mthumb, or instead I use -target
armv7em-none-eabi), then while calling lld to link I get an error like:
ld.lld: error: main.o: could not infer e_machine from bitcode target
triple thumbv7em-none--eabi
LLD 4.0.0

If you are building your own LLD, can you see if the attached patch fixes
it for you? If not, can you add `--reproduce /tmp/repro.tar` to your link
command line and upload repro.tar somewhere for us?

@Renato, do you know if there is any way for us to
implement getBitcodeMachineKind in a sane way so that we don't fill in a
long tail of cases of inferring EM_* from the triple? It seems that the
EM_* choices are buried very deep in the backend, so it might not be
feasible :frowning:

I am working on Linux and these two problems seem to exist on both 4.0rc2
and the current head.

I am really happy with the progress LLVM has made, with the upcoming 4.0
release, I have managed to compile and link programs for my embedded
projects without any of the gcc arm toolchain! There are still some
shortcomings with the linker scripts compared to ld that make those a bit
trickier to work with, even with the added MEMORY section parsing in the
current head.

What kind of issues have you been running into?

Linker scripts are certainly one area that we have been slowly filling in
the long tail of compatibility, it would be great if you filed bugs with
any issues you encounter! If the files are small enough and you can share
them, it can be as simple as just adding `--reproduce /tmp/repro.tar` to
your link command line (or setting LLD_REPRODUCE=/tmp/repro.tar in the
environment) and attach repro.tar to a bug report whenever you encounter an
issue (even if repro.tar is larger, you can try hosting on google drive /
dropbox etc., compressing with xz also often helps). Or if it is a parsing
problem just attaching the problematic linker script to the bug report
might be enough.

Other than the above, the only other bother I've hit is having to re-link
with `--oformat binary` to get raw binary output, since there I haven't
found an objcopy equivalent in llvm. I can add feature requests for these
missing features, as well as the bug reports for the above two issues, on
the bugzilla if that is appropriate.

Regarding objcopy, that is something that we have wanted to write for a
long time (but it has never quite been a priority). As objcopy supports a
bewildering set of different options, it would be useful for you to file a
bug about what specific options/features you need. We don't really have a
bugzilla component for objcopy yet, so you can just throw it under llvm-nm
(another one of the binutils, close enough) in
https://bugs.llvm.org/enter_bug.cgi?product=tools

-- Sean Silva

thumb.patch (371 Bytes)

#3

Sean Silva via llvm-dev <llvm-dev@lists.llvm.org> writes:

For that triple, Clang seems to be calling into GCC driver
(/usr/bin/arm-none-eabi-gcc) for linking and GCC doesn't recognize
-fuse-ld=lld (supposedly -fuse-ld=gold selects ld.gold, -fuse-ld=bfd
selects ld.bfd and you would expect -fuse-lld=lld to select ld.lld, but it
doesn't work like that apparently).

Gosh, that is an old issue. I was sure we had converted to used ld
everywhere and deleted the old code for using the gcc driver as a
linker.

Doing that is probably the correct way to fix this issue. Having gcc
itelf support -fuse-ld=lld would be an awesome independent improvement.

Cheers,
Rafael

#4

I can probably find some time to revive that support next week
(sorry it's GDC week this week and I won't likely have time
for anything compiler-related).

#5

Hi Sean

#6

You still need a runtime library, even with freestanding. Both
__aeabi_memclr4 and __aeabi_uidiv are helpers as defined by EABI and
their presence is required even with freestanding.

Joerg

#7

IIRC, most users (kernel, android) just implement them or hook into
external libraries. It's not complicated and there's a lot of prior
art.

--renato

#8

This is really odd, but since LLD is not officially supported on ARM
yet (Peter is almost there!), we haven't got around using it for
anything serious.

It smells like yet-another clang driver kerfuffle when cross-compiling.

cheers,
--renato

#9

Not really. The generic none target has a fallback in clang to GCC,
because it can't know by definition how to link for such a target.
At this point in time it might be better to just error out...

Joerg

#10

Hi Sean

For that triple, Clang seems to be calling into GCC driver
(/usr/bin/arm-none-eabi-gcc) for linking and GCC doesn't recognize
-fuse-ld=lld (supposedly -fuse-ld=gold selects ld.gold, -fuse-ld=bfd
selects ld.bfd and you would expect -fuse-lld=lld to select ld.lld, but it
doesn't work like that apparently).

@Renato: Do you know why Clang is forwarding to GCC like this for
linking? If we need to live with this arrangement for a long time we should
definitely invest in teaching GCC about -fuse-ld=lld

@Davide: is there any chance you could revive that GCC patch to add
support for LLD?

(though it does seem like we still fail H.J.'s test case in
https://bugs.llvm.org//show_bug.cgi?id=28414 has not yet been fixed. I
further reduced it. Seems to be related to symbol versioning. We don't
recognize that `.symver bar, foo@@VERS` is a definition of `foo`.
)

If you are building your own LLD, can you see if the attached patch fixes
it for you? If not, can you add `--reproduce /tmp/repro.tar` to your link
command line and upload repro.tar somewhere for us?

Thanks! I applied that patch and it links with -flto now, but it looks
like there are now some other problems. First, it seems to be dropping
things that I try to KEEP in the linker script, even when I don't use
--gc-sections, namely my interrupt vector table.

This sounds like it might be an issue in the LTO internalization logic. Can
you file a bug?
If that is what is happening, then you should be able to add
`__attribute__((used))` to the interrupt vector table to avoid it from
being lost by LTO.

Slightly longer description, if you're interested: As part of LTO, there
are optimization passes that run on the IR which can have GC-like effects,
and it is LLD's responsibility to ensure that symbols are correctly marked
as needing to be kept. We want to tell the IR optimizer precise info about
what is live. Even if a function `foo` is dead and will be stripped by the
regular linker GC later in the link, if the IR optimizer doesn't know that
`foo` is dead then `foo` can still prevent its callees from being
profitable to inline somewhere else that is actually live (the inliner
might see two calls when there is actually only one). It sounds like
currently the KEEP's in the linker script are the only thing informing the
linker that the interrupt vector table must be kept, and I don't think LLD
factors that into what it communicates to the IR optimizer.

Second, when using thin LTO, it starts trying to link symbols
__aeabi_memclr4 and __aeabi_uidiv, which I don't think it should since I am
using -ffreestanding -fno-builtins when compiling (with full lto or no lto,
it does not try to link these symbols).

I just sent an email to register for a bugzilla account, I can report
there with a minimally reproducible example for these problems.

@Renato, do you know if there is any way for us to
implement getBitcodeMachineKind in a sane way so that we don't fill in a
long tail of cases of inferring EM_* from the triple? It seems that the
EM_* choices are buried very deep in the backend, so it might not be
feasible :frowning:

What kind of issues have you been running into?

Linker scripts are certainly one area that we have been slowly filling in
the long tail of compatibility, it would be great if you filed bugs with
any issues you encounter! If the files are small enough and you can share
them, it can be as simple as just adding `--reproduce /tmp/repro.tar` to
your link command line (or setting LLD_REPRODUCE=/tmp/repro.tar in the
environment) and attach repro.tar to a bug report whenever you encounter an
issue (even if repro.tar is larger, you can try hosting on google drive /
dropbox etc., compressing with xz also often helps). Or if it is a parsing
problem just attaching the problematic linker script to the bug report
might be enough.

I haven't hit anything that looked like a bug, just features that aren't
implemented, or that work slightly differently between ld linker scripts.
I'd be happy to make additional feature requests for those on the bug
tracker, too.

I said "bug" because well it is "bug"zilla, but yes, feature requests /
compatibility issues are definitely very welcome. "ticket" is probably a
better word, but "bug" is too ingrained for me :slight_smile:

Regarding objcopy, that is something that we have wanted to write for a

long time (but it has never quite been a priority). As objcopy supports a
bewildering set of different options, it would be useful for you to file a
bug about what specific options/features you need. We don't really have a
bugzilla component for objcopy yet, so you can just throw it under llvm-nm
(another one of the binutils, close enough) in
https://bugs.llvm.org/enter_bug.cgi?product=tools

Sounds good.

Thanks for the patch and the help.

No problem. Thanks for helping us find and fix our bugs :slight_smile:

-- Sean Silva