LTO, ifuncs, and lld

Hey LLVM folks,

I've run into an interesting assertion. In one of HardenedBSD's
feature branches, we're working on integration llvm's Cross-DSO CFI
implementation. Using Cross-DSO CFI requires building libs with LTO,
which causes clang to emit LLVM IR intermediate object files rather
than ELF intermediate object files.

I've found that with lld, attempting to link LLVM IR intermediate
object files hits an assert in lld. I've created a reproduction test
case in this tiny little repo: https://github.com/lattera/ifunc_repro

The assertion I hit is detailed in the commit message of the initial
commit:
https://github.com/lattera/ifunc_repro/commit/0be98f9e81a1c91e80b135da6bb8d073d7a0c6f7

HardenedBSD's Cross-DSO CFI feature branch uses clang/llvm/lld 7.0.1.
I'm more than happy to test out patches to help address this issue.

Please let me know if you have any questions, comments, or concerns.

Thanks,

Hi Shawn,

Can you please create a reproducer tarball (using ld.lld --reproduce) so that we don’t need to install HardenedBSD in order to reproduce?

Peter

Hey Peter,

Here you go!

https://hardenedbsd.org/~shawn/2018-11-28_reproduce-01.tar

Thanks,

https://reviews.llvm.org/D55046 fixes your reproducer here. Let me know if that works for you.

Note that I had to rebuild one of your bitcode files to be a regular object file, this is because LTO doesn’t want the .eh_frame terminator to live in a bitcode file.

$ mv ./usr/lib/crtendS.o ./usr/lib/crtendS.o.bak

$ ~/l4/ra/bin/llc -o ./usr/lib/crtendS.o ./usr/lib/crtendS.o.bak -filetype=obj -relocation-model=pic

Peter

Thanks for providing the patch! I got around to testing it this
morning and it appears it fixes compilation, but produces a
non-working system.

I know that's kinda vague and I'll have more details soon, including
sample binaries. I at least wanted to give a status update so you
didn't think you were being ignored.

Thanks,

It looks like this commit breaks CSU initialization with
statically-compiled applications.

With a very simple application at [1], compiled with:
cc -g -O0 -flto -static -o pid pid.c

The application segfaults:

# lldb ./pid
(lldb) target create "./pid"
Current executable set to './pid' (x86_64).
(lldb) run
Process 84115 launching
Process 84115 launched: '/root/pid' (x86_64)
Process 84115 stopped
* thread #1, name = 'pid', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
    frame #0: 0x000000000024e6d0 pid`__je_malloc_tsd_boot0 [inlined] tsd_fetch_impl(init=true, minimal=false) at tsd.h:265
(lldb) bt
* thread #1, name = 'pid', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
  * frame #0: 0x000000000024e6d0 pid`__je_malloc_tsd_boot0 [inlined] tsd_fetch_impl(init=true, minimal=false) at tsd.h:265
    frame #1: 0x000000000024e6d0 pid`__je_malloc_tsd_boot0 [inlined] tsd_fetch at tsd.h:292
    frame #2: 0x000000000024e6d0 pid`__je_malloc_tsd_boot0 at jemalloc_tsd.c:266
    frame #3: 0x0000000000225581 pid`malloc_init [inlined] malloc_init_hard at jemalloc_jemalloc.c:1527
    frame #4: 0x00000000002254ca pid`malloc_init at jemalloc_jemalloc.c:221
    frame #5: 0x000000000021b208 pid`handle_static_init(argc=1, argv=0x00006f9d260072f8, env=0x00006f9d26007308) at ignore_init.c:124
    frame #6: 0x000000000021b103 pid`_start(ap=<unavailable>, cleanup=<unavailable>) at crt1.c:75
(lldb)

[1]: https://gist.github.com/lattera/758b28c1e315cd70e670dd5211388864

The CSU can be found here:
https://github.com/HardenedBSD/hardenedBSD/tree/hardened/current/master/lib/csu

I'm working on amd64 (so crt1.c would be at lib/csu/amd64/crt1.c). The
handle_static_init function is here:
https://github.com/HardenedBSD/hardenedBSD/blob/hardened/current/master/lib/csu/common/ignore_init.c

Thanks,

It's at this point where I think about filing a full bug report with
llvm. Any hints before I do?

Hello Shawn,

I've not been following the thread in detail so apologies if this
isn't related. I know of one problem with LLD, static linking and
ifuncs (https://bugs.llvm.org/show_bug.cgi?id=38074) . There is a
patch for X86 at https://reviews.llvm.org/D54145 but I don't think
that it has landed yet. The AArch64 version
https://reviews.llvm.org/D54314 did land, but it looks like it is
causing some problems with sanitisers
https://bugs.llvm.org/show_bug.cgi?id=40250 so it is clear some more
work is needed in this area.

Peter

Shawn,

It might also help to try patching in https://reviews.llvm.org/D57371 which fixes a number of bugs around ifuncs in statically linked binaries.

That said, it’s unclear whether this issue has anything to do with ifuncs. Looking briefly at the jemalloc code it might have something to do with TLS?

Peter