Linking Linux kernel with LLD

Hi all!

Recently there was a discussion on this mailing list about linking Linux and GRUB with LLD.

I have actually tried to link Linux kernel with LLD and (with some modifications) it linked successfully (but didn't run well).

Here is the list of modifications I had to do in order to link the kernel (I have used llvmlinux with clang and mainline with gcc, the results are similar):

1. LLD patches:
  - D28094 (Implemented support for R_386_PC8/R_386_8 relocations)
  - D28857 (Allow relative relocations to a absolute value defined in linker script)
  - [not actually used] D28612 (Added support for --emit-relocs)

2. Kernel configuration. I tried "make tinyconfig" and then changed to CONFIG_64BIT=y with menuconfig. I wanted the minimal kernel configuration for x86_64. After I get it working, I will try defconfig.

3. In arch/x86/Makefile changed the M16_CFLAGS to just "-m16". I think that cc-option should test if compiler supports '-m16' option and if it does, use it, if it doesn't use '-m32 -Wa,./arch/x86/boot/code16gcc.h' instead. I don't know why, but this test failed for me and LLD didn't accept this header, so I just simply changed the flag to '-m16' because I didn't want to investigate why did the test failed. It's just a quick workaround, but it works.

3. In arch/x86/boot/Makefile and arch/x86/realmode/rm/Makefile added "-m elf_i386" to LDFLAGS_setup.elf and LDFLAGS_realmode.elf accordingly. The problem is that the linker is invoked as "ld.lld -m elf_x86_64" and it ignores the OUTPUT_FORMAT from linker scripts. Being invoked as "ld.lld -m elf_x86_64 -m elf_i386" doesn't break anything and LLD even produces the right format of the binaries. Again a quick workaround, but it works.

4. In arch/x86/realmode/rm/Makefile removed "--emit-relocs" from LDFLAGS_realmode.elf. This is the problem with D28612 — I applied the patch to support --emit-relocs option, but LLD segfaulted. Reported here: 31579 – ld.lld doesn't support --emmit-relocs (-q)

5. In arch/x86/kernel/vmlinux.lds.S commented out the "CONSTRUCTORS", because LLD doesn't support it.

6. In arch/x86/boot/setup.ld replaced 5*512 with precalculated value of 2560 because it doesn't seem that LLD supports math inside ASSERT in linker scripts.

7. In arch/x86/boot/setup.ld commented out the line ". = ASSERT(_end <= 0x8000, "Setup too big!");" because it seems to always fire this assertion, and the kernel is not going to be working anyway, so I just want it linked even if it will be unusable.

8. In arch/x86/lib/copy_user_64.S removed the line "bad_from_user:" (probably it was just a mistake), in arch/x86/kernel/vmlinux.lds.S removed semicolon after "*(.apicdrivers)" (also probably just a mistake).

Finally the kernel was built and it obviously didn't run (only printed out "No setup signature found..." but this is some result as well). Probably, the result could be better if the --emit-relocs option didn't fail and CONSTRUCTORS were supported. I really don't know what to do about the assertion that I have commented out.

I also tried using tools provided by LLVM instead of binutils (HOSTCC=clang HOSTLD=ld.lld CC=clang LD=ld.lld AR=llvm-ar NM=llvm-nm), it seems to work with llvmlinux, but clang will not work with the mainline. Clang doesn't seem to use integrated assembler, so I had to write a simple wrapper script around llvm-mc to use instead GNU as. I also have few patches to make llvm-objcopy to work with latest LLVM and support needed subset of flags, will publish it soon.

You can find my workarounds as a patch (the kernel will not work with this applied!) and my kernel config attached.

Regards,
Dmitry

kernel-lld-workarounds.patch (3 KB)

kernel-config (20.5 KB)

Wow, great work! I’m sure you are the first person who linked the Linux kernel using LLD. :slight_smile:

I think “no setup signature found” error is an error of the boot loader (probably grub). The resulting binary probably lacked magic bytes or something that a boot loader uses to identify bootable images. It should be fixable by investigating what is actually missing and fix it.

Hi all!

Recently there was a discussion on this mailing list about linking Linux
and GRUB with LLD.

I have actually tried to link Linux kernel with LLD and (with some
modifications) it linked successfully (but didn't run well).

Awesome! Thanks for digging into this!

When Michael, Davide, and I were debugging the issues with running the
LLD-linked FreeBSD kernel, we found that there was really no alternative
but to get familiar with the bootloader code and do printf debugging to
understand what the bootloader didn't like. (even hooking into qemu's
debugger integration didn't work; it is just too early for the usual
debugger integration to work I think (interrupts and stuff not in a "sane"
state etc.)). I'm happy to share more info about that experience if it
would be useful to you.

I'm guessing Ed (CC'd) also had some similar experiences getting the
FreeBSD bootloaders working with LLD and might have some advice as well
w.r.t. a good workflow and what tools to use.

Here is the list of modifications I had to do in order to link the kernel
(I have used llvmlinux with clang and mainline with gcc, the results are
similar):

1. LLD patches:
  - D28094 (Implemented support for R_386_PC8/R_386_8 relocations)
  - D28857 (Allow relative relocations to a absolute value defined in
linker script)
  - [not actually used] D28612 (Added support for --emit-relocs)

2. Kernel configuration. I tried "make tinyconfig" and then changed to
CONFIG_64BIT=y with menuconfig. I wanted the minimal kernel configuration
for x86_64. After I get it working, I will try defconfig.

3. In arch/x86/Makefile changed the M16_CFLAGS to just "-m16". I think
that cc-option should test if compiler supports '-m16' option and if it
does, use it, if it doesn't use '-m32 -Wa,./arch/x86/boot/code16gcc.h'
instead. I don't know why, but this test failed for me and LLD didn't
accept this header, so I just simply changed the flag to '-m16' because I
didn't want to investigate why did the test failed. It's just a quick
workaround, but it works.

These seem to be compiler flags. It is probably best to use GCC as the
compiler for now (i.e. only swap out the linker). That will allow focusing
on the LLD issues. Clang (and other LLVM tools) have their own issues with
GNU compatibility, but things will be more difficult to debug when there
are more moving pieces.

-- Sean Silva

One thing I remember is that Linux’s build system does directly tamper with object files with custom tools, so those tools might be unhappy with LLD’s binaries or just do the wrong thing (sometimes such tools have hardcoded assumptions etc.). One specific case I remember is that linux’s copy_from_user/copy_to_user implementations use inline asm to build up a table which is sorted by a custom tool (I think this modification happens post-link even).
http://lxr.free-electrons.com/source/arch/x86/include/asm/uaccess.h

– Sean Silva